diff --git "a/data/cvpr2024_papers_with_details.csv" "b/data/cvpr2024_papers_with_details.csv"
new file mode 100644--- /dev/null
+++ "b/data/cvpr2024_papers_with_details.csv"
@@ -0,0 +1,44044 @@
+Title,Authors,Link,arXiv_link,other_link,pdf_path,arXiv_title,summary,primary_category,categories
+No title found,No authors listed, ,,https://pine.libguides.com/c.php?g=997445&p=7219661,,,,,nan
+CapsFusion: Rethinking Image-Text Data at Scale,Qiying Yu · Quan Sun · Xiaosong Zhang · Yufeng Cui · Yufeng Cui · Fan Zhang · Yue Cao · Xinlong Wang · Jingjing Liu, ,https://arxiv.org/abs/2310.20550,,,CapsFusion: Rethinking Image-Text Data at Scale,"Large multimodal models demonstrate remarkable generalist ability to perform
+diverse multimodal tasks in a zero-shot manner. Large-scale web-based
+image-text pairs contribute fundamentally to this success, but suffer from
+excessive noise. Recent studies use alternative captions synthesized by
+captioning models and have achieved notable benchmark performance. However, our
+experiments reveal significant Scalability Deficiency and World Knowledge Loss
+issues in models trained with synthetic captions, which have been largely
+obscured by their initial benchmark success. Upon closer examination, we
+identify the root cause as the overly-simplified language structure and lack of
+knowledge details in existing synthetic captions. To provide higher-quality and
+more scalable multimodal pretraining data, we propose CapsFusion, an advanced
+framework that leverages large language models to consolidate and refine
+information from both web-based image-text pairs and synthetic captions.
+Extensive experiments show that CapsFusion captions exhibit remarkable
+all-round superiority over existing captions in terms of model performance
+(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
+efficiency (requiring 11-16 times less computation than baselines), world
+knowledge depth, and scalability. These effectiveness, efficiency and
+scalability advantages position CapsFusion as a promising candidate for future
+scaling of LMM training.",cs.CV,nan
+Semantic-Aware Multi-Label Adversarial Attacks,Hassan Mahmood · Ehsan Elhamifar, ,https://arxiv.org/abs/2401.16001,,2401.16001.pdf,LESSON: Multi-Label Adversarial False Data Injection Attack for Deep Learning Locational Detection,"Deep learning methods can not only detect false data injection attacks (FDIA)
+but also locate attacks of FDIA. Although adversarial false data injection
+attacks (AFDIA) based on deep learning vulnerabilities have been studied in the
+field of single-label FDIA detection, the adversarial attack and defense
+against multi-label FDIA locational detection are still not involved. To bridge
+this gap, this paper first explores the multi-label adversarial example attacks
+against multi-label FDIA locational detectors and proposes a general
+multi-label adversarial attack framework, namely muLti-labEl adverSarial falSe
+data injectiON attack (LESSON). The proposed LESSON attack framework includes
+three key designs, namely Perturbing State Variables, Tailored Loss Function
+Design, and Change of Variables, which can help find suitable multi-label
+adversarial perturbations within the physical constraints to circumvent both
+Bad Data Detection (BDD) and Neural Attack Location (NAL). Four typical LESSON
+attacks based on the proposed framework and two dimensions of attack objectives
+are examined, and the experimental results demonstrate the effectiveness of the
+proposed attack framework, posing serious and pressing security concerns in
+smart grids.",cs.CR,['cs.CR']
+Towards Better Vision-Inspired Vision-Language Models,Yun-Hao Cao · Kaixiang Ji · Ziyuan Huang · Chuanyang Zheng · Jiajia Liu · Jian Wang · Jingdong Chen · Ming Yang, ,,https://www.youtube.com/watch?v=d91e0EwAIZc,,,,,nan
+HINTED: Hard Instance Enhanced Detector with Mixed-Density Feature Fusion for Sparsely-Supervised 3D Object Detection,Qiming Xia · Wei Ye · Hai Wu · Shijia Zhao · Leyuan Xing · Xun Huang · Jinhao Deng · Xin Li · Chenglu Wen · Cheng Wang,https://github.com/xmuqimingxia/HINTED,https://arxiv.org/abs/2308.04556,,2308.04556.pdf,FocalFormer3D : Focusing on Hard Instance for 3D Object Detection,"False negatives (FN) in 3D object detection, {\em e.g.}, missing predictions
+of pedestrians, vehicles, or other obstacles, can lead to potentially dangerous
+situations in autonomous driving. While being fatal, this issue is understudied
+in many current 3D detection methods. In this work, we propose Hard Instance
+Probing (HIP), a general pipeline that identifies \textit{FN} in a multi-stage
+manner and guides the models to focus on excavating difficult instances. For 3D
+object detection, we instantiate this method as FocalFormer3D, a simple yet
+effective detector that excels at excavating difficult objects and improving
+prediction recall. FocalFormer3D features a multi-stage query generation to
+discover hard objects and a box-level transformer decoder to efficiently
+distinguish objects from massive object candidates. Experimental results on the
+nuScenes and Waymo datasets validate the superior performance of FocalFormer3D.
+The advantage leads to strong performance on both detection and tracking, in
+both LiDAR and multi-modal settings. Notably, FocalFormer3D achieves a 70.5 mAP
+and 73.9 NDS on nuScenes detection benchmark, while the nuScenes tracking
+benchmark shows 72.1 AMOTA, both ranking 1st place on the nuScenes LiDAR
+leaderboard. Our code is available at
+\url{https://github.com/NVlabs/FocalFormer3D}.",cs.CV,['cs.CV']
+"DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences, Neuron Visualisations, and Visual Counterfactual Explanations",Maximilian Augustin · Yannic Neuhaus · Matthias Hein, ,https://arxiv.org/abs/2311.17833,,2311.17833.pdf,"DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences, Neuron Visualisations, and Visual Counterfactual Explanations","While deep learning has led to huge progress in complex image classification
+tasks like ImageNet, unexpected failure modes, e.g. via spurious features, call
+into question how reliably these classifiers work in the wild. Furthermore, for
+safety-critical tasks the black-box nature of their decisions is problematic,
+and explanations or at least methods which make decisions plausible are needed
+urgently. In this paper, we address these problems by generating images that
+optimize a classifier-derived objective using a framework for guided image
+generation. We analyze the decisions of image classifiers by visual
+counterfactual explanations (VCEs), detection of systematic mistakes by
+analyzing images where classifiers maximally disagree, and visualization of
+neurons and spurious features. In this way, we validate existing observations,
+e.g. the shape bias of adversarially robust models, as well as novel failure
+modes, e.g. systematic errors of zero-shot CLIP classifiers. Moreover, our VCEs
+outperform previous work while being more versatile.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models,Fei Deng · Qifei Wang · Wei Wei · Tingbo Hou · Matthias Grundmann, ,https://arxiv.org/abs/2402.08714,,2402.08714.pdf,PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models,"Reward finetuning has emerged as a promising approach to aligning foundation
+models with downstream objectives. Remarkable success has been achieved in the
+language domain by using reinforcement learning (RL) to maximize rewards that
+reflect human preference. However, in the vision domain, existing RL-based
+reward finetuning methods are limited by their instability in large-scale
+training, rendering them incapable of generalizing to complex, unseen prompts.
+In this paper, we propose Proximal Reward Difference Prediction (PRDP),
+enabling stable black-box reward finetuning for diffusion models for the first
+time on large-scale prompt datasets with over 100K prompts. Our key innovation
+is the Reward Difference Prediction (RDP) objective that has the same optimal
+solution as the RL objective while enjoying better training stability.
+Specifically, the RDP objective is a supervised regression objective that tasks
+the diffusion model with predicting the reward difference of generated image
+pairs from their denoising trajectories. We theoretically prove that the
+diffusion model that obtains perfect reward difference prediction is exactly
+the maximizer of the RL objective. We further develop an online algorithm with
+proximal updates to stably optimize the RDP objective. In experiments, we
+demonstrate that PRDP can match the reward maximization ability of
+well-established RL-based methods in small-scale training. Furthermore, through
+large-scale training on text prompts from the Human Preference Dataset v2 and
+the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a
+diverse set of complex, unseen prompts whereas RL-based methods completely
+fail.",cs.LG,"['cs.LG', 'cs.AI']"
+SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology,Saarthak Kapse · Pushpak Pati · Srijan Das · Jingwei Zhang · Chao Chen · Maria Vakalopoulou · Joel Saltz · Dimitris Samaras · Rajarsi Gupta · Prateek Prasanna,https://github.com/bmi-imaginelab/SI-MIL,https://arxiv.org/abs/2312.15010,,2312.15010.pdf,SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology,"Introducing interpretability and reasoning into Multiple Instance Learning
+(MIL) methods for Whole Slide Image (WSI) analysis is challenging, given the
+complexity of gigapixel slides. Traditionally, MIL interpretability is limited
+to identifying salient regions deemed pertinent for downstream tasks, offering
+little insight to the end-user (pathologist) regarding the rationale behind
+these selections. To address this, we propose Self-Interpretable MIL (SI-MIL),
+a method intrinsically designed for interpretability from the very outset.
+SI-MIL employs a deep MIL framework to guide an interpretable branch grounded
+on handcrafted pathological features, facilitating linear predictions. Beyond
+identifying salient regions, SI-MIL uniquely provides feature-level
+interpretations rooted in pathological insights for WSIs. Notably, SI-MIL, with
+its linear prediction constraints, challenges the prevalent myth of an
+inevitable trade-off between model interpretability and performance,
+demonstrating competitive results compared to state-of-the-art methods on
+WSI-level prediction tasks across three cancer types. In addition, we
+thoroughly benchmark the local and global-interpretability of SI-MIL in terms
+of statistical analysis, a domain expert study, and desiderata of
+interpretability, namely, user-friendliness and faithfulness.",cs.CV,['cs.CV']
+Diffusion Models Without Attention,Jing Nathan Yan · Jiatao Gu · Alexander Rush, ,,https://www.semanticscholar.org/paper/Diffusion-Models-Without-Attention-Yan-Gu/31245344a6eb6cd897a71928dc4b174ab75e4070,,,,,nan
+DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models,Nastaran Saadati · Minh Pham · Nasla Saleem · Joshua R. Waite · Aditya Balu · Zhanhong Jiang · Chinmay Hegde · Soumik Sarkar, ,https://arxiv.org/abs/2404.08079,,2404.08079.pdf,DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models,"Recent advances in decentralized deep learning algorithms have demonstrated
+cutting-edge performance on various tasks with large pre-trained models.
+However, a pivotal prerequisite for achieving this level of competitiveness is
+the significant communication and computation overheads when updating these
+models, which prohibits the applications of them to real-world scenarios. To
+address this issue, drawing inspiration from advanced model merging techniques
+without requiring additional training, we introduce the Decentralized Iterative
+Merging-And-Training (DIMAT) paradigm--a novel decentralized deep learning
+framework. Within DIMAT, each agent is trained on their local data and
+periodically merged with their neighboring agents using advanced model merging
+techniques like activation matching until convergence is achieved. DIMAT
+provably converges with the best available rate for nonconvex functions with
+various first-order methods, while yielding tighter error bounds compared to
+the popular existing approaches. We conduct a comprehensive empirical analysis
+to validate DIMAT's superiority over baselines across diverse computer vision
+tasks sourced from multiple datasets. Empirical results validate our
+theoretical claims by showing that DIMAT attains faster and higher initial gain
+in accuracy with independent and identically distributed (IID) and non-IID
+data, incurring lower communication overhead. This DIMAT paradigm presents a
+new opportunity for the future decentralized learning, enhancing its
+adaptability to real-world with sparse and light-weight communication and
+computation.",cs.LG,"['cs.LG', 'cs.CV', 'math.OC']"
+Mask4Align: Aligned Entity Prompting with Color Masks for Multi-Entity Localization Problem,Haoquan Zhang · Ronggang Huang · Yi Xie · Huaidong Zhang, ,https://arxiv.org/abs/2310.05364,,2310.05364.pdf,Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths,"The objective of Entity Alignment (EA) is to identify equivalent entity pairs
+from multiple Knowledge Graphs (KGs) and create a more comprehensive and
+unified KG. The majority of EA methods have primarily focused on the structural
+modality of KGs, lacking exploration of multi-modal information. A few
+multi-modal EA methods have made good attempts in this field. Still, they have
+two shortcomings: (1) inconsistent and inefficient modality modeling that
+designs complex and distinct models for each modality; (2) ineffective modality
+fusion due to the heterogeneous nature of modalities in EA. To tackle these
+challenges, we propose PathFusion, consisting of two main components: (1) MSP,
+a unified modeling approach that simplifies the alignment process by
+constructing paths connecting entities and modality nodes to represent multiple
+modalities; (2) IRF, an iterative fusion method that effectively combines
+information from different modalities using the path as an information carrier.
+Experimental results on real-world datasets demonstrate the superiority of
+PathFusion over state-of-the-art methods, with 22.4%-28.9% absolute improvement
+on Hits@1, and 0.194-0.245 absolute improvement on MRR.",cs.CL,"['cs.CL', 'cs.AI']"
+Hearing Anything Anywhere,Mason Wang · Ryosuke Sawata · Samuel Clarke · Ruohan Gao · Shangzhe Wu · Jiajun Wu, ,,https://zenodo.org/records/11195833,,,,,nan
+OmniSDF: Scene Reconstruction using Omnidirectional Signed Distance Functions and Adaptive Binoctrees,Hakyeong Kim · Andreas Meuleman · Hyeonjoong Jang · James Tompkin · Min H. Kim,https://vclab.kaist.ac.kr/cvpr2024p2/index.html,https://arxiv.org/abs/2404.00678,,2404.00678.pdf,OmniSDF: Scene Reconstruction using Omnidirectional Signed Distance Functions and Adaptive Binoctrees,"We present a method to reconstruct indoor and outdoor static scene geometry
+and appearance from an omnidirectional video moving in a small circular sweep.
+This setting is challenging because of the small baseline and large depth
+ranges, making it difficult to find ray crossings. To better constrain the
+optimization, we estimate geometry as a signed distance field within a
+spherical binoctree data structure and use a complementary efficient tree
+traversal strategy based on a breadth-first search for sampling. Unlike regular
+grids or trees, the shape of this structure well-matches the camera setting,
+creating a better memory-quality trade-off. From an initial depth estimate, the
+binoctree is adaptively subdivided throughout the optimization; previous
+methods use a fixed depth that leaves the scene undersampled. In comparison
+with three neural optimization methods and two non-neural methods, ours shows
+decreased geometry error on average, especially in a detailed scene, while
+significantly reducing the required number of voxels to represent such details.",cs.CV,"['cs.CV', 'cs.GR']"
+Understanding and Improving Source-free Domain Adaptation from a Theoretical Perspective,Yu Mitsuzumi · Akisato Kimura · Hisashi Kashima, ,,https://akisatok.tech/news/a-paper-accepted-to-cvpr2024,,,,,nan
+BANF: Band-limited Neural Fields for Levels of Detail Reconstruction,Ahan Shabanov · Shrisudhan Govindarajan · Cody Reading · Leili Goli · Daniel Rebain · Kwang Moo Yi · Andrea Tagliasacchi, ,https://arxiv.org/abs/2404.13024,,2404.13024.pdf,BANF: Band-limited Neural Fields for Levels of Detail Reconstruction,"Largely due to their implicit nature, neural fields lack a direct mechanism
+for filtering, as Fourier analysis from discrete signal processing is not
+directly applicable to these representations. Effective filtering of neural
+fields is critical to enable level-of-detail processing in downstream
+applications, and support operations that involve sampling the field on regular
+grids (e.g. marching cubes). Existing methods that attempt to decompose neural
+fields in the frequency domain either resort to heuristics or require extensive
+modifications to the neural field architecture. We show that via a simple
+modification, one can obtain neural fields that are low-pass filtered, and in
+turn show how this can be exploited to obtain a frequency decomposition of the
+entire signal. We demonstrate the validity of our technique by investigating
+level-of-detail reconstruction, and showing how coarser representations can be
+computed effectively.",cs.CV,"['cs.CV', 'eess.IV']"
+PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution,Honghao Chen · Xiangxiang Chu · Renyongjian · Xin Zhao · Kaiqi Huang, ,https://arxiv.org/abs/2403.07589,,2403.07589.pdf,PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution,"Recently, some large kernel convnets strike back with appealing performance
+and efficiency. However, given the square complexity of convolution, scaling up
+kernels can bring about an enormous amount of parameters and the proliferated
+parameters can induce severe optimization problem. Due to these issues, current
+CNNs compromise to scale up to 51x51 in the form of stripe convolution (i.e.,
+51x5 + 5x51) and start to saturate as the kernel size continues growing. In
+this paper, we delve into addressing these vital issues and explore whether we
+can continue scaling up kernels for more performance gains. Inspired by human
+vision, we propose a human-like peripheral convolution that efficiently reduces
+over 90% parameter count of dense grid convolution through parameter sharing,
+and manage to scale up kernel size to extremely large. Our peripheral
+convolution behaves highly similar to human, reducing the complexity of
+convolution from O(K^2) to O(logK) without backfiring performance. Built on
+this, we propose Parameter-efficient Large Kernel Network (PeLK). Our PeLK
+outperforms modern vision Transformers and ConvNet architectures like Swin,
+ConvNeXt, RepLKNet and SLaK on various vision tasks including ImageNet
+classification, semantic segmentation on ADE20K and object detection on MS
+COCO. For the first time, we successfully scale up the kernel size of CNNs to
+an unprecedented 101x101 and demonstrate consistent improvements.",cs.CV,['cs.CV']
+Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos,Mehmet Saygin Seyfioglu · Wisdom Ikezogwo · Fatemeh Ghezloo · Ranjay Krishna · Linda Shapiro, ,https://arxiv.org/abs/2312.04746,,2312.04746.pdf,Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos,"Diagnosis in histopathology requires a global whole slide images (WSIs)
+analysis, requiring pathologists to compound evidence from different WSI
+patches. The gigapixel scale of WSIs poses a challenge for histopathology
+multi-modal models. Training multi-model models for histopathology requires
+instruction tuning datasets, which currently contain information for individual
+image patches, without a spatial grounding of the concepts within each patch
+and without a wider view of the WSI. Therefore, they lack sufficient diagnostic
+capacity for histopathology. To bridge this gap, we introduce Quilt-Instruct, a
+large-scale dataset of 107,131 histopathology-specific instruction
+question/answer pairs, grounded within diagnostically relevant image patches
+that make up the WSI. Our dataset is collected by leveraging educational
+histopathology videos from YouTube, which provides spatial localization of
+narrations by automatically extracting the narrators' cursor positions.
+Quilt-Instruct supports contextual reasoning by extracting diagnosis and
+supporting facts from the entire WSI. Using Quilt-Instruct, we train
+Quilt-LLaVA, which can reason beyond the given single image patch, enabling
+diagnostic reasoning across patches. To evaluate Quilt-LLaVA, we propose a
+comprehensive evaluation dataset created from 985 images and 1283
+human-generated question-answers. We also thoroughly evaluate Quilt-LLaVA using
+public histopathology datasets, where Quilt-LLaVA significantly outperforms
+SOTA by over 10% on relative GPT-4 score and 4% and 9% on open and closed set
+VQA. Our code, data, and model are publicly accessible at
+quilt-llava.github.io.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation,Javier Tirado-Garín · Javier Civera,https://github.com/javrtg/C2P,https://arxiv.org/abs/2312.05995,,2312.05995.pdf,From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation,"Estimating the relative camera pose from $n \geq 5$ correspondences between
+two calibrated views is a fundamental task in computer vision. This process
+typically involves two stages: 1) estimating the essential matrix between the
+views, and 2) disambiguating among the four candidate relative poses that
+satisfy the epipolar geometry. In this paper, we demonstrate a novel approach
+that, for the first time, bypasses the second stage. Specifically, we show that
+it is possible to directly estimate the correct relative camera pose from
+correspondences without needing a post-processing step to enforce the
+cheirality constraint on the correspondences. Building on recent advances in
+certifiable non-minimal optimization, we frame the relative pose estimation as
+a Quadratically Constrained Quadratic Program (QCQP). By applying the
+appropriate constraints, we ensure the estimation of a camera pose that
+corresponds to a valid 3D geometry and that is globally optimal when certified.
+We validate our method through exhaustive synthetic and real-world experiments,
+confirming the efficacy, efficiency and accuracy of the proposed approach. Code
+is available at https://github.com/javrtg/C2P.",cs.CV,['cs.CV']
+Diffusion-based Blind Text Image Super-Resolution,Yuzhe Zhang · jiawei zhang · Hao Li · Zhouxia Wang · Luwei Hou · Dongqing Zou · Liheng Bian, ,https://arxiv.org/abs/2312.08886,,2312.08886.pdf,Diffusion-based Blind Text Image Super-Resolution,"Recovering degraded low-resolution text images is challenging, especially for
+Chinese text images with complex strokes and severe degradation in real-world
+scenarios. Ensuring both text fidelity and style realness is crucial for
+high-quality text image super-resolution. Recently, diffusion models have
+achieved great success in natural image synthesis and restoration due to their
+powerful data distribution modeling abilities and data generation capabilities.
+In this work, we propose an Image Diffusion Model (IDM) to restore text images
+with realistic styles. For diffusion models, they are not only suitable for
+modeling realistic image distribution but also appropriate for learning text
+distribution. Since text prior is important to guarantee the correctness of the
+restored text structure according to existing arts, we also propose a Text
+Diffusion Model (TDM) for text recognition which can guide IDM to generate text
+images with correct structures. We further propose a Mixture of Multi-modality
+module (MoM) to make these two diffusion models cooperate with each other in
+all the diffusion steps. Extensive experiments on synthetic and real-world
+datasets demonstrate that our Diffusion-based Blind Text Image Super-Resolution
+(DiffTSR) can restore text images with more accurate text structures as well as
+more realistic appearances simultaneously.",cs.CV,['cs.CV']
+Language-driven Grasp Detection,An Dinh Vuong · Minh Nhat VU · Baoru Huang · Nghia Nguyen · Hieu Le · Thieu Vo · Thieu Vo · Anh Nguyen,https://airvlab.github.io/grasp-anything/,https://ar5iv.labs.arxiv.org/html/2309.09818,,2309.09818.pdf,Grasp-Anything: Large-scale Grasp Dataset from Foundation Models,"Foundation models such as ChatGPT have made significant strides in robotic
+tasks due to their universal representation of real-world domains. In this
+paper, we leverage foundation models to tackle grasp detection, a persistent
+challenge in robotics with broad industrial applications. Despite numerous
+grasp datasets, their object diversity remains limited compared to real-world
+figures. Fortunately, foundation models possess an extensive repository of
+real-world knowledge, including objects we encounter in our daily lives. As a
+consequence, a promising solution to the limited representation in previous
+grasp datasets is to harness the universal knowledge embedded in these
+foundation models. We present Grasp-Anything, a new large-scale grasp dataset
+synthesized from foundation models to implement this solution. Grasp-Anything
+excels in diversity and magnitude, boasting 1M samples with text descriptions
+and more than 3M objects, surpassing prior datasets. Empirically, we show that
+Grasp-Anything successfully facilitates zero-shot grasp detection on
+vision-based tasks and real-world robotic experiments. Our dataset and code are
+available at https://grasp-anything-2023.github.io.",cs.RO,"['cs.RO', 'cs.CV']"
+Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs,Shengbang Tong · Zhuang Liu · Zhuang Liu · Yuexiang Zhai · Yi Ma · Yann LeCun · Saining Xie, ,http://export.arxiv.org/abs/2401.06209,,2401.06209.pdf,Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs,"Is vision good enough for language? Recent advancements in multimodal models
+primarily stem from the powerful reasoning abilities of large language models
+(LLMs). However, the visual component typically depends only on the
+instance-level contrastive language-image pre-training (CLIP). Our research
+reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still
+exhibit systematic shortcomings. To understand the roots of these errors, we
+explore the gap between the visual embedding space of CLIP and vision-only
+self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP
+perceives as similar despite their clear visual differences. With these pairs,
+we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes
+areas where state-of-the-art systems, including GPT-4V, struggle with
+straightforward questions across nine basic visual patterns, often providing
+incorrect answers and hallucinated explanations. We further evaluate various
+CLIP-based vision-and-language models and found a notable correlation between
+visual patterns that challenge CLIP models and those problematic for multimodal
+LLMs. As an initial effort to address these issues, we propose a Mixture of
+Features (MoF) approach, demonstrating that integrating vision self-supervised
+learning features with MLLMs can significantly enhance their visual grounding
+capabilities. Together, our research suggests visual representation learning
+remains an open challenge, and accurate visual grounding is crucial for future
+successful multimodal systems.",cs.CV,['cs.CV']
+Evaluating Transferability in Retrieval Tasks: An Approach Using MMD and Kernel Methods,Mengyu Dai · Amir Hossein Raffiee · Aashish Jain · Joshua Correa, ,,https://ieeexplore.ieee.org/document/10452779,,,,,nan
+Alpha Invariance: On Inverse Scaling Between Distance and Volume Density in Neural Radiance Fields,Joshua Ahn · Haochen Wang · Raymond A. Yeh · Greg Shakhnarovich,https://pals.ttic.edu/p/alpha-invariance,https://arxiv.org/abs/2404.02155,,2404.02155.pdf,Alpha Invariance: On Inverse Scaling Between Distance and Volume Density in Neural Radiance Fields,"Scale-ambiguity in 3D scene dimensions leads to magnitude-ambiguity of
+volumetric densities in neural radiance fields, i.e., the densities double when
+scene size is halved, and vice versa. We call this property alpha invariance.
+For NeRFs to better maintain alpha invariance, we recommend 1) parameterizing
+both distance and volume densities in log space, and 2) a
+discretization-agnostic initialization strategy to guarantee high ray
+transmittance. We revisit a few popular radiance field models and find that
+these systems use various heuristics to deal with issues arising from scene
+scaling. We test their behaviors and show our recipe to be more robust.",cs.CV,['cs.CV']
+Prompt-Driven Referring Image Segmentation with Instance Contrasting,Chao Shang · Zichen Song · Heqian Qiu · Lanxiao Wang · Fanman Meng · Hongliang Li, ,https://arxiv.org/abs/2310.19721,,2310.19721.pdf,Promise:Prompt-driven 3D Medical Image Segmentation Using Pretrained Image Foundation Models,"To address prevalent issues in medical imaging, such as data acquisition
+challenges and label availability, transfer learning from natural to medical
+image domains serves as a viable strategy to produce reliable segmentation
+results. However, several existing barriers between domains need to be broken
+down, including addressing contrast discrepancies, managing anatomical
+variability, and adapting 2D pretrained models for 3D segmentation tasks. In
+this paper, we propose ProMISe,a prompt-driven 3D medical image segmentation
+model using only a single point prompt to leverage knowledge from a pretrained
+2D image foundation model. In particular, we use the pretrained vision
+transformer from the Segment Anything Model (SAM) and integrate lightweight
+adapters to extract depth-related (3D) spatial context without updating the
+pretrained weights. For robust results, a hybrid network with complementary
+encoders is designed, and a boundary-aware loss is proposed to achieve precise
+boundaries. We evaluate our model on two public datasets for colon and pancreas
+tumor segmentations, respectively. Compared to the state-of-the-art
+segmentation methods with and without prompt engineering, our proposed method
+achieves superior performance. The code is publicly available at
+https://github.com/MedICL-VU/ProMISe.",eess.IV,"['eess.IV', 'cs.CV']"
+DreamVideo: Composing Your Dream Videos with Customized Subject and Motion,Yujie Wei · Shiwei Zhang · Zhiwu Qing · Hangjie Yuan · Zhiheng Liu · Yu Liu · Yingya Zhang · Jingren Zhou · Hongming Shan,https://dreamvideo-t2v.github.io/,https://arxiv.org/abs/2312.04433,,2312.04433.pdf,DreamVideo: Composing Your Dream Videos with Customized Subject and Motion,"Customized generation using diffusion models has made impressive progress in
+image generation, but remains unsatisfactory in the challenging video
+generation task, as it requires the controllability of both subjects and
+motions. To that end, we present DreamVideo, a novel approach to generating
+personalized videos from a few static images of the desired subject and a few
+videos of target motion. DreamVideo decouples this task into two stages,
+subject learning and motion learning, by leveraging a pre-trained video
+diffusion model. The subject learning aims to accurately capture the fine
+appearance of the subject from provided images, which is achieved by combining
+textual inversion and fine-tuning of our carefully designed identity adapter.
+In motion learning, we architect a motion adapter and fine-tune it on the given
+videos to effectively model the target motion pattern. Combining these two
+lightweight and efficient adapters allows for flexible customization of any
+subject with any motion. Extensive experimental results demonstrate the
+superior performance of our DreamVideo over the state-of-the-art methods for
+customized video generation. Our project page is at
+https://dreamvideo-t2v.github.io.",cs.CV,['cs.CV']
+Multi-Attribute Interactions Matter for 3D Visual Grounding,Can Xu · Yuehui Han · Rui Xu · Le Hui · Jin Xie · Jian Yang, ,https://arxiv.org/abs/2404.19696,,2404.19696.pdf,Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners,"3D visual grounding is a challenging task that often requires direct and
+dense supervision, notably the semantic label for each object in the scene. In
+this paper, we instead study the naturally supervised setting that learns from
+only 3D scene and QA pairs, where prior works underperform. We propose the
+Language-Regularized Concept Learner (LARC), which uses constraints from
+language as regularization to significantly improve the accuracy of
+neuro-symbolic concept learners in the naturally supervised setting. Our
+approach is based on two core insights: the first is that language constraints
+(e.g., a word's relation to another) can serve as effective regularization for
+structured representations in neuro-symbolic models; the second is that we can
+query large language models to distill such constraints from language
+properties. We show that LARC improves performance of prior works in naturally
+supervised 3D visual grounding, and demonstrates a wide range of 3D visual
+reasoning capabilities-from zero-shot composition, to data efficiency and
+transferability. Our method represents a promising step towards regularizing
+structured visual reasoning frameworks with language-based priors, for learning
+in settings without dense supervision.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering,Vivek Gopalakrishnan · Neel Dey · Polina Golland, ,https://arxiv.org/abs/2312.06358,,2312.06358.pdf,Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering,"Surgical decisions are informed by aligning rapid portable 2D intraoperative
+images (e.g., X-rays) to a high-fidelity 3D preoperative reference scan (e.g.,
+CT). 2D/3D image registration often fails in practice: conventional
+optimization methods are prohibitively slow and susceptible to local minima,
+while neural networks trained on small datasets fail on new patients or require
+impractical landmark supervision. We present DiffPose, a self-supervised
+approach that leverages patient-specific simulation and differentiable
+physics-based rendering to achieve accurate 2D/3D registration without relying
+on manually labeled data. Preoperatively, a CNN is trained to regress the pose
+of a randomly oriented synthetic X-ray rendered from the preoperative CT. The
+CNN then initializes rapid intraoperative test-time optimization that uses the
+differentiable X-ray renderer to refine the solution. Our work further proposes
+several geometrically principled methods for sampling camera poses from
+$\mathbf{SE}(3)$, for sparse differentiable rendering, and for driving
+registration in the tangent space $\mathfrak{se}(3)$ with geodesic and
+multiscale locality-sensitive losses. DiffPose achieves sub-millimeter accuracy
+across surgical datasets at intraoperative speeds, improving upon existing
+unsupervised methods by an order of magnitude and even outperforming supervised
+baselines. Our code is available at https://github.com/eigenvivek/DiffPose.",cs.CV,['cs.CV']
+MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant,Chenlu Zhan · Gaoang Wang · Yu LIN · Hongwei Wang · Jian Wu, ,https://arxiv.org/abs/2403.04290,,2403.04290.pdf,MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant,"Medical generative models, acknowledged for their high-quality sample
+generation ability, have accelerated the fast growth of medical applications.
+However, recent works concentrate on separate medical generation models for
+distinct medical tasks and are restricted to inadequate medical multi-modal
+knowledge, constraining medical comprehensive diagnosis. In this paper, we
+propose MedM2G, a Medical Multi-Modal Generative framework, with the key
+innovation to align, extract, and generate medical multi-modal within a unified
+model. Extending beyond single or two medical modalities, we efficiently align
+medical multi-modal through the central alignment approach in the unified
+space. Significantly, our framework extracts valuable clinical knowledge by
+preserving the medical visual invariant of each imaging modal, thereby
+enhancing specific medical information for multi-modal generation. By
+conditioning the adaptive cross-guided parameters into the multi-flow diffusion
+framework, our model promotes flexible interactions among medical multi-modal
+for generation. MedM2G is the first medical generative model that unifies
+medical generation tasks of text-to-image, image-to-text, and unified
+generation of medical modalities (CT, MRI, X-ray). It performs 5 medical
+generation tasks across 10 datasets, consistently outperforming various
+state-of-the-art works.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG']"
+SeD: Semantic-Aware Discriminator for Image Super-Resolution,Bingchen Li · Xin Li · Hanxin Zhu · YEYING JIN · Ruoyu Feng · Zhizheng Zhang · Zhibo Chen, ,https://arxiv.org/abs/2402.19387,,2402.19387.pdf,SeD: Semantic-Aware Discriminator for Image Super-Resolution,"Generative Adversarial Networks (GANs) have been widely used to recover vivid
+textures in image super-resolution (SR) tasks. In particular, one discriminator
+is utilized to enable the SR network to learn the distribution of real-world
+high-quality images in an adversarial training manner. However, the
+distribution learning is overly coarse-grained, which is susceptible to virtual
+textures and causes counter-intuitive generation results. To mitigate this, we
+propose the simple and effective Semantic-aware Discriminator (denoted as SeD),
+which encourages the SR network to learn the fine-grained distributions by
+introducing the semantics of images as a condition. Concretely, we aim to
+excavate the semantics of images from a well-trained semantic extractor. Under
+different semantics, the discriminator is able to distinguish the real-fake
+images individually and adaptively, which guides the SR network to learn the
+more fine-grained semantic-aware textures. To obtain accurate and abundant
+semantics, we take full advantage of recently popular pretrained vision models
+(PVMs) with extensive datasets, and then incorporate its semantic features into
+the discriminator through a well-designed spatial cross-attention module. In
+this way, our proposed semantic-aware discriminator empowered the SR network to
+produce more photo-realistic and pleasing images. Extensive experiments on two
+typical tasks, i.e., SR and Real SR have demonstrated the effectiveness of our
+proposed methods.",eess.IV,"['eess.IV', 'cs.CV']"
+Taming Self-Training for Open-Vocabulary Object Detection,Shiyu Zhao · Samuel Schulter · Long Zhao · Zhixing Zhang · Vijay Kumar BG · Yumin Suh · Manmohan Chandraker · Dimitris N. Metaxas, ,https://arxiv.org/abs/2308.06412,,2308.06412.pdf,Taming Self-Training for Open-Vocabulary Object Detection,"Recent studies have shown promising performance in open-vocabulary object
+detection (OVD) by utilizing pseudo labels (PLs) from pretrained vision and
+language models (VLMs). However, teacher-student self-training, a powerful and
+widely used paradigm to leverage PLs, is rarely explored for OVD. This work
+identifies two challenges of using self-training in OVD: noisy PLs from VLMs
+and frequent distribution changes of PLs. To address these challenges, we
+propose SAS-Det that tames self-training for OVD from two key perspectives.
+First, we present a split-and-fusion (SAF) head that splits a standard
+detection into an open-branch and a closed-branch. This design can reduce noisy
+supervision from pseudo boxes. Moreover, the two branches learn complementary
+knowledge from different training data, significantly enhancing performance
+when fused together. Second, in our view, unlike in closed-set tasks, the PL
+distributions in OVD are solely determined by the teacher model. We introduce a
+periodic update strategy to decrease the number of updates to the teacher,
+thereby decreasing the frequency of changes in PL distributions, which
+stabilizes the training process. Extensive experiments demonstrate SAS-Det is
+both efficient and effective. SAS-Det outperforms recent models of the same
+scale by a clear margin and achieves 37.4 AP50 and 29.1 APr on novel categories
+of the COCO and LVIS benchmarks, respectively. Code is available at
+\url{https://github.com/xiaofeng94/SAS-Det}.",cs.CV,['cs.CV']
+Edit One for All: Interactive Batch Image Editing,Thao Nguyen · Utkarsh Ojha · Yuheng Li · Haotian Liu · Yong Jae Lee,https://thaoshibe.github.io/edit-one-for-all,https://arxiv.org/abs/2401.10219,,2401.10219.pdf,Edit One for All: Interactive Batch Image Editing,"In recent years, image editing has advanced remarkably. With increased human
+control, it is now possible to edit an image in a plethora of ways; from
+specifying in text what we want to change, to straight up dragging the contents
+of the image in an interactive point-based manner. However, most of the focus
+has remained on editing single images at a time. Whether and how we can
+simultaneously edit large batches of images has remained understudied. With the
+goal of minimizing human supervision in the editing process, this paper
+presents a novel method for interactive batch image editing using StyleGAN as
+the medium. Given an edit specified by users in an example image (e.g., make
+the face frontal), our method can automatically transfer that edit to other
+test images, so that regardless of their initial state (pose), they all arrive
+at the same final state (e.g., all facing front). Extensive experiments
+demonstrate that edits performed using our method have similar visual quality
+to existing single-image-editing methods, while having more visual consistency
+and saving significant time and human effort.",cs.CV,['cs.CV']
+Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning,Desai Xie · Jiahao Li · Hao Tan · Xin Sun · Zhixin Shu · Yi Zhou · Sai Bi · Soren Pirk · Soeren Pirk · ARIE KAUFMAN,https://desaixie.github.io/carve-3d/,https://arxiv.org/abs/2312.13980v1,,2312.13980v1.pdf,Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning,"Recent advancements in the text-to-3D task leverage finetuned text-to-image
+diffusion models to generate multi-view images, followed by NeRF
+reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
+suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
+training longer with SFT improves consistency, it also causes distribution
+shift, which reduces diversity and realistic details. We argue that the SFT of
+multi-view diffusion models resembles the instruction finetuning stage of the
+LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
+Essentially, RLFT methods optimize models beyond their SFT data distribution by
+using their own outputs, effectively mitigating distribution shift. To this
+end, we introduce Carve3D, a RLFT method coupled with the Multi-view
+Reconstruction Consistency (MRC) metric, to improve the consistency of
+multi-view diffusion models. To compute MRC on a set of multi-view images, we
+compare them with their corresponding renderings of the reconstructed NeRF at
+the same viewpoints. We validate the robustness of MRC with extensive
+experiments conducted under controlled inconsistency levels. We enhance the
+base RLFT algorithm to stabilize the training process, reduce distribution
+shift, and identify scaling laws. Through qualitative and quantitative
+experiments, along with a user study, we demonstrate Carve3D's improved
+multi-view consistency, the resulting superior NeRF reconstruction quality, and
+minimal distribution shift compared to longer SFT. Project webpage:
+https://desaixie.github.io/carve-3d.",cs.CV,"['cs.CV', 'cs.LG']"
+Density-Guided Semi-Supervised 3D Semantic Segmentation with Dual-Space Hardness Sampling,Jianan Li · Qiulei Dong, ,https://arxiv.org/abs/2306.08045,,2306.08045.pdf,Efficient 3D Semantic Segmentation with Superpoint Transformer,"We introduce a novel superpoint-based transformer architecture for efficient
+semantic segmentation of large-scale 3D scenes. Our method incorporates a fast
+algorithm to partition point clouds into a hierarchical superpoint structure,
+which makes our preprocessing 7 times faster than existing superpoint-based
+approaches. Additionally, we leverage a self-attention mechanism to capture the
+relationships between superpoints at multiple scales, leading to
+state-of-the-art performance on three challenging benchmark datasets: S3DIS
+(76.0% mIoU 6-fold validation), KITTI-360 (63.5% on Val), and DALES (79.6%).
+With only 212k parameters, our approach is up to 200 times more compact than
+other state-of-the-art models while maintaining similar performance.
+Furthermore, our model can be trained on a single GPU in 3 hours for a fold of
+the S3DIS dataset, which is 7x to 70x fewer GPU-hours than the best-performing
+methods. Our code and models are accessible at
+github.com/drprojects/superpoint_transformer.",cs.CV,['cs.CV']
+Unifying Automatic and Interactive Matting with Pretrained ViTs,Zixuan Ye · Wenze Liu · He Guo · Yujia Liang · Chaoyi Hong · Hao Lu · Zhiguo Cao, ,,https://dl.acm.org/doi/10.1016/j.inffus.2023.102091,,,,,nan
+S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes,Xingyi Li · Zhiguo Cao · Yizheng Wu · Kewei Wang · Ke Xian · Zhe Wang · Guosheng Lin, ,https://arxiv.org/abs/2403.06205,,2403.06205.pdf,S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes,"Current 3D stylization methods often assume static scenes, which violates the
+dynamic nature of our real world. To address this limitation, we present
+S-DyRF, a reference-based spatio-temporal stylization method for dynamic neural
+radiance fields. However, stylizing dynamic 3D scenes is inherently challenging
+due to the limited availability of stylized reference images along the temporal
+axis. Our key insight lies in introducing additional temporal cues besides the
+provided reference. To this end, we generate temporal pseudo-references from
+the given stylized reference. These pseudo-references facilitate the
+propagation of style information from the reference to the entire dynamic 3D
+scene. For coarse style transfer, we enforce novel views and times to mimic the
+style details present in pseudo-references at the feature level. To preserve
+high-frequency details, we create a collection of stylized temporal pseudo-rays
+from temporal pseudo-references. These pseudo-rays serve as detailed and
+explicit stylization guidance for achieving fine style transfer. Experiments on
+both synthetic and real-world datasets demonstrate that our method yields
+plausible stylized results of space-time view synthesis on dynamic 3D scenes.",cs.CV,['cs.CV']
+Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis,Simon Niedermayr · Josef Stumpfegger · rüdiger westermann,https://keksboter.github.io/c3dgs/,https://arxiv.org/abs/2401.02436,,2401.02436.pdf,Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis,"Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian
+splat representation has been introduced for novel view synthesis from sparse
+image sets. Making such representations suitable for applications like network
+streaming and rendering on low-power devices requires significantly reduced
+memory consumption as well as improved rendering efficiency. We propose a
+compressed 3D Gaussian splat representation that utilizes sensitivity-aware
+vector clustering with quantization-aware training to compress directional
+colors and Gaussian parameters. The learned codebooks have low bitrates and
+achieve a compression rate of up to $31\times$ on real-world scenes with only
+minimal degradation of visual quality. We demonstrate that the compressed splat
+representation can be efficiently rendered with hardware rasterization on
+lightweight GPUs at up to $4\times$ higher framerates than reported via an
+optimized GPU compute pipeline. Extensive experiments across multiple datasets
+demonstrate the robustness and rendering speed of the proposed approach.",cs.CV,"['cs.CV', 'cs.GR']"
+ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images,Nicolas Bourriez · Ihab Bendidi · Cohen Ethan · Gabriel Watkinson · Maxime Sanchez · Guillaume Bollot · Auguste Genovesio, ,https://arxiv.org/abs/2311.15264,,2311.15264.pdf,ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images,"Unlike color photography images, which are consistently encoded into RGB
+channels, biological images encompass various modalities, where the type of
+microscopy and the meaning of each channel varies with each experiment.
+Importantly, the number of channels can range from one to a dozen and their
+correlation is often comparatively much lower than RGB, as each of them brings
+specific information content. This aspect is largely overlooked by methods
+designed out of the bioimage field, and current solutions mostly focus on
+intra-channel spatial attention, often ignoring the relationship between
+channels, yet crucial in most biological applications. Importantly, the
+variable channel type and count prevent the projection of several experiments
+to a unified representation for large scale pre-training. In this study, we
+propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture
+employing an Inter-Channel Attention mechanism on images with an arbitrary
+number, order and type of channels. We also introduce IDRCell100k, a bioimage
+dataset with a rich set of 79 experiments covering 7 microscope modalities,
+with a multitude of channel types, and counts varying from 1 to 10 per
+experiment. Our architecture, trained in a self-supervised manner, outperforms
+existing approaches in several biologically relevant downstream tasks.
+Additionally, it can be used to bridge the gap for the first time between
+assays with different microscopes, channel numbers or types by embedding
+various image and experimental modalities into a unified biological image
+representation. The latter should facilitate interdisciplinary studies and pave
+the way for better adoption of deep learning in biological image-based
+analyses. Code and Data available at https://github.com/nicoboou/chadavit.",cs.CV,"['cs.CV', 'cs.LG']"
+Generating Enhanced Negatives for Training Language-Based Object Detectors,Shiyu Zhao · Long Zhao · Vijay Kumar BG · Yumin Suh · Dimitris N. Metaxas · Manmohan Chandraker · Samuel Schulter, ,https://arxiv.org/abs/2401.00094,,2401.00094.pdf,Generating Enhanced Negatives for Training Language-Based Object Detectors,"The recent progress in language-based open-vocabulary object detection can be
+largely attributed to finding better ways of leveraging large-scale data with
+free-form text annotations. Training such models with a discriminative
+objective function has proven successful, but requires good positive and
+negative samples. However, the free-form nature and the open vocabulary of
+object descriptions make the space of negatives extremely large. Prior works
+randomly sample negatives or use rule-based techniques to build them. In
+contrast, we propose to leverage the vast knowledge built into modern
+generative models to automatically build negatives that are more relevant to
+the original data. Specifically, we use large-language-models to generate
+negative text descriptions, and text-to-image diffusion models to also generate
+corresponding negative images. Our experimental analysis confirms the relevance
+of the generated negative data, and its use in language-based detectors
+improves performance on two complex benchmarks. Code is available at
+\url{https://github.com/xiaofeng94/Gen-Enhanced-Negs}.",cs.CV,['cs.CV']
+Named Entity Driven Zero-Shot Image Manipulation,Zhida Feng · Li Chen · Jing Tian · Jiaxiang Liu · Shikun Feng,https://github.com/feng-zhida/StyleEntity,https://arxiv.org/abs/2307.13497,,2307.13497.pdf,Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction,"The Zero-Shot Learning (ZSL) task pertains to the identification of entities
+or relations in texts that were not seen during training. ZSL has emerged as a
+critical research area due to the scarcity of labeled data in specific domains,
+and its applications have grown significantly in recent years. With the advent
+of large pretrained language models, several novel methods have been proposed,
+resulting in substantial improvements in ZSL performance. There is a growing
+demand, both in the research community and industry, for a comprehensive ZSL
+framework that facilitates the development and accessibility of the latest
+methods and pretrained models.In this study, we propose a novel ZSL framework
+called Zshot that aims to address the aforementioned challenges. Our primary
+objective is to provide a platform that allows researchers to compare different
+state-of-the-art ZSL methods with standard benchmark datasets. Additionally, we
+have designed our framework to support the industry with readily available APIs
+for production under the standard SpaCy NLP pipeline. Our API is extendible and
+evaluable, moreover, we include numerous enhancements such as boosting the
+accuracy with pipeline ensembling and visualization utilities available as a
+SpaCy extension.",cs.CL,"['cs.CL', 'cs.AI', 'cs.LG']"
+Learned Scanpaths Aid Blind Panoramic Video Quality Assessment,Kanglong FAN · Wen Wen · Mu Li · YIFAN PENG · Kede Ma,https://github.com/kalofan/AutoScanpathQA,https://arxiv.org/abs/2404.00252,,2404.00252.pdf,Learned Scanpaths Aid Blind Panoramic Video Quality Assessment,"Panoramic videos have the advantage of providing an immersive and interactive
+viewing experience. Nevertheless, their spherical nature gives rise to various
+and uncertain user viewing behaviors, which poses significant challenges for
+panoramic video quality assessment (PVQA). In this work, we propose an
+end-to-end optimized, blind PVQA method with explicit modeling of user viewing
+patterns through visual scanpaths. Our method consists of two modules: a
+scanpath generator and a quality assessor. The scanpath generator is initially
+trained to predict future scanpaths by minimizing their expected code length
+and then jointly optimized with the quality assessor for quality prediction.
+Our blind PVQA method enables direct quality assessment of panoramic images by
+treating them as videos composed of identical frames. Experiments on three
+public panoramic image and video quality datasets, encompassing both synthetic
+and authentic distortions, validate the superiority of our blind PVQA model
+over existing methods.",eess.IV,"['eess.IV', 'cs.CV']"
+Molecular Data Programming: Towards Molecule Pseudo-labeling with Systematic Weak Supervision,Xin Juan · Kaixiong Zhou · Ninghao Liu · Tianlong Chen · Xin Wang, ,https://arxiv.org/abs/2309.05203,,2309.05203.pdf,From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery,"Molecule discovery serves as a cornerstone in numerous scientific domains,
+fueling the development of new materials and innovative drug designs. Recent
+developments of in-silico molecule discovery have highlighted the promising
+results of cross-modal techniques, which bridge molecular structures with their
+descriptive annotations. However, these cross-modal methods frequently
+encounter the issue of data scarcity, hampering their performance and
+application. In this paper, we address the low-resource challenge by utilizing
+artificially-real data generated by Large Language Models (LLMs). We first
+introduce a retrieval-based prompting strategy to construct high-quality pseudo
+data, then explore the optimal method to effectively leverage this pseudo data.
+Experiments show that using pseudo data for domain adaptation outperforms all
+existing methods, while also requiring a smaller model scale, reduced data size
+and lower training cost, highlighting its efficiency. Furthermore, our method
+shows a sustained improvement as the volume of pseudo data increases, revealing
+the great potential of pseudo data in advancing low-resource cross-modal
+molecule discovery. Our code and data are available at
+https://github.com/SCIR-HI/ArtificiallyR2R.",cs.CL,['cs.CL']
+ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation,Xiaoqi Li · Mingxu Zhang · Yiran Geng · Haoran Geng · Haoran Geng · Yuxing Long · Yan Shen · Renrui Zhang · Jiaming Liu · Hao Dong,https://sites.google.com/view/manipllm,https://arxiv.org/abs/2312.16217,,2312.16217.pdf,ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation,"Robot manipulation relies on accurately predicting contact points and
+end-effector directions to ensure successful operation. However, learning-based
+robot manipulation, trained on a limited category within a simulator, often
+struggles to achieve generalizability, especially when confronted with
+extensive categories. Therefore, we introduce an innovative approach for robot
+manipulation that leverages the robust reasoning capabilities of Multimodal
+Large Language Models (MLLMs) to enhance the stability and generalization of
+manipulation. By fine-tuning the injected adapters, we preserve the inherent
+common sense and reasoning ability of the MLLMs while equipping them with the
+ability for manipulation. The fundamental insight lies in the introduced
+fine-tuning paradigm, encompassing object category understanding, affordance
+prior reasoning, and object-centric pose prediction to stimulate the reasoning
+ability of MLLM in manipulation. During inference, our approach utilizes an RGB
+image and text prompt to predict the end effector's pose in chain of thoughts.
+After the initial contact is established, an active impedance adaptation policy
+is introduced to plan the upcoming waypoints in a closed-loop manner. Moreover,
+in real world, we design a test-time adaptation (TTA) strategy for manipulation
+to enable the model better adapt to the current real-world scene configuration.
+Experiments in simulator and real-world show the promising performance of
+ManipLLM. More details and demonstrations can be found at
+https://sites.google.com/view/manipllm.",cs.CV,"['cs.CV', 'cs.RO']"
+Consistent Prompting for Rehearsal-Free Continual Learning,Zhanxin Gao · Jun Cen · Xiaobin Chang,https://github.com/Zhanxin-Gao/CPrompt,https://arxiv.org/abs/2403.08568,,2403.08568.pdf,Consistent Prompting for Rehearsal-Free Continual Learning,"Continual learning empowers models to adapt autonomously to the ever-changing
+environment or data streams without forgetting old knowledge. Prompt-based
+approaches are built on frozen pre-trained models to learn the task-specific
+prompts and classifiers efficiently. Existing prompt-based methods are
+inconsistent between training and testing, limiting their effectiveness. Two
+types of inconsistency are revealed. Test predictions are made from all
+classifiers while training only focuses on the current task classifier without
+holistic alignment, leading to Classifier inconsistency. Prompt inconsistency
+indicates that the prompt selected during testing may not correspond to the one
+associated with this task during training. In this paper, we propose a novel
+prompt-based method, Consistent Prompting (CPrompt), for more aligned training
+and testing. Specifically, all existing classifiers are exposed to prompt
+training, resulting in classifier consistency learning. In addition, prompt
+consistency learning is proposed to enhance prediction robustness and boost
+prompt selection accuracy. Our Consistent Prompting surpasses its prompt-based
+counterparts and achieves state-of-the-art performance on multiple continual
+learning benchmarks. Detailed analysis shows that improvements come from more
+consistent training and testing.",cs.CV,"['cs.CV', 'cs.LG']"
+Pixel-level Semantic Correspondence through Layout-aware Representation Learning and Multi-scale Matching Integration,Yixuan Sun · Zhangyue Yin · Haibo Wang · Yan Wang · Xipeng Qiu · Weifeng Ge · Wenqiang Zhang, ,https://ar5iv.labs.arxiv.org/html/2401.11739,,2401.11739.pdf,EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models,"Diffusion models have recently received increasing research attention for
+their remarkable transfer abilities in semantic segmentation tasks. However,
+generating fine-grained segmentation masks with diffusion models often requires
+additional training on annotated datasets, leaving it unclear to what extent
+pre-trained diffusion models alone understand the semantic relations of their
+generated images. To address this question, we leverage the semantic knowledge
+extracted from Stable Diffusion (SD) and aim to develop an image segmentor
+capable of generating fine-grained segmentation maps without any additional
+training. The primary difficulty stems from the fact that semantically
+meaningful feature maps typically exist only in the spatially lower-dimensional
+layers, which poses a challenge in directly extracting pixel-level semantic
+relations from these feature maps. To overcome this issue, our framework
+identifies semantic correspondences between image pixels and spatial locations
+of low-dimensional feature maps by exploiting SD's generation process and
+utilizes them for constructing image-resolution segmentation maps. In extensive
+experiments, the produced segmentation maps are demonstrated to be well
+delineated and capture detailed parts of the images, indicating the existence
+of highly accurate pixel-level semantic knowledge in diffusion models.",cs.CV,"['cs.CV', 'cs.LG']"
+Model Adaptation for Time Constrained Embodied Control,Jaehyun Song · Minjong Yoo · Honguk Woo, ,,https://ieeexplore.ieee.org/document/10510652,,,,,nan
+360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries,Huajian Huang · Changkun Liu · Yipeng Zhu · Hui Cheng · Tristan Braud · Sai-Kit Yeung, ,https://arxiv.org/abs/2311.17389,,2311.17389.pdf,360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries,"Portable 360$^\circ$ cameras are becoming a cheap and efficient tool to
+establish large visual databases. By capturing omnidirectional views of a
+scene, these cameras could expedite building environment models that are
+essential for visual localization. However, such an advantage is often
+overlooked due to the lack of valuable datasets. This paper introduces a new
+benchmark dataset, 360Loc, composed of 360$^\circ$ images with ground truth
+poses for visual localization. We present a practical implementation of
+360$^\circ$ mapping combining 360$^\circ$ images with lidar data to generate
+the ground truth 6DoF poses. 360Loc is the first dataset and benchmark that
+explores the challenge of cross-device visual positioning, involving
+360$^\circ$ reference frames, and query frames from pinhole, ultra-wide FoV
+fisheye, and 360$^\circ$ cameras. We propose a virtual camera approach to
+generate lower-FoV query frames from 360$^\circ$ images, which ensures a fair
+comparison of performance among different query types in visual localization
+tasks. We also extend this virtual camera approach to feature matching-based
+and pose regression-based methods to alleviate the performance loss caused by
+the cross-device domain gap, and evaluate its effectiveness against
+state-of-the-art baselines. We demonstrate that omnidirectional visual
+localization is more robust in challenging large-scale scenes with symmetries
+and repetitive structures. These results provide new insights into 360-camera
+mapping and omnidirectional visual localization with cross-device queries.",cs.CV,['cs.CV']
+Layout-Agnostic Scene Text Image Synthesis with Diffusion Models,Qilong Zhangli · Jindong Jiang · Di Liu · Licheng Yu · Xiaoliang Dai · Ankit Ramchandani · Guan Pang · Dimitris N. Metaxas · Praveen Krishnan, ,https://arxiv.org/abs/2312.04884,,2312.04884.pdf,UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models,"Text-to-Image (T2I) generation methods based on diffusion model have garnered
+significant attention in the last few years. Although these image synthesis
+methods produce visually appealing results, they frequently exhibit spelling
+errors when rendering text within the generated images. Such errors manifest as
+missing, incorrect or extraneous characters, thereby severely constraining the
+performance of text image generation based on diffusion models. To address the
+aforementioned issue, this paper proposes a novel approach for text image
+generation, utilizing a pre-trained diffusion model (i.e., Stable Diffusion
+[27]). Our approach involves the design and training of a light-weight
+character-level text encoder, which replaces the original CLIP encoder and
+provides more robust text embeddings as conditional guidance. Then, we
+fine-tune the diffusion model using a large-scale dataset, incorporating local
+attention control under the supervision of character-level segmentation maps.
+Finally, by employing an inference stage refinement process, we achieve a
+notably high sequence accuracy when synthesizing text in arbitrarily given
+images. Both qualitative and quantitative results demonstrate the superiority
+of our method to the state of the art. Furthermore, we showcase several
+potential applications of the proposed UDiffText, including text-centric image
+synthesis, scene text editing, etc. Code and model will be available at
+https://github.com/ZYM-PKU/UDiffText .",cs.CV,['cs.CV']
+Amodal Completion via Progressive Mixed Context Diffusion,Katherine Xu · Lingzhi Zhang · Jianbo Shi,https://k8xu.github.io/amodal,https://arxiv.org/abs/2312.15540,,2312.15540.pdf,Amodal Completion via Progressive Mixed Context Diffusion,"Our brain can effortlessly recognize objects even when partially hidden from
+view. Seeing the visible of the hidden is called amodal completion; however,
+this task remains a challenge for generative AI despite rapid progress. We
+propose to sidestep many of the difficulties of existing approaches, which
+typically involve a two-step process of predicting amodal masks and then
+generating pixels. Our method involves thinking outside the box, literally! We
+go outside the object bounding box to use its context to guide a pre-trained
+diffusion inpainting model, and then progressively grow the occluded object and
+trim the extra background. We overcome two technical challenges: 1) how to be
+free of unwanted co-occurrence bias, which tends to regenerate similar
+occluders, and 2) how to judge if an amodal completion has succeeded. Our
+amodal completion method exhibits improved photorealistic completion results
+compared to existing approaches in numerous successful completion cases. And
+the best part? It doesn't require any special training or fine-tuning of
+models.",cs.CV,['cs.CV']
+Make Pixels Dance: High-Dynamic Video Generation,Yan Zeng · Guoqiang Wei · Jiani Zheng · Jiaxin Zou · Yang Wei · Yuchen Zhang · Yuchen Zhang · Hang Li, ,https://arxiv.org/abs/2311.10982,,2311.10982.pdf,Make Pixels Dance: High-Dynamic Video Generation,"Creating high-dynamic videos such as motion-rich actions and sophisticated
+visual effects poses a significant challenge in the field of artificial
+intelligence. Unfortunately, current state-of-the-art video generation methods,
+primarily focusing on text-to-video generation, tend to produce video clips
+with minimal motions despite maintaining high fidelity. We argue that relying
+solely on text instructions is insufficient and suboptimal for video
+generation. In this paper, we introduce PixelDance, a novel approach based on
+diffusion models that incorporates image instructions for both the first and
+last frames in conjunction with text instructions for video generation.
+Comprehensive experimental results demonstrate that PixelDance trained with
+public data exhibits significantly better proficiency in synthesizing videos
+with complex scenes and intricate motions, setting a new standard for video
+generation.",cs.CV,['cs.CV']
+MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World,Yining Hong · Zishuo Zheng · Peihao Chen · Yian Wang · Junyan Li · Chuang Gan, ,https://arxiv.org/abs/2401.08577,,2401.08577.pdf,MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World,"Human beings possess the capability to multiply a melange of multisensory
+cues while actively exploring and interacting with the 3D world. Current
+multi-modal large language models, however, passively absorb sensory data as
+inputs, lacking the capacity to actively interact with the objects in the 3D
+environment and dynamically collect their multisensory information. To usher in
+the study of this area, we propose MultiPLY, a multisensory embodied large
+language model that could incorporate multisensory interactive data, including
+visual, audio, tactile, and thermal information into large language models,
+thereby establishing the correlation among words, actions, and percepts. To
+this end, we first collect Multisensory Universe, a large-scale multisensory
+interaction dataset comprising 500k data by deploying an LLM-powered embodied
+agent to engage with the 3D environment. To perform instruction tuning with
+pre-trained LLM on such generated data, we first encode the 3D scene as
+abstracted object-centric representations and then introduce action tokens
+denoting that the embodied agent takes certain actions within the environment,
+as well as state tokens that represent the multisensory state observations of
+the agent at each time step. In the inference time, MultiPLY could generate
+action tokens, instructing the agent to take the action in the environment and
+obtain the next multisensory state observation. The observation is then
+appended back to the LLM via state tokens to generate subsequent text or action
+tokens. We demonstrate that MultiPLY outperforms baselines by a large margin
+through a diverse set of embodied tasks involving object retrieval, tool use,
+multisensory captioning, and task decomposition.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG', 'cs.RO']"
+Referring Expression Counting,Siyang Dai · Jun Liu · Ngai-Man Cheung, ,https://arxiv.org/abs/2405.15658,,2405.15658.pdf,HDC: Hierarchical Semantic Decoding with Counting Assistance for Generalized Referring Expression Segmentation,"The newly proposed Generalized Referring Expression Segmentation (GRES)
+amplifies the formulation of classic RES by involving multiple/non-target
+scenarios. Recent approaches focus on optimizing the last modality-fused
+feature which is directly utilized for segmentation and object-existence
+identification. However, the attempt to integrate all-grained information into
+a single joint representation is impractical in GRES due to the increased
+complexity of the spatial relationships among instances and deceptive text
+descriptions. Furthermore, the subsequent binary target justification across
+all referent scenarios fails to specify their inherent differences, leading to
+ambiguity in object understanding. To address the weakness, we propose a
+$\textbf{H}$ierarchical Semantic $\textbf{D}$ecoding with $\textbf{C}$ounting
+Assistance framework (HDC). It hierarchically transfers complementary modality
+information across granularities, and then aggregates each well-aligned
+semantic correspondence for multi-level decoding. Moreover, with complete
+semantic context modeling, we endow HDC with explicit counting capability to
+facilitate comprehensive object perception in multiple/single/non-target
+settings. Experimental results on gRefCOCO, Ref-ZOM, R-RefCOCO, and RefCOCO
+benchmarks demonstrate the effectiveness and rationality of HDC which
+outperforms the state-of-the-art GRES methods by a remarkable margin. Code will
+be available $\href{https://github.com/RobertLuo1/HDC}{here}$.",cs.CV,"['cs.CV', 'cs.AI']"
+UnionFormer: Unified-Learning Transformer with Multi-View Representation for Image Manipulation Detection and Localization,Shuaibo Li · Wei Ma · Jianwei Guo · Shibiao Xu · Benchong Li · Xiaopeng Zhang, ,,https://ieeexplore.ieee.org/abstract/document/10155416,,,,,nan
+Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimization,Takuhiro Kaneko, ,,https://adversarr.github.io/ps/Papers/2024/03/14/pac-nerf-physics-augmented-continuum-neural-radiance-fields-for-geometry-agnostic-system-identification/,,,,,nan
+SonicVisionLM: Playing Sound with Vision Language Models,Zhifeng Xie · Shengye Yu · Qile He · Mengtian Li, ,https://arxiv.org/abs/2401.04394,,2401.04394.pdf,SonicVisionLM: Playing Sound with Vision Language Models,"There has been a growing interest in the task of generating sound for silent
+videos, primarily because of its practicality in streamlining video
+post-production. However, existing methods for video-sound generation attempt
+to directly create sound from visual representations, which can be challenging
+due to the difficulty of aligning visual representations with audio
+representations. In this paper, we present SonicVisionLM, a novel framework
+aimed at generating a wide range of sound effects by leveraging vision-language
+models(VLMs). Instead of generating audio directly from video, we use the
+capabilities of powerful VLMs. When provided with a silent video, our approach
+first identifies events within the video using a VLM to suggest possible sounds
+that match the video content. This shift in approach transforms the challenging
+task of aligning image and audio into more well-studied sub-problems of
+aligning image-to-text and text-to-audio through the popular diffusion models.
+To improve the quality of audio recommendations with LLMs, we have collected an
+extensive dataset that maps text descriptions to specific sound effects and
+developed a time-controlled audio adapter. Our approach surpasses current
+state-of-the-art methods for converting video to audio, enhancing
+synchronization with the visuals, and improving alignment between audio and
+video components. Project page:
+https://yusiissy.github.io/SonicVisionLM.github.io/",cs.MM,"['cs.MM', 'cs.SD', 'eess.AS']"
+A Dual-Augmentor Framework for Domain Generalization in 3D Human Pose Estimation,Qucheng Peng · Ce Zheng · Chen Chen, ,https://arxiv.org/abs/2403.11310,,2403.11310.pdf,A Dual-Augmentor Framework for Domain Generalization in 3D Human Pose Estimation,"3D human pose data collected in controlled laboratory settings present
+challenges for pose estimators that generalize across diverse scenarios. To
+address this, domain generalization is employed. Current methodologies in
+domain generalization for 3D human pose estimation typically utilize
+adversarial training to generate synthetic poses for training. Nonetheless,
+these approaches exhibit several limitations. First, the lack of prior
+information about the target domain complicates the application of suitable
+augmentation through a single pose augmentor, affecting generalization on
+target domains. Moreover, adversarial training's discriminator tends to enforce
+similarity between source and synthesized poses, impeding the exploration of
+out-of-source distributions. Furthermore, the pose estimator's optimization is
+not exposed to domain shifts, limiting its overall generalization ability.
+  To address these limitations, we propose a novel framework featuring two pose
+augmentors: the weak and the strong augmentors. Our framework employs
+differential strategies for generation and discrimination processes,
+facilitating the preservation of knowledge related to source poses and the
+exploration of out-of-source distributions without prior information about
+target poses. Besides, we leverage meta-optimization to simulate domain shifts
+in the optimization process of the pose estimator, thereby improving its
+generalization ability. Our proposed approach significantly outperforms
+existing methods, as demonstrated through comprehensive experiments on various
+benchmark datasets.Our code will be released at
+\url{https://github.com/davidpengucf/DAF-DG}.",cs.CV,['cs.CV']
+ProMotion: Prototypes As Motion Learners,Yawen Lu · Dongfang Liu · Qifan Wang · Cheng Han · Yiming Cui · Yiming Cui · Zhiwen Cao · Xueling Zhang · Yingjie Victor Chen · Heng Fan, ,https://ar5iv.labs.arxiv.org/html/2304.11523,,2304.11523.pdf,TransFlow: Transformer as Flow Learner,"Optical flow is an indispensable building block for various important
+computer vision tasks, including motion estimation, object tracking, and
+disparity measurement. In this work, we propose TransFlow, a pure transformer
+architecture for optical flow estimation. Compared to dominant CNN-based
+methods, TransFlow demonstrates three advantages. First, it provides more
+accurate correlation and trustworthy matching in flow estimation by utilizing
+spatial self-attention and cross-attention mechanisms between adjacent frames
+to effectively capture global dependencies; Second, it recovers more
+compromised information (e.g., occlusion and motion blur) in flow estimation
+through long-range temporal association in dynamic scenes; Third, it enables a
+concise self-learning paradigm and effectively eliminate the complex and
+laborious multi-stage pre-training procedures. We achieve the state-of-the-art
+results on the Sintel, KITTI-15, as well as several downstream tasks, including
+video object detection, interpolation and stabilization. For its efficacy, we
+hope TransFlow could serve as a flexible baseline for optical flow estimation.",cs.CV,['cs.CV']
+Event-assisted Low-Light Video Object Segmentation,Li Hebei · Jin Wang · Jiahui Yuan · Yue Li · Wenming Weng · Yansong Peng · Yueyi Zhang · Zhiwei Xiong · Xiaoyan Sun, ,https://arxiv.org/abs/2404.01945,,2404.01945.pdf,Event-assisted Low-Light Video Object Segmentation,"In the realm of video object segmentation (VOS), the challenge of operating
+under low-light conditions persists, resulting in notably degraded image
+quality and compromised accuracy when comparing query and memory frames for
+similarity computation. Event cameras, characterized by their high dynamic
+range and ability to capture motion information of objects, offer promise in
+enhancing object visibility and aiding VOS methods under such low-light
+conditions. This paper introduces a pioneering framework tailored for low-light
+VOS, leveraging event camera data to elevate segmentation accuracy. Our
+approach hinges on two pivotal components: the Adaptive Cross-Modal Fusion
+(ACMF) module, aimed at extracting pertinent features while fusing image and
+event modalities to mitigate noise interference, and the Event-Guided Memory
+Matching (EGMM) module, designed to rectify the issue of inaccurate matching
+prevalent in low-light settings. Additionally, we present the creation of a
+synthetic LLE-DAVIS dataset and the curation of a real-world LLE-VOS dataset,
+encompassing frames and events. Experimental evaluations corroborate the
+efficacy of our method across both datasets, affirming its effectiveness in
+low-light scenarios.",cs.CV,['cs.CV']
+Towards Backward-Compatible Continual Learning of Image Compression,Zhihao Duan · Ming Lu · Justin Yang · Jiangpeng He · Zhan Ma · Fengqing Zhu, ,https://arxiv.org/abs/2402.18862,,2402.18862.pdf,Towards Backward-Compatible Continual Learning of Image Compression,"This paper explores the possibility of extending the capability of
+pre-trained neural image compressors (e.g., adapting to new data or target
+bitrates) without breaking backward compatibility, the ability to decode
+bitstreams encoded by the original model. We refer to this problem as continual
+learning of image compression. Our initial findings show that baseline
+solutions, such as end-to-end fine-tuning, do not preserve the desired backward
+compatibility. To tackle this, we propose a knowledge replay training strategy
+that effectively addresses this issue. We also design a new model architecture
+that enables more effective continual learning than existing baselines.
+Experiments are conducted for two scenarios: data-incremental learning and
+rate-incremental learning. The main conclusion of this paper is that neural
+image compressors can be fine-tuned to achieve better performance (compared to
+their pre-trained version) on new data and rates without compromising backward
+compatibility. Our code is available at
+https://gitlab.com/viper-purdue/continual-compression",eess.IV,['eess.IV']
+Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses,Inhee Lee · Byungjun Kim · Hanbyul Joo, ,http://export.arxiv.org/abs/2404.14410,,2404.14410.pdf,Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses,"In this paper, we present a method to reconstruct the world and multiple
+dynamic humans in 3D from a monocular video input. As a key idea, we represent
+both the world and multiple humans via the recently emerging 3D Gaussian
+Splatting (3D-GS) representation, enabling to conveniently and efficiently
+compose and render them together. In particular, we address the scenarios with
+severely limited and sparse observations in 3D human reconstruction, a common
+challenge encountered in the real world. To tackle this challenge, we introduce
+a novel approach to optimize the 3D-GS representation in a canonical space by
+fusing the sparse cues in the common space, where we leverage a pre-trained 2D
+diffusion model to synthesize unseen views while keeping the consistency with
+the observed 2D appearances. We demonstrate our method can reconstruct
+high-quality animatable 3D humans in various challenging examples, in the
+presence of occlusion, image crops, few-shot, and extremely sparse
+observations. After reconstruction, our method is capable of not only rendering
+the scene in any novel views at arbitrary time instances, but also editing the
+3D scene by removing individual humans or applying different motions for each
+human. Through various experiments, we demonstrate the quality and efficiency
+of our methods over alternative existing approaches.",cs.CV,['cs.CV']
+EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling,Haiyang Liu · Zihao Zhu · Giorgio Becherini · YICHEN PENG · Mingyang Su · YOU ZHOU · Xuefei Zhe · Naoya Iwamoto · Bo Zheng · Michael J. Black,https://pantomatrix.github.io/EMAGE/,https://arxiv.org/abs/2401.00374,,2401.00374.pdf,EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling,"We propose EMAGE, a framework to generate full-body human gestures from audio
+and masked gestures, encompassing facial, local body, hands, and global
+movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new
+mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with
+FLAME head parameters and further refines the modeling of head, neck, and
+finger movements, offering a community-standardized, high-quality 3D motion
+captured dataset. EMAGE leverages masked body gesture priors during training to
+boost inference performance. It involves a Masked Audio Gesture Transformer,
+facilitating joint training on audio-to-gesture generation and masked gesture
+reconstruction to effectively encode audio and body gesture hints. Encoded body
+hints from masked gestures are then separately employed to generate facial and
+body movements. Moreover, EMAGE adaptively merges speech features from the
+audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance
+the results' fidelity and diversity. Experiments demonstrate that EMAGE
+generates holistic gestures with state-of-the-art performance and is flexible
+in accepting predefined spatial-temporal gesture inputs, generating complete,
+audio-synchronized results. Our code and dataset are available
+https://pantomatrix.github.io/EMAGE/",cs.CV,['cs.CV']
+A Semi-supervised Nighttime Dehazing Baseline with Spatial-Frequency Aware and Realistic Brightness Constraint,Xiaofeng Cong · Jie Gui · Jing Zhang · Junming Hou · Hao Shen,https://github.com/Xiaofeng-life/SFSNiD/,https://arxiv.org/abs/2403.18548,,2403.18548.pdf,A Semi-supervised Nighttime Dehazing Baseline with Spatial-Frequency Aware and Realistic Brightness Constraint,"Existing research based on deep learning has extensively explored the problem
+of daytime image dehazing. However, few studies have considered the
+characteristics of nighttime hazy scenes. There are two distinctions between
+nighttime and daytime haze. First, there may be multiple active colored light
+sources with lower illumination intensity in nighttime scenes, which may cause
+haze, glow and noise with localized, coupled and frequency inconsistent
+characteristics. Second, due to the domain discrepancy between simulated and
+real-world data, unrealistic brightness may occur when applying a dehazing
+model trained on simulated data to real-world data. To address the above two
+issues, we propose a semi-supervised model for real-world nighttime dehazing.
+First, the spatial attention and frequency spectrum filtering are implemented
+as a spatial-frequency domain information interaction module to handle the
+first issue. Second, a pseudo-label-based retraining strategy and a local
+window-based brightness loss for semi-supervised training process is designed
+to suppress haze and glow while achieving realistic brightness. Experiments on
+public benchmarks validate the effectiveness of the proposed method and its
+superiority over state-of-the-art methods. The source code and Supplementary
+Materials are placed in the https://github.com/Xiaofeng-life/SFSNiD.",cs.CV,['cs.CV']
+How to Configure Good In-Context Sequence for Visual Question Answering,Li Li · Jiawei Peng · huiyi chen · Chongyang Gao · Xu Yang, ,https://arxiv.org/abs/2312.01571,,2312.01571.pdf,How to Configure Good In-Context Sequence for Visual Question Answering,"Inspired by the success of Large Language Models in dealing with new tasks
+via In-Context Learning (ICL) in NLP, researchers have also developed Large
+Vision-Language Models (LVLMs) with ICL capabilities. However, when
+implementing ICL using these LVLMs, researchers usually resort to the simplest
+way like random sampling to configure the in-context sequence, thus leading to
+sub-optimal results. To enhance the ICL performance, in this study, we use
+Visual Question Answering (VQA) as case study to explore diverse in-context
+configurations to find the powerful ones. Additionally, through observing the
+changes of the LVLM outputs by altering the in-context sequence, we gain
+insights into the inner properties of LVLMs, improving our understanding of
+them. Specifically, to explore in-context configurations, we design diverse
+retrieval methods and employ different strategies to manipulate the retrieved
+demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2,
+VizWiz, and OK-VQA, we uncover three important inner properties of the applied
+LVLM and demonstrate which strategies can consistently improve the ICL VQA
+performance. Our code is provided in:
+https://github.com/GaryJiajia/OFv2_ICL_VQA.",cs.CV,"['cs.CV', 'cs.AI']"
+Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval,Minkuk Kim · Hyeon Bae Kim · Jinyoung Moon · Jinwoo Choi · Seong Tae Kim, ,https://arxiv.org/abs/2404.07610,,2404.07610.pdf,Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval,"There has been significant attention to the research on dense video
+captioning, which aims to automatically localize and caption all events within
+untrimmed video. Several studies introduce methods by designing dense video
+captioning as a multitasking problem of event localization and event captioning
+to consider inter-task relations. However, addressing both tasks using only
+visual input is challenging due to the lack of semantic content. In this study,
+we address this by proposing a novel framework inspired by the cognitive
+information processing of humans. Our model utilizes external memory to
+incorporate prior knowledge. The memory retrieval method is proposed with
+cross-modal video-to-text matching. To effectively incorporate retrieved text
+features, the versatile encoder and the decoder with visual and textual
+cross-attention modules are designed. Comparative experiments have been
+conducted to show the effectiveness of the proposed method on ActivityNet
+Captions and YouCook2 datasets. Experimental results show promising performance
+of our model without extensive pretraining from a large video dataset.",cs.CV,['cs.CV']
+Towards Text-guided 3D Scene Composition,Qihang Zhang · Chaoyang Wang · Aliaksandr Siarohin · Peiye Zhuang · Yinghao Xu · Ceyuan Yang · Dahua Lin · Bolei Zhou · Sergey Tulyakov · Hsin-Ying Lee, ,https://arxiv.org/abs/2312.08885,,2312.08885.pdf,SceneWiz3D: Towards Text-guided 3D Scene Composition,"We are witnessing significant breakthroughs in the technology for generating
+3D objects from text. Existing approaches either leverage large text-to-image
+models to optimize a 3D representation or train 3D generators on object-centric
+datasets. Generating entire scenes, however, remains very challenging as a
+scene contains multiple 3D objects, diverse and scattered. In this work, we
+introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes
+from text. We marry the locality of objects with globality of scenes by
+introducing a hybrid 3D representation: explicit for objects and implicit for
+scenes. Remarkably, an object, being represented explicitly, can be either
+generated from text using conventional text-to-3D approaches, or provided by
+users. To configure the layout of the scene and automatically place objects, we
+apply the Particle Swarm Optimization technique during the optimization
+process. Furthermore, it is difficult for certain parts of the scene (e.g.,
+corners, occlusion) to receive multi-view supervision, leading to inferior
+geometry. We incorporate an RGBD panorama diffusion model to mitigate it,
+resulting in high-quality geometry. Extensive evaluation supports that our
+approach achieves superior quality over previous approaches, enabling the
+generation of detailed and view-consistent 3D scenes.",cs.CV,['cs.CV']
+Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs,Kanchana Ranasinghe · Satya Narayan Shukla · Omid Poursaeed · Michael Ryoo · Tsung-Yu Lin, ,https://arxiv.org/abs/2404.07449,,2404.07449.pdf,Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs,"Integration of Large Language Models (LLMs) into visual domain tasks,
+resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in
+vision-language tasks, particularly for visual question answering (VQA).
+However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial
+reasoning and localization awareness. Despite generating highly descriptive and
+elaborate textual answers, these models fail at simple tasks like
+distinguishing a left vs right location. In this work, we explore how
+image-space coordinate based instruction fine-tuning objectives could inject
+spatial awareness into V-LLMs. We discover optimal coordinate representations,
+data-efficient instruction fine-tuning objectives, and pseudo-data generation
+strategies that lead to improved spatial awareness in V-LLMs. Additionally, our
+resulting model improves VQA across image and video domains, reduces undesired
+hallucination, and generates better contextual object descriptions. Experiments
+across 5 vision-language tasks involving 14 different datasets establish the
+clear performance improvements achieved by our proposed framework.",cs.CV,['cs.CV']
+Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning,Leonardo Iurada · Marco Ciccone · Tatiana Tommasi,https://iurada.github.io/PX,https://arxiv.org/abs/2405.00906,,2405.00906.pdf,LOTUS: Improving Transformer Efficiency with Sparsity Pruning and Data Lottery Tickets,"Vision transformers have revolutionized computer vision, but their
+computational demands present challenges for training and deployment. This
+paper introduces LOTUS (LOttery Transformers with Ultra Sparsity), a novel
+method that leverages data lottery ticket selection and sparsity pruning to
+accelerate vision transformer training while maintaining accuracy. Our approach
+focuses on identifying and utilizing the most informative data subsets and
+eliminating redundant model parameters to optimize the training process.
+Through extensive experiments, we demonstrate the effectiveness of LOTUS in
+achieving rapid convergence and high accuracy with significantly reduced
+computational requirements. This work highlights the potential of combining
+data selection and sparsity techniques for efficient vision transformer
+training, opening doors for further research and development in this area.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Fully Geometric Panoramic Localization,Junho Kim · Jiwon Jeong · Young Min Kim,https://82magnolia.github.io/fgpl/,https://arxiv.org/abs/2403.19904,,2403.19904.pdf,Fully Geometric Panoramic Localization,"We introduce a lightweight and accurate localization method that only
+utilizes the geometry of 2D-3D lines. Given a pre-captured 3D map, our approach
+localizes a panorama image, taking advantage of the holistic 360 view. The
+system mitigates potential privacy breaches or domain discrepancies by avoiding
+trained or hand-crafted visual descriptors. However, as lines alone can be
+ambiguous, we express distinctive yet compact spatial contexts from
+relationships between lines, namely the dominant directions of parallel lines
+and the intersection between non-parallel lines. The resulting representations
+are efficient in processing time and memory compared to conventional visual
+descriptor-based methods. Given the groups of dominant line directions and
+their intersections, we accelerate the search process to test thousands of pose
+candidates in less than a millisecond without sacrificing accuracy. We
+empirically show that the proposed 2D-3D matching can localize panoramas for
+challenging scenes with similar structures, dramatic domain shifts or
+illumination changes. Our fully geometric approach does not involve extensive
+parameter tuning or neural network training, making it a practical algorithm
+that can be readily deployed in the real world. Project page including the code
+is available through this link: https://82magnolia.github.io/fgpl/.",cs.CV,['cs.CV']
+VS: Reconstructing Clothed 3D Human from Single Image via Vertex Shift,Leyuan Liu · Yuhan Li · Yunqi Gao · Changxin Gao · Yuanyuan Liu · Jingying Chen,https://github.com/naivate/VS.git,https://arxiv.org/abs/2309.13524,,2309.13524.pdf,Global-correlated 3D-decoupling Transformer for Clothed Avatar Reconstruction,"Reconstructing 3D clothed human avatars from single images is a challenging
+task, especially when encountering complex poses and loose clothing. Current
+methods exhibit limitations in performance, largely attributable to their
+dependence on insufficient 2D image features and inconsistent query methods.
+Owing to this, we present the Global-correlated 3D-decoupling Transformer for
+clothed Avatar reconstruction (GTA), a novel transformer-based architecture
+that reconstructs clothed human avatars from monocular images. Our approach
+leverages transformer architectures by utilizing a Vision Transformer model as
+an encoder for capturing global-correlated image features. Subsequently, our
+innovative 3D-decoupling decoder employs cross-attention to decouple tri-plane
+features, using learnable embeddings as queries for cross-plane generation. To
+effectively enhance feature fusion with the tri-plane 3D feature and human body
+prior, we propose a hybrid prior fusion strategy combining spatial and
+prior-enhanced queries, leveraging the benefits of spatial localization and
+human body prior knowledge. Comprehensive experiments on CAPE and THuman2.0
+datasets illustrate that our method outperforms state-of-the-art approaches in
+both geometry and texture reconstruction, exhibiting high robustness to
+challenging poses and loose clothing, and producing higher-resolution textures.
+Codes will be available at https://github.com/River-Zhang/GTA.",cs.CV,"['cs.CV', 'cs.AI']"
+Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network,Yong Shu · Liquan Shen · Xiangyu Hu · Mengyao Li · Zihao Zhou,https://github.com/yungsyu99/Real-HDRV,https://arxiv.org/abs/2405.00244,,2405.00244.pdf,Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network,"As an important and practical way to obtain high dynamic range (HDR) video,
+HDR video reconstruction from sequences with alternating exposures is still
+less explored, mainly due to the lack of large-scale real-world datasets.
+Existing methods are mostly trained on synthetic datasets, which perform poorly
+in real scenes. In this work, to facilitate the development of real-world HDR
+video reconstruction, we present Real-HDRV, a large-scale real-world benchmark
+dataset for HDR video reconstruction, featuring various scenes, diverse motion
+patterns, and high-quality labels. Specifically, our dataset contains 500
+LDRs-HDRs video pairs, comprising about 28,000 LDR frames and 4,000 HDR labels,
+covering daytime, nighttime, indoor, and outdoor scenes. To our best knowledge,
+our dataset is the largest real-world HDR video reconstruction dataset.
+Correspondingly, we propose an end-to-end network for HDR video reconstruction,
+where a novel two-stage strategy is designed to perform alignment sequentially.
+Specifically, the first stage performs global alignment with the adaptively
+estimated global offsets, reducing the difficulty of subsequent alignment. The
+second stage implicitly performs local alignment in a coarse-to-fine manner at
+the feature level using the adaptive separable convolution. Extensive
+experiments demonstrate that: (1) models trained on our dataset can achieve
+better performance on real scenes than those trained on synthetic datasets; (2)
+our method outperforms previous state-of-the-art methods. Our dataset is
+available at https://github.com/yungsyu99/Real-HDRV.",cs.CV,['cs.CV']
+Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding,Jin-Chuan Shi · Miao Wang · Haobin Duan · Shaohua Guan,https://buaavrcg.github.io/LEGaussians/,https://arxiv.org/abs/2311.18482,,2311.18482.pdf,Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding,"Open-vocabulary querying in 3D space is challenging but essential for scene
+understanding tasks such as object localization and segmentation.
+Language-embedded scene representations have made progress by incorporating
+language features into 3D spaces. However, their efficacy heavily depends on
+neural networks that are resource-intensive in training and rendering. Although
+recent 3D Gaussians offer efficient and high-quality novel view synthesis,
+directly embedding language features in them leads to prohibitive memory usage
+and decreased performance. In this work, we introduce Language Embedded 3D
+Gaussians, a novel scene representation for open-vocabulary query tasks.
+Instead of embedding high-dimensional raw semantic features on 3D Gaussians, we
+propose a dedicated quantization scheme that drastically alleviates the memory
+requirement, and a novel embedding procedure that achieves smoother yet high
+accuracy query, countering the multi-view feature inconsistencies and the
+high-frequency inductive bias in point-based representations. Our comprehensive
+experiments show that our representation achieves the best visual quality and
+language querying accuracy across current language-embedded representations,
+while maintaining real-time rendering frame rates on a single desktop GPU.",cs.CV,"['cs.CV', 'cs.GR']"
+GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians,Shenhan Qian · Tobias Kirschstein · Liam Schoneveld · Davide Davoli · Simon Giebenhain · Matthias Nießner,https://shenhanqian.github.io/gaussian-avatars,https://arxiv.org/abs/2312.02069,,2312.02069.pdf,GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians,"We introduce GaussianAvatars, a new method to create photorealistic head
+avatars that are fully controllable in terms of expression, pose, and
+viewpoint. The core idea is a dynamic 3D representation based on 3D Gaussian
+splats that are rigged to a parametric morphable face model. This combination
+facilitates photorealistic rendering while allowing for precise animation
+control via the underlying parametric model, e.g., through expression transfer
+from a driving sequence or by manually changing the morphable model parameters.
+We parameterize each splat by a local coordinate frame of a triangle and
+optimize for explicit displacement offset to obtain a more accurate geometric
+representation. During avatar reconstruction, we jointly optimize for the
+morphable model parameters and Gaussian splat parameters in an end-to-end
+fashion. We demonstrate the animation capabilities of our photorealistic avatar
+in several challenging scenarios. For instance, we show reenactments from a
+driving video, where our method outperforms existing works by a significant
+margin.",cs.CV,['cs.CV']
+Garment Recovery with Shape and Deformation Priors,Ren Li · Corentin Dumery · Benoît Guillard · Pascal Fua, ,https://arxiv.org/abs/2311.10356,,2311.10356.pdf,Garment Recovery with Shape and Deformation Priors,"While modeling people wearing tight-fitting clothing has made great strides
+in recent years, loose-fitting clothing remains a challenge. We propose a
+method that delivers realistic garment models from real-world images,
+regardless of garment shape or deformation. To this end, we introduce a fitting
+approach that utilizes shape and deformation priors learned from synthetic data
+to accurately capture garment shapes and deformations, including large ones.
+Not only does our approach recover the garment geometry accurately, it also
+yields models that can be directly used by downstream applications such as
+animation and simulation.",cs.CV,['cs.CV']
+Neighbor Relations Matter in Video Scene Detection,Jiawei Tan · Hongxing Wang · Jiaxin Li · Zhilong Ou · Zhangbin Qian, ,,https://www.semanticscholar.org/paper/Characters-Link-Shots:-Character-Attention-Network-Tan-Wang/031a0952b156f36ea9da7113ade868754100e4b7,,,,,nan
+The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective,Wenqi Jia · Miao Liu · Hao Jiang · Ishwarya Ananthabhotla · James Rehg · Vamsi Krishna Ithapu · Ruohan Gao, ,https://arxiv.org/abs/2312.12870,,2312.12870.pdf,The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective,"In recent years, the thriving development of research related to egocentric
+videos has provided a unique perspective for the study of conversational
+interactions, where both visual and audio signals play a crucial role. While
+most prior work focus on learning about behaviors that directly involve the
+camera wearer, we introduce the Ego-Exocentric Conversational Graph Prediction
+problem, marking the first attempt to infer exocentric conversational
+interactions from egocentric videos. We propose a unified multi-modal framework
+-- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of
+conversation behaviors -- speaking and listening -- for both the camera wearer
+as well as all other social partners present in the egocentric video.
+Specifically, we adopt the self-attention mechanism to model the
+representations across-time, across-subjects, and across-modalities. To
+validate our method, we conduct experiments on a challenging egocentric video
+dataset that includes multi-speaker and multi-conversation scenarios. Our
+results demonstrate the superior performance of our method compared to a series
+of baselines. We also present detailed ablation studies to assess the
+contribution of each component in our model. Check our project page at
+https://vjwq.github.io/AV-CONV/.",cs.CV,['cs.CV']
+Dense Vision Transformer Compression with Few Samples,Hanxiao Zhang · Yifan Zhou · Guo-Hua Wang, ,https://arxiv.org/abs/2403.18708,,2403.18708.pdf,Dense Vision Transformer Compression with Few Samples,"Few-shot model compression aims to compress a large model into a more compact
+one with only a tiny training set (even without labels). Block-level pruning
+has recently emerged as a leading technique in achieving high accuracy and low
+latency in few-shot CNN compression. But, few-shot compression for Vision
+Transformers (ViT) remains largely unexplored, which presents a new challenge.
+In particular, the issue of sparse compression exists in traditional CNN
+few-shot methods, which can only produce very few compressed models of
+different model sizes. This paper proposes a novel framework for few-shot ViT
+compression named DC-ViT. Instead of dropping the entire block, DC-ViT
+selectively eliminates the attention module while retaining and reusing
+portions of the MLP module. DC-ViT enables dense compression, which outputs
+numerous compressed models that densely populate the range of model complexity.
+DC-ViT outperforms state-of-the-art few-shot compression methods by a
+significant margin of 10 percentage points, along with lower latency in the
+compression of ViT and its variants.",cs.CV,['cs.CV']
+Structure-from-Motion from Pixel-wise Correspondences,Philipp Lindenberger · Paul-Edouard Sarlin · Marc Pollefeys, ,http://export.arxiv.org/abs/2306.13643,,2306.13643.pdf,LightGlue: Local Feature Matching at Light Speed,"We introduce LightGlue, a deep neural network that learns to match local
+features across images. We revisit multiple design decisions of SuperGlue, the
+state of the art in sparse matching, and derive simple but effective
+improvements. Cumulatively, they make LightGlue more efficient - in terms of
+both memory and computation, more accurate, and much easier to train. One key
+property is that LightGlue is adaptive to the difficulty of the problem: the
+inference is much faster on image pairs that are intuitively easy to match, for
+example because of a larger visual overlap or limited appearance change. This
+opens up exciting prospects for deploying deep matchers in latency-sensitive
+applications like 3D reconstruction. The code and trained models are publicly
+available at https://github.com/cvg/LightGlue.",cs.CV,['cs.CV']
+KITRO: Refining Human Mesh by 2D Clues and Kinematic-tree Rotation,Fengyuan Yang · Kerui Gu · Angela Yao,https://github.com/MartaYang/KITRO,https://arxiv.org/abs/2405.19833,,2405.19833.pdf,KITRO: Refining Human Mesh by 2D Clues and Kinematic-tree Rotation,"2D keypoints are commonly used as an additional cue to refine estimated 3D
+human meshes. Current methods optimize the pose and shape parameters with a
+reprojection loss on the provided 2D keypoints. Such an approach, while simple
+and intuitive, has limited effectiveness because the optimal solution is hard
+to find in ambiguous parameter space and may sacrifice depth. Additionally,
+divergent gradients from distal joints complicate and deviate the refinement of
+proximal joints in the kinematic chain. To address these, we introduce
+Kinematic-Tree Rotation (KITRO), a novel mesh refinement strategy that
+explicitly models depth and human kinematic-tree structure. KITRO treats
+refinement from a bone-wise perspective. Unlike previous methods which perform
+gradient-based optimizations, our method calculates bone directions in closed
+form. By accounting for the 2D pose, bone length, and parent joint's depth, the
+calculation results in two possible directions for each child joint. We then
+use a decision tree to trace binary choices for all bones along the human
+skeleton's kinematic-tree to select the most probable hypothesis. Our
+experiments across various datasets and baseline models demonstrate that KITRO
+significantly improves 3D joint estimation accuracy and achieves an ideal 2D
+fit simultaneously. Our code available at: https://github.com/MartaYang/KITRO.",cs.CV,['cs.CV']
+Orthogonal Adaptation for Modular Customization of Diffusion Models,Ryan Po · Guandao Yang · Kfir Aberman · Gordon Wetzstein, ,https://arxiv.org/abs/2312.02432,,2312.02432.pdf,Orthogonal Adaptation for Modular Customization of Diffusion Models,"Customization techniques for text-to-image models have paved the way for a
+wide range of previously unattainable applications, enabling the generation of
+specific concepts across diverse contexts and styles. While existing methods
+facilitate high-fidelity customization for individual concepts or a limited,
+pre-defined set of them, they fall short of achieving scalability, where a
+single model can seamlessly render countless concepts. In this paper, we
+address a new problem called Modular Customization, with the goal of
+efficiently merging customized models that were fine-tuned independently for
+individual concepts. This allows the merged model to jointly synthesize
+concepts in one image without compromising fidelity or incurring any additional
+computational costs.
+  To address this problem, we introduce Orthogonal Adaptation, a method
+designed to encourage the customized models, which do not have access to each
+other during fine-tuning, to have orthogonal residual weights. This ensures
+that during inference time, the customized models can be summed with minimal
+interference.
+  Our proposed method is both simple and versatile, applicable to nearly all
+optimizable weights in the model architecture. Through an extensive set of
+quantitative and qualitative evaluations, our method consistently outperforms
+relevant baselines in terms of efficiency and identity preservation,
+demonstrating a significant leap toward scalable customization of diffusion
+models.",cs.CV,['cs.CV']
+Open-World Human-Object Interaction Detection via Multi-modal Prompts,Jie Yang · Bingliang Li · Ailing Zeng · Ailing Zeng · Lei Zhang · Ruimao Zhang, ,,https://openreview.net/forum?id=qrv4wcmmxe,,,,,nan
+Adaptive Multi-Modal Cross-Entropy Loss for Stereo Matching,Peng Xu · Zhiyu Xiang · Chengyu Qiao · Jingyun Fu · Tianyu Pu, ,https://arxiv.org/abs/2306.15612,,2306.15612.pdf,Adaptive Multi-Modal Cross-Entropy Loss for Stereo Matching,"Despite the great success of deep learning in stereo matching, recovering
+accurate disparity maps is still challenging. Currently, L1 and cross-entropy
+are the two most widely used losses for stereo network training. Compared with
+the former, the latter usually performs better thanks to its probability
+modeling and direct supervision to the cost volume. However, how to accurately
+model the stereo ground-truth for cross-entropy loss remains largely
+under-explored. Existing works simply assume that the ground-truth
+distributions are uni-modal, which ignores the fact that most of the edge
+pixels can be multi-modal. In this paper, a novel adaptive multi-modal
+cross-entropy loss (ADL) is proposed to guide the networks to learn different
+distribution patterns for each pixel. Moreover, we optimize the disparity
+estimator to further alleviate the bleeding or misalignment artifacts in
+inference. Extensive experimental results show that our method is generic and
+can help classic stereo networks regain state-of-the-art performance. In
+particular, GANet with our method ranks $1^{st}$ on both the KITTI 2015 and
+2012 benchmarks among the published methods. Meanwhile, excellent
+synthetic-to-realistic generalization performance can be achieved by simply
+replacing the traditional loss with ours.",cs.CV,['cs.CV']
+EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,Yaofang Liu · Xiaodong Cun · Xuebo Liu · Xintao Wang · Yong Zhang · Haoxin Chen · Yang Liu · Tieyong Zeng · Raymond Chan · Ying Shan, ,https://arxiv.org/abs/2310.11440,,2310.11440.pdf,EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,"The vision and language generative models have been overgrown in recent
+years. For video generation, various open-sourced models and public-available
+services have been developed to generate high-quality videos. However, these
+methods often use a few metrics, e.g., FVD or IS, to evaluate the performance.
+We argue that it is hard to judge the large conditional generative models from
+the simple metrics since these models are often trained on very large datasets
+with multi-aspect abilities. Thus, we propose a novel framework and pipeline
+for exhaustively evaluating the performance of the generated videos. Our
+approach involves generating a diverse and comprehensive list of 700 prompts
+for text-to-video generation, which is based on an analysis of real-world user
+data and generated with the assistance of a large language model. Then, we
+evaluate the state-of-the-art video generative models on our carefully designed
+benchmark, in terms of visual qualities, content qualities, motion qualities,
+and text-video alignment with 17 well-selected objective metrics. To obtain the
+final leaderboard of the models, we further fit a series of coefficients to
+align the objective metrics to the users' opinions. Based on the proposed human
+alignment method, our final score shows a higher correlation than simply
+averaging the metrics, showing the effectiveness of the proposed evaluation
+method.",cs.CV,['cs.CV']
+HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud,WENCAN CHENG · WENCAN CHENG · Hao Tang · Luc Van Gool · Jong Hwan Ko, ,https://arxiv.org/abs/2404.03159,,2404.03159.pdf,HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud,"Extracting keypoint locations from input hand frames, known as 3D hand pose
+estimation, is a critical task in various human-computer interaction
+applications. Essentially, the 3D hand pose estimation can be regarded as a 3D
+point subset generative problem conditioned on input frames. Thanks to the
+recent significant progress on diffusion-based generative models, hand pose
+estimation can also benefit from the diffusion model to estimate keypoint
+locations with high quality. However, directly deploying the existing diffusion
+models to solve hand pose estimation is non-trivial, since they cannot achieve
+the complex permutation mapping and precise localization. Based on this
+motivation, this paper proposes HandDiff, a diffusion-based hand pose
+estimation model that iteratively denoises accurate hand pose conditioned on
+hand-shaped image-point clouds. In order to recover keypoint permutation and
+accurate location, we further introduce joint-wise condition and local detail
+condition. Experimental results demonstrate that the proposed HandDiff
+significantly outperforms the existing approaches on four challenging hand pose
+benchmark datasets. Codes and pre-trained models are publicly available at
+https://github.com/cwc1260/HandDiff.",cs.CV,['cs.CV']
+Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning,Sicong Shen · Yang Zhou · Bingzheng Wei · Eric Chang · Yan Xu, ,https://arxiv.org/abs/2312.03732,,2312.03732.pdf,A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA,"As large language models (LLMs) have become increasingly compute and memory
+intensive, parameter-efficient fine-tuning (PEFT) methods are now a common
+strategy to fine-tune LLMs. A popular PEFT method is Low-Rank Adapters (LoRA),
+which adds trainable low-rank ""adapters"" to selected layers. Each adapter
+consists of a low-rank matrix product, multiplicatively scaled by a
+rank-dependent factor. This scaling factor, which divides adapters by a factor
+of the rank, results in slowed learning and stunted performance for LoRA with
+higher-rank adapters. Consequently, the use of LoRA in practice has generally
+been limited to very low ranks. In this work, we study the impact of the
+scaling factor on the learning process and prove that LoRA adapters should be
+divided by a factor of the square root of the rank. Modifying LoRA with the
+appropriate scaling factor, which we call the rank-stabilized LoRA (rsLoRA)
+method, easily provides for a fine-tuning compute/performance trade-off, where
+larger ranks can be used to trade off increased computational resources during
+training for better fine-tuning performance, with no change in inference
+computing cost.",cs.CL,"['cs.CL', 'cs.LG', 'I.2.7']"
+En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data,Yifang Men · Biwen Lei · Yuan Yao · Miaomiao Cui · Zhouhui Lian · Xuansong Xie,https://menyifang.github.io/projects/En3D/index.html,https://arxiv.org/abs/2401.01173,,2401.01173.pdf,En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data,"We present En3D, an enhanced generative scheme for sculpting high-quality 3D
+human avatars. Unlike previous works that rely on scarce 3D datasets or limited
+2D collections with imbalanced viewing angles and imprecise pose priors, our
+approach aims to develop a zero-shot 3D generative scheme capable of producing
+visually realistic, geometrically accurate and content-wise diverse 3D humans
+without relying on pre-existing 3D or 2D assets. To address this challenge, we
+introduce a meticulously crafted workflow that implements accurate physical
+modeling to learn the enhanced 3D generative model from synthetic 2D data.
+During inference, we integrate optimization modules to bridge the gap between
+realistic appearances and coarse 3D shapes. Specifically, En3D comprises three
+modules: a 3D generator that accurately models generalizable 3D humans with
+realistic appearance from synthesized balanced, diverse, and structured human
+images; a geometry sculptor that enhances shape quality using multi-view normal
+constraints for intricate human anatomy; and a texturing module that
+disentangles explicit texture maps with fidelity and editability, leveraging
+semantical UV partitioning and a differentiable rasterizer. Experimental
+results show that our approach significantly outperforms prior works in terms
+of image quality, geometry accuracy and content diversity. We also showcase the
+applicability of our generated avatars for animation and editing, as well as
+the scalability of our approach for content-style free adaptation.",cs.CV,['cs.CV']
+Differentiable Point-based Inverse Rendering,Hoon-Gyu Chung · Seokjun Choi · Seung-Hwan Baek,https://hg-chung.github.io/DPIR/,https://arxiv.org/abs/2312.02480,,2312.02480.pdf,Differentiable Point-based Inverse Rendering,"We present differentiable point-based inverse rendering, DPIR, an
+analysis-by-synthesis method that processes images captured under diverse
+illuminations to estimate shape and spatially-varying BRDF. To this end, we
+adopt point-based rendering, eliminating the need for multiple samplings per
+ray, typical of volumetric rendering, thus significantly enhancing the speed of
+inverse rendering. To realize this idea, we devise a hybrid point-volumetric
+representation for geometry and a regularized basis-BRDF representation for
+reflectance. The hybrid geometric representation enables fast rendering through
+point-based splatting while retaining the geometric details and stability
+inherent to SDF-based representations. The regularized basis-BRDF mitigates the
+ill-posedness of inverse rendering stemming from limited light-view angular
+samples. We also propose an efficient shadow detection method using point-based
+shadow map rendering. Our extensive evaluations demonstrate that DPIR
+outperforms prior works in terms of reconstruction accuracy, computational
+efficiency, and memory footprint. Furthermore, our explicit point-based
+representation and rendering enables intuitive geometry and reflectance
+editing.",cs.CV,['cs.CV']
+ICP-Flow: LiDAR Scene Flow Estimation with ICP,Yancong Lin · Holger Caesar,https://github.com/yanconglin/ICP-Flow,https://arxiv.org/abs/2402.17351,,2402.17351.pdf,ICP-Flow: LiDAR Scene Flow Estimation with ICP,"Scene flow characterizes the 3D motion between two LiDAR scans captured by an
+autonomous vehicle at nearby timesteps. Prevalent methods consider scene flow
+as point-wise unconstrained flow vectors that can be learned by either
+large-scale training beforehand or time-consuming optimization at inference.
+However, these methods do not take into account that objects in autonomous
+driving often move rigidly. We incorporate this rigid-motion assumption into
+our design, where the goal is to associate objects over scans and then estimate
+the locally rigid transformations. We propose ICP-Flow, a learning-free flow
+estimator. The core of our design is the conventional Iterative Closest Point
+(ICP) algorithm, which aligns the objects over time and outputs the
+corresponding rigid transformations. Crucially, to aid ICP, we propose a
+histogram-based initialization that discovers the most likely translation, thus
+providing a good starting point for ICP. The complete scene flow is then
+recovered from the rigid transformations. We outperform state-of-the-art
+baselines, including supervised models, on the Waymo dataset and perform
+competitively on Argoverse-v2 and nuScenes. Further, we train a feedforward
+neural network, supervised by the pseudo labels from our model, and achieve top
+performance among all models capable of real-time inference. We validate the
+advantage of our model on scene flow estimation with longer temporal gaps, up
+to 0.4 seconds where other models fail to deliver meaningful results.",cs.CV,['cs.CV']
+Rolling Shutter Correction with Intermediate Distortion Flow Estimation,Mingdeng Cao · Sidi Yang · Yujiu Yang · Yinqiang Zheng,https://github.com/ljzycmd/DFRSC,https://arxiv.org/abs/2404.06350,,2404.06350.pdf,Rolling Shutter Correction with Intermediate Distortion Flow Estimation,"This paper proposes to correct the rolling shutter (RS) distorted images by
+estimating the distortion flow from the global shutter (GS) to RS directly.
+Existing methods usually perform correction using the undistortion flow from
+the RS to GS. They initially predict the flow from consecutive RS frames,
+subsequently rescaling it as the displacement fields from the RS frame to the
+underlying GS image using time-dependent scaling factors. Following this,
+RS-aware forward warping is employed to convert the RS image into its GS
+counterpart. Nevertheless, this strategy is prone to two shortcomings. First,
+the undistortion flow estimation is rendered inaccurate by merely linear
+scaling the flow, due to the complex non-linear motion nature. Second, RS-aware
+forward warping often results in unavoidable artifacts. To address these
+limitations, we introduce a new framework that directly estimates the
+distortion flow and rectifies the RS image with the backward warping operation.
+More specifically, we first propose a global correlation-based flow attention
+mechanism to estimate the initial distortion flow and GS feature jointly, which
+are then refined by the following coarse-to-fine decoder layers. Additionally,
+a multi-distortion flow prediction strategy is integrated to mitigate the issue
+of inaccurate flow estimation further. Experimental results validate the
+effectiveness of the proposed method, which outperforms state-of-the-art
+approaches on various benchmarks while maintaining high efficiency. The project
+is available at \url{https://github.com/ljzycmd/DFRSC}.",cs.CV,['cs.CV']
+Programmable Motion Generation for Open-set Motion Control Tasks,Hanchao Liu · Xiaohang Zhan · Shaoli Huang · Tai-Jiang Mu · Ying Shan, ,https://arxiv.org/abs/2405.19283,,2405.19283.pdf,Programmable Motion Generation for Open-Set Motion Control Tasks,"Character animation in real-world scenarios necessitates a variety of
+constraints, such as trajectories, key-frames, interactions, etc. Existing
+methodologies typically treat single or a finite set of these constraint(s) as
+separate control tasks. They are often specialized, and the tasks they address
+are rarely extendable or customizable. We categorize these as solutions to the
+close-set motion control problem. In response to the complexity of practical
+motion control, we propose and attempt to solve the open-set motion control
+problem. This problem is characterized by an open and fully customizable set of
+motion control tasks. To address this, we introduce a new paradigm,
+programmable motion generation. In this paradigm, any given motion control task
+is broken down into a combination of atomic constraints. These constraints are
+then programmed into an error function that quantifies the degree to which a
+motion sequence adheres to them. We utilize a pre-trained motion generation
+model and optimize its latent code to minimize the error function of the
+generated motion. Consequently, the generated motion not only inherits the
+prior of the generative model but also satisfies the required constraints.
+Experiments show that we can generate high-quality motions when addressing a
+wide range of unseen tasks. These tasks encompass motion control by motion
+dynamics, geometric constraints, physical laws, interactions with scenes,
+objects or the character own body parts, etc. All of these are achieved in a
+unified approach, without the need for ad-hoc paired training data collection
+or specialized network designs. During the programming of novel tasks, we
+observed the emergence of new skills beyond those of the prior model. With the
+assistance of large language models, we also achieved automatic programming. We
+hope that this work will pave the way for the motion control of general AI
+agents.",cs.CV,['cs.CV']
+Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation,Bingxin Ke · Anton Obukhov · Shengyu Huang · Nando Metzger · Rodrigo Caye Daudt · Konrad Schindler, ,https://arxiv.org/abs/2312.02145,,2312.02145.pdf,Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation,"Monocular depth estimation is a fundamental computer vision task. Recovering
+3D depth from a single image is geometrically ill-posed and requires scene
+understanding, so it is not surprising that the rise of deep learning has led
+to a breakthrough. The impressive progress of monocular depth estimators has
+mirrored the growth in model capacity, from relatively modest CNNs to large
+Transformer architectures. Still, monocular depth estimators tend to struggle
+when presented with images with unfamiliar content and layout, since their
+knowledge of the visual world is restricted by the data seen during training,
+and challenged by zero-shot generalization to new domains. This motivates us to
+explore whether the extensive priors captured in recent generative diffusion
+models can enable better, more generalizable depth estimation. We introduce
+Marigold, a method for affine-invariant monocular depth estimation that is
+derived from Stable Diffusion and retains its rich prior knowledge. The
+estimator can be fine-tuned in a couple of days on a single GPU using only
+synthetic training data. It delivers state-of-the-art performance across a wide
+range of datasets, including over 20% performance gains in specific cases.
+Project page: https://marigoldmonodepth.github.io.",cs.CV,['cs.CV']
+I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions,Chengfeng Zhao · Juze Zhang · Jiashen Du · Ziwei Shan · Junye Wang · Jingyi Yu · Jingya Wang · Lan Xu, ,https://arxiv.org/abs/2312.08869,,2312.08869.pdf,I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions,"We are living in a world surrounded by diverse and ""smart"" devices with rich
+modalities of sensing ability. Conveniently capturing the interactions between
+us humans and these objects remains far-reaching. In this paper, we present
+I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the
+human and object in a novel setting: using a minimal amount of RGB camera and
+object-mounted Inertial Measurement Unit (IMU). It combines general motion
+inference and category-aware refinement. For the former, we introduce a
+holistic human-object tracking method to fuse the IMU signals and the RGB
+stream and progressively recover the human motions and subsequently the
+companion object motions. For the latter, we tailor a category-aware motion
+diffusion model, which is conditioned on both the raw IMU observations and the
+results from the previous stage under over-parameterization representation. It
+significantly refines the initial results and generates vivid body, hand, and
+object motions. Moreover, we contribute a large dataset with ground truth human
+and object motions, dense RGB inputs, and rich object-mounted IMU measurements.
+Extensive experiments demonstrate the effectiveness of I'm-HOI under a hybrid
+capture setting. Our dataset and code will be released to the community.",cs.CV,['cs.CV']
+From a Bird’s Eye View to See: Joint Camera and Subject Registration without the Camera Calibration,Zekun Qian · Ruize Han · Wei Feng · Song Wang,https://github.com/zekunqian/bevsee,,https://allainews.com/item/from-a-birds-eye-view-to-see-joint-camera-and-subject-registration-without-the-camera-calibration-2024-04-30/,,,,,nan
+LMDrive: Closed-Loop End-to-End Driving  with Large Language Models,Hao Shao · Yuxuan Hu · Letian Wang · Guanglu Song · Steven L. Waslander · Yu Liu · Hongsheng Li, ,https://arxiv.org/abs/2312.07488,,2312.07488.pdf,LMDrive: Closed-Loop End-to-End Driving with Large Language Models,"Despite significant recent progress in the field of autonomous driving,
+modern methods still struggle and can incur serious accidents when encountering
+long-tail unforeseen events and challenging urban scenarios. On the one hand,
+large language models (LLM) have shown impressive reasoning capabilities that
+approach ""Artificial General Intelligence"". On the other hand, previous
+autonomous driving methods tend to rely on limited-format inputs (e.g. sensor
+data and navigation waypoints), restricting the vehicle's ability to understand
+language information and interact with humans. To this end, this paper
+introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous
+driving framework. LMDrive uniquely processes and integrates multi-modal sensor
+data with natural language instructions, enabling interaction with humans and
+navigation software in realistic instructional settings. To facilitate further
+research in language-based closed-loop autonomous driving, we also publicly
+release the corresponding dataset which includes approximately 64K
+instruction-following data clips, and the LangAuto benchmark that tests the
+system's ability to handle complex instructions and challenging driving
+scenarios. Extensive closed-loop experiments are conducted to demonstrate
+LMDrive's effectiveness. To the best of our knowledge, we're the very first
+work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes,
+models, and datasets can be found at https://github.com/opendilab/LMDrive",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+TUMTraf V2X Cooperative Perception Dataset,Walter Zimmer · Gerhard Arya Wardana · Suren Sritharan · Xingcheng Zhou · Rui Song · Alois Knoll,https://tum-traffic-dataset.github.io/tumtraf-v2x,https://arxiv.org/abs/2403.01316,,2403.01316.pdf,TUMTraf V2X Cooperative Perception Dataset,"Cooperative perception offers several benefits for enhancing the capabilities
+of autonomous vehicles and improving road safety. Using roadside sensors in
+addition to onboard sensors increases reliability and extends the sensor range.
+External sensors offer higher situational awareness for automated vehicles and
+prevent occlusions. We propose CoopDet3D, a cooperative multi-modal fusion
+model, and TUMTraf-V2X, a perception dataset, for the cooperative 3D object
+detection and tracking task. Our dataset contains 2,000 labeled point clouds
+and 5,000 labeled images from five roadside and four onboard sensors. It
+includes 30k 3D boxes with track IDs and precise GPS and IMU data. We labeled
+eight categories and covered occlusion scenarios with challenging driving
+maneuvers, like traffic violations, near-miss events, overtaking, and U-turns.
+Through multiple experiments, we show that our CoopDet3D camera-LiDAR fusion
+model achieves an increase of +14.36 3D mAP compared to a vehicle camera-LiDAR
+fusion model. Finally, we make our dataset, model, labeling tool, and dev-kit
+publicly available on our website:
+https://tum-traffic-dataset.github.io/tumtraf-v2x.",cs.CV,['cs.CV']
+Unleashing Unlabeled Data: A Paradigm for Cross-View Geo-Localization,Guopeng Li · Ming Qian · Gui-Song Xia, ,https://arxiv.org/abs/2403.14198v1,,2403.14198v1.pdf,Unleashing Unlabeled Data: A Paradigm for Cross-View Geo-Localization,"This paper investigates the effective utilization of unlabeled data for
+large-area cross-view geo-localization (CVGL), encompassing both unsupervised
+and semi-supervised settings. Common approaches to CVGL rely on
+ground-satellite image pairs and employ label-driven supervised training.
+However, the cost of collecting precise cross-view image pairs hinders the
+deployment of CVGL in real-life scenarios. Without the pairs, CVGL will be more
+challenging to handle the significant imaging and spatial gaps between ground
+and satellite images. To this end, we propose an unsupervised framework
+including a cross-view projection to guide the model for retrieving initial
+pseudo-labels and a fast re-ranking mechanism to refine the pseudo-labels by
+leveraging the fact that ``the perfectly paired ground-satellite image is
+located in a unique and identical scene"". The framework exhibits competitive
+performance compared with supervised works on three open-source benchmarks. Our
+code and models will be released on https://github.com/liguopeng0923/UCVGL.",cs.CV,['cs.CV']
+Deep Imbalanced Regression via Hierarchical Classification Adjustment,Haipeng Xiong · Angela Yao, ,https://arxiv.org/abs/2310.17154,,2310.17154.pdf,Deep Imbalanced Regression via Hierarchical Classification Adjustment,"Regression tasks in computer vision, such as age estimation or counting, are
+often formulated into classification by quantizing the target space into
+classes. Yet real-world data is often imbalanced -- the majority of training
+samples lie in a head range of target values, while a minority of samples span
+a usually larger tail range. By selecting the class quantization, one can
+adjust imbalanced regression targets into balanced classification outputs,
+though there are trade-offs in balancing classification accuracy and
+quantization error. To improve regression performance over the entire range of
+data, we propose to construct hierarchical classifiers for solving imbalanced
+regression tasks. The fine-grained classifiers limit the quantization error
+while being modulated by the coarse predictions to ensure high accuracy.
+Standard hierarchical classification approaches, however, when applied to the
+regression problem, fail to ensure that predicted ranges remain consistent
+across the hierarchy. As such, we propose a range-preserving distillation
+process that can effectively learn a single classifier from the set of
+hierarchical classifiers. Our novel hierarchical classification adjustment
+(HCA) for imbalanced regression shows superior results on three diverse tasks:
+age estimation, crowd counting and depth estimation. We will release the source
+code upon acceptance.",cs.CV,['cs.CV']
+Ensemble Diversity Facilitates Adversarial Transferability,Bowen Tang · Zheng Wang · Yi Bin · Qi Dou · Yang Yang · Heng Tao Shen, ,https://arxiv.org/abs/2403.16405,,2403.16405.pdf,Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Models,"The integration of an ensemble of deep learning models has been extensively
+explored to enhance defense against adversarial attacks. The diversity among
+sub-models increases the attack cost required to deceive the majority of the
+ensemble, thereby improving the adversarial robustness. While existing
+approaches mainly center on increasing diversity in feature representations or
+dispersion of first-order gradients with respect to input, the limited
+correlation between these diversity metrics and adversarial robustness
+constrains the performance of ensemble adversarial defense. In this work, we
+aim to enhance ensemble diversity by reducing attack transferability. We
+identify second-order gradients, which depict the loss curvature, as a key
+factor in adversarial robustness. Computing the Hessian matrix involved in
+second-order gradients is computationally expensive. To address this, we
+approximate the Hessian-vector product using differential approximation. Given
+that low curvature provides better robustness, our ensemble model was designed
+to consider the influence of curvature among different sub-models. We introduce
+a novel regularizer to train multiple more-diverse low-curvature network
+models. Extensive experiments across various datasets demonstrate that our
+ensemble model exhibits superior robustness against a range of attacks,
+underscoring the effectiveness of our approach.",cs.LG,"['cs.LG', 'cs.CR', 'cs.CV']"
+DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation,Haonan Lin, ,https://arxiv.org/abs/2403.19235,,2403.19235.pdf,DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation,"While large-scale pre-trained text-to-image models can synthesize diverse and
+high-quality human-centered images, novel challenges arise with a nuanced task
+of ""identity fine editing"": precisely modifying specific features of a subject
+while maintaining its inherent identity and context. Existing personalization
+methods either require time-consuming optimization or learning additional
+encoders, adept in ""identity re-contextualization"". However, they often
+struggle with detailed and sensitive tasks like human face editing. To address
+these challenges, we introduce DreamSalon, a noise-guided, staged-editing
+framework, uniquely focusing on detailed image manipulations and
+identity-context preservation. By discerning editing and boosting stages via
+the frequency and gradient of predicted noises, DreamSalon first performs
+detailed manipulations on specific features in the editing stage, guided by
+high-frequency information, and then employs stochastic denoising in the
+boosting stage to improve image quality. For more precise editing, DreamSalon
+semantically mixes source and target textual prompts, guided by differences in
+their embedding covariances, to direct the model's focus on specific
+manipulation areas. Our experiments demonstrate DreamSalon's ability to
+efficiently and faithfully edit fine details on human faces, outperforming
+existing methods both qualitatively and quantitatively.",cs.CV,['cs.CV']
+RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation,Zeyuan Yang · LIU JIAGENG · Peihao Chen · Anoop Cherian · Tim Marks · Jonathan Le Roux · Chuang Gan, ,,https://github.com/zchoi/Awesome-Embodied-Agent-with-LLMs,,,,,nan
+FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models,Adrian Bulat · Yassine Ouali · Georgios Tzimiropoulos, ,https://arxiv.org/abs/2405.10286,,2405.10286.pdf,FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models,"Despite noise and caption quality having been acknowledged as important
+factors impacting vision-language contrastive pre-training, in this paper, we
+show that the full potential of improving the training process by addressing
+such issues is yet to be realized. Specifically, we firstly study and analyze
+two issues affecting training: incorrect assignment of negative pairs, and low
+caption quality and diversity. Then, we devise effective solutions for
+addressing both problems, which essentially require training with multiple true
+positive pairs. Finally, we propose training with sigmoid loss to address such
+a requirement. We show very large gains over the current state-of-the-art for
+both image recognition ($\sim +6\%$ on average over 11 datasets) and image
+retrieval ($\sim +19\%$ on Flickr30k and $\sim +15\%$ on MSCOCO).",cs.CV,"['cs.CV', 'cs.AI']"
+Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning,Da-Wei Zhou · Hai-Long Sun · Han-Jia Ye · De-Chuan Zhan,https://github.com/sun-hailong/CVPR24-Ease,https://arxiv.org/abs/2403.12030v1,,2403.12030v1.pdf,Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning,"Class-Incremental Learning (CIL) requires a learning system to continually
+learn new classes without forgetting. Despite the strong performance of
+Pre-Trained Models (PTMs) in CIL, a critical issue persists: learning new
+classes often results in the overwriting of old ones. Excessive modification of
+the network causes forgetting, while minimal adjustments lead to an inadequate
+fit for new classes. As a result, it is desired to figure out a way of
+efficient model updating without harming former knowledge. In this paper, we
+propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. To enable model
+updating without conflict, we train a distinct lightweight adapter module for
+each new task, aiming to create task-specific subspaces. These adapters span a
+high-dimensional feature space, enabling joint decision-making across multiple
+subspaces. As data evolves, the expanding subspaces render the old class
+classifiers incompatible with new-stage spaces. Correspondingly, we design a
+semantic-guided prototype complement strategy that synthesizes old classes' new
+features without using any old class instance. Extensive experiments on seven
+benchmark datasets verify EASE's state-of-the-art performance. Code is
+available at: https://github.com/sun-hailong/CVPR24-Ease",cs.CV,"['cs.CV', 'cs.LG']"
+Generating Handwritten Mathematical Expressions From Symbol Graphs: An End-to-End Pipeline,Yu chen · Fei Gao · YanguangZhang · Maoying Qiao · Nannan Wang,https://github.com/AiArt-HDU/HMEG,,https://link.springer.com/chapter/10.1007/978-3-031-41676-7_9,,,,,nan
+AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation,Jeongsoo Choi · Se Jin Park · Minsu Kim · Yong Man Ro, ,https://arxiv.org/html/2312.02512v2,,2312.02512v2.pdf,AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation,"This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech
+Translation (AV2AV) framework, where the input and output of the system are
+multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key
+advantages can be brought: 1) We can perform real-like conversations with
+individuals worldwide in a virtual meeting by utilizing our own primary
+languages. In contrast to Speech-to-Speech Translation (A2A), which solely
+translates between audio modalities, the proposed AV2AV directly translates
+between audio-visual speech. This capability enhances the dialogue experience
+by presenting synchronized lip movements along with the translated speech. 2)
+We can improve the robustness of the spoken language translation system. By
+employing the complementary information of audio-visual speech, the system can
+effectively translate spoken language even in the presence of acoustic noise,
+showcasing robust performance. To mitigate the problem of the absence of a
+parallel AV2AV translation dataset, we propose to train our spoken language
+translation system with the audio-only dataset of A2A. This is done by learning
+unified audio-visual speech representations through self-supervised learning in
+advance to train the translation system. Moreover, we propose an AV-Renderer
+that can generate raw audio and video in parallel. It is designed with
+zero-shot speaker modeling, thus the speaker in source audio-visual speech can
+be maintained at the target translated audio-visual speech. The effectiveness
+of AV2AV is evaluated with extensive experiments in a many-to-many language
+translation setting. Demo page is available on
+https://choijeongsoo.github.io/av2av.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM', 'eess.AS']"
+DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation,Zeeshan Hayder · Xuming He,https://zeeshanhayder.github.io/DSGG,https://arxiv.org/abs/2403.14886,,2403.14886.pdf,DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation,"Scene graph generation aims to capture detailed spatial and semantic
+relationships between objects in an image, which is challenging due to
+incomplete labelling, long-tailed relationship categories, and relational
+semantic overlap. Existing Transformer-based methods either employ distinct
+queries for objects and predicates or utilize holistic queries for relation
+triplets and hence often suffer from limited capacity in learning low-frequency
+relationships. In this paper, we present a new Transformer-based method, called
+DSGG, that views scene graph detection as a direct graph prediction problem
+based on a unique set of graph-aware queries. In particular, each graph-aware
+query encodes a compact representation of both the node and all of its
+relations in the graph, acquired through the utilization of a relaxed sub-graph
+matching during the training process. Moreover, to address the problem of
+relational semantic overlap, we utilize a strategy for relation distillation,
+aiming to efficiently learn multiple instances of semantic relationships.
+Extensive experiments on the VG and the PSG datasets show that our model
+achieves state-of-the-art results, showing a significant improvement of 3.5\%
+and 6.7\% in mR@50 and mR@100 for the scene-graph generation task and achieves
+an even more substantial improvement of 8.5\% and 10.3\% in mR@50 and mR@100
+for the panoptic scene graph generation task. Code is available at
+\url{https://github.com/zeeshanhayder/DSGG}.",cs.CV,['cs.CV']
+Learn from View Correlation: An Anchor Enhancement Strategy for Multi-view Clustering,Suyuan Liu · KE LIANG · Zhibin Dong · Siwei Wang · Xihong Yang · sihang zhou �� En Zhu · Xinwang Liu, ,https://arxiv.org/abs/2309.00024,,2309.00024.pdf,Efficient Multi-View Graph Clustering with Local and Global Structure Preservation,"Anchor-based multi-view graph clustering (AMVGC) has received abundant
+attention owing to its high efficiency and the capability to capture
+complementary structural information across multiple views. Intuitively, a
+high-quality anchor graph plays an essential role in the success of AMVGC.
+However, the existing AMVGC methods only consider single-structure information,
+i.e., local or global structure, which provides insufficient information for
+the learning task. To be specific, the over-scattered global structure leads to
+learned anchors failing to depict the cluster partition well. In contrast, the
+local structure with an improper similarity measure results in potentially
+inaccurate anchor assignment, ultimately leading to sub-optimal clustering
+performance. To tackle the issue, we propose a novel anchor-based multi-view
+graph clustering framework termed Efficient Multi-View Graph Clustering with
+Local and Global Structure Preservation (EMVGC-LG). Specifically, a unified
+framework with a theoretical guarantee is designed to capture local and global
+information. Besides, EMVGC-LG jointly optimizes anchor construction and graph
+learning to enhance the clustering quality. In addition, EMVGC-LG inherits the
+linear complexity of existing AMVGC methods respecting the sample number, which
+is time-economical and scales well with the data size. Extensive experiments
+demonstrate the effectiveness and efficiency of our proposed method.",cs.LG,['cs.LG']
+SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation,Jiaben Chen · Huaizu Jiang, ,https://arxiv.org/abs/2308.16876v2,,2308.16876v2.pdf,SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation,"Human-centric video frame interpolation has great potential for improving
+people's entertainment experiences and finding commercial applications in the
+sports analysis industry, e.g., synthesizing slow-motion videos. Although there
+are multiple benchmark datasets available in the community, none of them is
+dedicated for human-centric scenarios. To bridge this gap, we introduce
+SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video
+frames of high-resolution ($\geq$720p) slow-motion sports videos crawled from
+YouTube. We re-train several state-of-the-art methods on our benchmark, and the
+results show a decrease in their accuracy compared to other datasets. It
+highlights the difficulty of our benchmark and suggests that it poses
+significant challenges even for the best-performing methods, as human bodies
+are highly deformable and occlusions are frequent in sports videos. To improve
+the accuracy, we introduce two loss terms considering the human-aware priors,
+where we add auxiliary supervision to panoptic segmentation and human keypoints
+detection, respectively. The loss terms are model agnostic and can be easily
+plugged into any video frame interpolation approaches. Experimental results
+validate the effectiveness of our proposed loss terms, leading to consistent
+performance improvement over 5 existing models, which establish strong baseline
+models on our benchmark. The dataset and code can be found at:
+https://neu-vi.github.io/SportsSlomo/.",cs.CV,['cs.CV']
+G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images,Zixiong Huang · Qi Chen · Libo Sun · Yifan Yang · Naizhou Wang · Qi Wu · Mingkui Tan, ,https://arxiv.org/abs/2404.07474,,2404.07474.pdf,G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images,"Novel view synthesis aims to generate new view images of a given view image
+collection. Recent attempts address this problem relying on 3D geometry priors
+(e.g., shapes, sizes, and positions) learned from multi-view images. However,
+such methods encounter the following limitations: 1) they require a set of
+multi-view images as training data for a specific scene (e.g., face, car or
+chair), which is often unavailable in many real-world scenarios; 2) they fail
+to extract the geometry priors from single-view images due to the lack of
+multi-view supervision. In this paper, we propose a Geometry-enhanced NeRF
+(G-NeRF), which seeks to enhance the geometry priors by a geometry-guided
+multi-view synthesis approach, followed by a depth-aware training. In the
+synthesis process, inspired that existing 3D GAN models can unconditionally
+synthesize high-fidelity multi-view images, we seek to adopt off-the-shelf 3D
+GAN models, such as EG3D, as a free source to provide geometry priors through
+synthesizing multi-view data. Simultaneously, to further improve the geometry
+quality of the synthetic data, we introduce a truncation method to effectively
+sample latent codes within 3D GAN models. To tackle the absence of multi-view
+supervision for single-view images, we design the depth-aware training
+approach, incorporating a depth-aware discriminator to guide geometry priors
+through depth maps. Experiments demonstrate the effectiveness of our method in
+terms of both qualitative and quantitative results.",cs.CV,['cs.CV']
+MaskPLAN: Masked Generative Layout Planning from Partial Input,Hang Zhang · Anton Savov · Benjamin Dillenburger, ,https://arxiv.org/abs/2312.05039,,2312.05039.pdf,SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control,"The field of generative image inpainting and object insertion has made
+significant progress with the recent advent of latent diffusion models.
+Utilizing a precise object mask can greatly enhance these applications.
+However, due to the challenges users encounter in creating high-fidelity masks,
+there is a tendency for these methods to rely on more coarse masks (e.g.,
+bounding box) for these applications. This results in limited control and
+compromised background content preservation. To overcome these limitations, we
+introduce SmartMask, which allows any novice user to create detailed masks for
+precise object insertion. Combined with a ControlNet-Inpaint model, our
+experiments demonstrate that SmartMask achieves superior object insertion
+quality, preserving the background content more effectively than previous
+methods. Notably, unlike prior works the proposed approach can also be used
+even without user-mask guidance, which allows it to perform mask-free object
+insertion at diverse positions and scales. Furthermore, we find that when used
+iteratively with a novel instruction-tuning based planning model, SmartMask can
+be used to design detailed layouts from scratch. As compared with user-scribble
+based layout design, we observe that SmartMask allows for better quality
+outputs with layout-to-image generation methods. Project page is available at
+https://smartmask-gen.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.HC', 'cs.LG', 'cs.MM']"
+OneLLM: One Framework to Align All Modalities with Language,Jiaming Han · Kaixiong Gong · Yiyuan Zhang · Jiaqi Wang · Kaipeng Zhang · Dahua Lin · Yu Qiao · Peng Gao · Xiangyu Yue, ,https://arxiv.org/abs/2312.03700,,2312.03700.pdf,OneLLM: One Framework to Align All Modalities with Language,"Multimodal large language models (MLLMs) have gained significant attention
+due to their strong multimodal understanding capability. However, existing
+works rely heavily on modality-specific encoders, which usually differ in
+architecture and are limited to common modalities. In this paper, we present
+OneLLM, an MLLM that aligns eight modalities to language using a unified
+framework. We achieve this through a unified multimodal encoder and a
+progressive multimodal alignment pipeline. In detail, we first train an image
+projection module to connect a vision encoder with LLM. Then, we build a
+universal projection module (UPM) by mixing multiple image projection modules
+and dynamic routing. Finally, we progressively align more modalities to LLM
+with the UPM. To fully leverage the potential of OneLLM in following
+instructions, we also curated a comprehensive multimodal instruction dataset,
+including 2M items from image, audio, video, point cloud, depth/normal map, IMU
+and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks,
+encompassing tasks such as multimodal captioning, question answering and
+reasoning, where it delivers excellent performance. Code, data, model and
+online demo are available at https://github.com/csuhan/OneLLM",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG', 'cs.MM']"
+Open-World Semantic Segmentation Including Class Similarity,Matteo Sodano · Federico Magistri · Lucas Nunes · Jens Behley · Cyrill Stachniss, ,https://arxiv.org/abs/2403.07532,,2403.07532.pdf,Open-World Semantic Segmentation Including Class Similarity,"Interpreting camera data is key for autonomously acting systems, such as
+autonomous vehicles. Vision systems that operate in real-world environments
+must be able to understand their surroundings and need the ability to deal with
+novel situations. This paper tackles open-world semantic segmentation, i.e.,
+the variant of interpreting image data in which objects occur that have not
+been seen during training. We propose a novel approach that performs accurate
+closed-world semantic segmentation and, at the same time, can identify new
+categories without requiring any additional training data. Our approach
+additionally provides a similarity measure for every newly discovered class in
+an image to a known category, which can be useful information in downstream
+tasks such as planning or mapping. Through extensive experiments, we show that
+our model achieves state-of-the-art results on classes known from training data
+as well as for anomaly segmentation and can distinguish between different
+unknown classes.",cs.CV,['cs.CV']
+MovieChat: From Dense Token to Sparse Memory for Long Video Understanding,Enxin Song · Wenhao Chai · Guanhong Wang · Haoyang Zhou · Feiyang Wu · Yucheng Zhang · Tian Ye · Haozhe Chi · Xun Guo · Yanting Zhang · Yan Lu · Jenq-Neng Hwang · Gaoang Wang, ,https://arxiv.org/abs/2307.16449,,2307.16449.pdf,MovieChat: From Dense Token to Sparse Memory for Long Video Understanding,"Recently, integrating video foundation models and large language models to
+build a video understanding system can overcome the limitations of specific
+pre-defined vision tasks. Yet, existing systems can only handle videos with
+very few frames. For long videos, the computation complexity, memory cost, and
+long-term temporal connection impose additional challenges. Taking advantage of
+the Atkinson-Shiffrin memory model, with tokens in Transformers being employed
+as the carriers of memory in combination with our specially designed memory
+mechanism, we propose the MovieChat to overcome these challenges. MovieChat
+achieves state-of-the-art performance in long video understanding, along with
+the released MovieChat-1K benchmark with 1K long video and 14K manual
+annotations for validation of the effectiveness of our method.",cs.CV,['cs.CV']
+Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models,Haoning Wu · Zicheng Zhang · Erli Zhang · Chaofeng Chen · Liang Liao · Annan Wang · Kaixin Xu · Chunyi Li · Jingwen Hou · Guangtao Zhai · Xue Geng · Wenxiu Sun · Qiong Yan · Weisi Lin,https://q-future.github.io/Q-Instruct,https://arxiv.org/abs/2311.06783,,2311.06783.pdf,Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models,"Multi-modality foundation models, as represented by GPT-4V, have brought a
+new paradigm for low-level visual perception and understanding tasks, that can
+respond to a broad range of natural human instructions in a model. While
+existing foundation models have shown exciting potentials on low-level visual
+tasks, their related abilities are still preliminary and need to be improved.
+In order to enhance these models, we conduct a large-scale subjective
+experiment collecting a vast number of real human feedbacks on low-level
+vision. Each feedback follows a pathway that starts with a detailed description
+on the low-level visual appearance (*e.g. clarity, color, brightness* of an
+image, and ends with an overall conclusion, with an average length of 45 words.
+The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
+18,973 images with diverse low-level appearance. Moreover, to enable foundation
+models to robustly respond to diverse types of questions, we design a
+GPT-participated conversion to process these feedbacks into diverse-format 200K
+instruction-response pairs. Experimental results indicate that the
+**Q-Instruct** consistently elevates low-level perception and understanding
+abilities across several foundational models. We anticipate that our datasets
+can pave the way for a future that general intelligence can perceive,
+understand low-level visual appearance and evaluate visual quality like a
+human. Our dataset, model zoo, and demo is published at:
+https://q-future.github.io/Q-Instruct.",cs.CV,"['cs.CV', 'cs.MM']"
+WaveFace: Authentic Face Restoration with Efficient Frequency Recovery,Yunqi Miao · Jiankang Deng · Jungong Han,https://yoqim.github.io/waveface_page/,https://arxiv.org/abs/2403.12760,,2403.12760.pdf,WaveFace: Authentic Face Restoration with Efficient Frequency Recovery,"Although diffusion models are rising as a powerful solution for blind face
+restoration, they are criticized for two problems: 1) slow training and
+inference speed, and 2) failure in preserving identity and recovering
+fine-grained facial details. In this work, we propose WaveFace to solve the
+problems in the frequency domain, where low- and high-frequency components
+decomposed by wavelet transformation are considered individually to maximize
+authenticity as well as efficiency. The diffusion model is applied to recover
+the low-frequency component only, which presents general information of the
+original image but 1/16 in size. To preserve the original identity, the
+generation is conditioned on the low-frequency component of low-quality images
+at each denoising step. Meanwhile, high-frequency components at multiple
+decomposition levels are handled by a unified network, which recovers complex
+facial details in a single step. Evaluations on four benchmark datasets show
+that: 1) WaveFace outperforms state-of-the-art methods in authenticity,
+especially in terms of identity preservation, and 2) authentic images are
+restored with the efficiency 10x faster than existing diffusion model-based BFR
+methods.",cs.CV,['cs.CV']
+MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection,Boyang Peng · Sanqing Qu · Yong Wu · Tianpei Zou · Lianghua He · Alois Knoll · Guang Chen · Changjun Jiang,https://github.com/ispc-lab/MAP,https://arxiv.org/abs/2403.04149,,2403.04149.pdf,MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection,"Deep learning has achieved remarkable progress in various applications,
+heightening the importance of safeguarding the intellectual property (IP) of
+well-trained models. It entails not only authorizing usage but also ensuring
+the deployment of models in authorized data domains, i.e., making models
+exclusive to certain target domains. Previous methods necessitate concurrent
+access to source training data and target unauthorized data when performing IP
+protection, making them risky and inefficient for decentralized private data.
+In this paper, we target a practical setting where only a well-trained source
+model is available and investigate how we can realize IP protection. To achieve
+this, we propose a novel MAsk Pruning (MAP) framework. MAP stems from an
+intuitive hypothesis, i.e., there are target-related parameters in a
+well-trained model, locating and pruning them is the key to IP protection.
+Technically, MAP freezes the source model and learns a target-specific binary
+mask to prevent unauthorized data usage while minimizing performance
+degradation on authorized data. Moreover, we introduce a new metric aimed at
+achieving a better balance between source and target performance degradation.
+To verify the effectiveness and versatility, we have evaluated MAP in a variety
+of scenarios, including vanilla source-available, practical source-free, and
+challenging data-free. Extensive experiments indicate that MAP yields new
+state-of-the-art performance.",cs.CV,['cs.CV']
+Unsegment Anything by Simulating Deformation,Jiahao Lu · Xingyi Yang · Xinchao Wang, ,https://arxiv.org/abs/2404.02585,,2404.02585.pdf,Unsegment Anything by Simulating Deformation,"Foundation segmentation models, while powerful, pose a significant risk: they
+enable users to effortlessly extract any objects from any digital content with
+a single click, potentially leading to copyright infringement or malicious
+misuse. To mitigate this risk, we introduce a new task ""Anything Unsegmentable""
+to grant any image ""the right to be unsegmented"". The ambitious pursuit of the
+task is to achieve highly transferable adversarial attacks against all
+prompt-based segmentation models, regardless of model parameterizations and
+prompts. We highlight the non-transferable and heterogeneous nature of
+prompt-specific adversarial noises. Our approach focuses on disrupting image
+encoder features to achieve prompt-agnostic attacks. Intriguingly, targeted
+feature attacks exhibit better transferability compared to untargeted ones,
+suggesting the optimal update direction aligns with the image manifold. Based
+on the observations, we design a novel attack named Unsegment Anything by
+Simulating Deformation (UAD). Our attack optimizes a differentiable deformation
+function to create a target deformed image, which alters structural information
+while preserving achievable feature distance by adversarial example. Extensive
+experiments verify the effectiveness of our approach, compromising a variety of
+promptable segmentation models with different architectures and prompt
+interfaces. We release the code at
+https://github.com/jiahaolu97/anything-unsegmentable.",cs.CV,['cs.CV']
+"Low-power, Continuous Remote Behavioral Localization with Event Cameras",Friedhelm Hamann · Suman Ghosh · Ignacio Juarez Martinez · Tom Hart · Alex Kacelnik · Guillermo Gallego,https://tub-rip.github.io/eventpenguins/,https://arxiv.org/abs/2312.03799,,2312.03799.pdf,"Low-power, Continuous Remote Behavioral Localization with Event Cameras","Researchers in natural science need reliable methods for quantifying animal
+behavior. Recently, numerous computer vision methods emerged to automate the
+process. However, observing wild species at remote locations remains a
+challenging task due to difficult lighting conditions and constraints on power
+supply and data storage. Event cameras offer unique advantages for
+battery-dependent remote monitoring due to their low power consumption and high
+dynamic range capabilities. We use this novel sensor to quantify a behavior in
+Chinstrap penguins called ecstatic display. We formulate the problem as a
+temporal action detection task, determining the start and end times of the
+behavior. For this purpose, we recorded a colony of breeding penguins in
+Antarctica for several weeks and labeled event data on 16 nests. The developed
+method consists of a generator of candidate time intervals (proposals) and a
+classifier of the actions within them. The experiments show that the event
+cameras' natural response to motion is effective for continuous behavior
+monitoring and detection, reaching a mean average precision (mAP) of 58% (which
+increases to 63% in good weather conditions). The results also demonstrate the
+robustness against various lighting conditions contained in the challenging
+dataset. The low-power capabilities of the event camera allow it to record
+significantly longer than with a conventional camera. This work pioneers the
+use of event cameras for remote wildlife observation, opening new
+interdisciplinary opportunities. https://tub-rip.github.io/eventpenguins/",cs.CV,"['cs.CV', 'cs.AI']"
+Text-to-3D using Gaussian Splatting,Zilong Chen · Feng Wang · Yikai Wang · Huaping Liu,https://gsgen3d.github.io/,https://arxiv.org/abs/2309.16585,,2309.16585.pdf,Text-to-3D using Gaussian Splatting,"Automatic text-to-3D generation that combines Score Distillation Sampling
+(SDS) with the optimization of volume rendering has achieved remarkable
+progress in synthesizing realistic 3D objects. Yet most existing text-to-3D
+methods by SDS and volume rendering suffer from inaccurate geometry, e.g., the
+Janus issue, since it is hard to explicitly integrate 3D priors into implicit
+3D representations. Besides, it is usually time-consuming for them to generate
+elaborate 3D models with rich colors. In response, this paper proposes GSGEN, a
+novel method that adopts Gaussian Splatting, a recent state-of-the-art
+representation, to text-to-3D generation. GSGEN aims at generating high-quality
+3D objects and addressing existing shortcomings by exploiting the explicit
+nature of Gaussian Splatting that enables the incorporation of 3D prior.
+Specifically, our method adopts a progressive optimization strategy, which
+includes a geometry optimization stage and an appearance refinement stage. In
+geometry optimization, a coarse representation is established under 3D point
+cloud diffusion prior along with the ordinary 2D SDS optimization, ensuring a
+sensible and 3D-consistent rough shape. Subsequently, the obtained Gaussians
+undergo an iterative appearance refinement to enrich texture details. In this
+stage, we increase the number of Gaussians by compactness-based densification
+to enhance continuity and improve fidelity. With these designs, our approach
+can generate 3D assets with delicate details and accurate geometry. Extensive
+evaluations demonstrate the effectiveness of our method, especially for
+capturing high-frequency components. Our code is available at
+https://github.com/gsgen3d/gsgen",cs.CV,['cs.CV']
+UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory,Haiwen Diao · Bo Wan · Ying Zhang · Xu Jia · Huchuan Lu · Long Chen,https://github.com/Paranioar/UniPT,https://arxiv.org/abs/2308.14316v2,,2308.14316v2.pdf,UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory,"Parameter-efficient transfer learning (PETL), i.e., fine-tuning a small
+portion of parameters, is an effective strategy for adapting pre-trained models
+to downstream domains. To further reduce the memory demand, recent PETL works
+focus on the more valuable memory-efficient characteristic. In this paper, we
+argue that the scalability, adaptability, and generalizability of
+state-of-the-art methods are hindered by structural dependency and pertinency
+on specific pre-trained backbones. To this end, we propose a new
+memory-efficient PETL strategy, Universal Parallel Tuning (UniPT), to mitigate
+these weaknesses. Specifically, we facilitate the transfer process via a
+lightweight and learnable parallel network, which consists of: 1) A parallel
+interaction module that decouples the sequential connections and processes the
+intermediate activations detachedly from the pre-trained network. 2) A
+confidence aggregation module that learns optimal strategies adaptively for
+integrating cross-layer features. We evaluate UniPT with different backbones
+(e.g., T5, VSE$\infty$, CLIP4Clip, Clip-ViL, and MDETR) on various
+vision-and-language and pure NLP tasks. Extensive ablations on 18 datasets have
+validated that UniPT can not only dramatically reduce memory consumption and
+outperform the best competitor, but also achieve competitive performance over
+other plain PETL methods with lower training memory overhead. Our code is
+publicly available at: https://github.com/Paranioar/UniPT.",cs.CV,"['cs.CV', 'cs.MM']"
+Single-View Refractive Index Tomography with Neural Fields,Brandon Zhao · Aviad Levis · Liam Connor · Pratul P. Srinivasan · Katherine Bouman, ,https://arxiv.org/abs/2309.04437,,2309.04437.pdf,Single View Refractive Index Tomography with Neural Fields,"Refractive Index Tomography is the inverse problem of reconstructing the
+continuously-varying 3D refractive index in a scene using 2D projected image
+measurements. Although a purely refractive field is not directly visible, it
+bends light rays as they travel through space, thus providing a signal for
+reconstruction. The effects of such fields appear in many scientific computer
+vision settings, ranging from refraction due to transparent cells in microscopy
+to the lensing of distant galaxies caused by dark matter in astrophysics.
+Reconstructing these fields is particularly difficult due to the complex
+nonlinear effects of the refractive field on observed images. Furthermore,
+while standard 3D reconstruction and tomography settings typically have access
+to observations of the scene from many viewpoints, many refractive index
+tomography problem settings only have access to images observed from a single
+viewpoint. We introduce a method that leverages prior knowledge of light
+sources scattered throughout the refractive medium to help disambiguate the
+single-view refractive index tomography problem. We differentiably trace curved
+rays through a neural field representation of the refractive field, and
+optimize its parameters to best reproduce the observed image. We demonstrate
+the efficacy of our approach by reconstructing simulated refractive fields,
+analyze the effects of light source distribution on the recovered field, and
+test our method on a simulated dark matter mapping problem where we
+successfully recover the 3D refractive field caused by a realistic dark matter
+distribution.",cs.CV,"['cs.CV', 'astro-ph.CO']"
+MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos,Jielin Qiu · Jiacheng Zhu · William Han · Aditesh Kumar · Karthik Mittal · Claire Jin · Zhengyuan Yang · Linjie Li · Jianfeng Wang · DING ZHAO · Bo Li · Lijuan Wang, ,https://arxiv.org/abs/2306.04216,,2306.04216.pdf,MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos,"Multimodal summarization with multimodal output (MSMO) has emerged as a
+promising research direction. Nonetheless, numerous limitations exist within
+existing public MSMO datasets, including insufficient maintenance, data
+inaccessibility, limited size, and the absence of proper categorization, which
+pose significant challenges. To address these challenges and provide a
+comprehensive dataset for this new direction, we have meticulously curated the
+\textbf{MMSum} dataset. Our new dataset features (1) Human-validated summaries
+for both video and textual content, providing superior human instruction and
+labels for multimodal learning. (2) Comprehensively and meticulously arranged
+categorization, spanning 17 principal categories and 170 subcategories to
+encapsulate a diverse array of real-world scenarios. (3) Benchmark tests
+performed on the proposed dataset to assess various tasks and methods,
+including \textit{video summarization}, \textit{text summarization}, and
+\textit{multimodal summarization}. To champion accessibility and collaboration,
+we will release the \textbf{MMSum} dataset and the data collection tool as
+fully open-source resources, fostering transparency and accelerating future
+developments. Our project website can be found
+at~\url{https://mmsum-dataset.github.io/}",cs.CV,"['cs.CV', 'cs.MM']"
+Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching,Shitong Shao · Zeyuan Yin · Muxin Zhou · Xindong Zhang · Zhiqiang Shen, ,https://arxiv.org/abs/2311.17950,,2311.17950.pdf,Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching,"The lightweight ""local-match-global"" matching introduced by SRe2L
+successfully creates a distilled dataset with comprehensive information on the
+full 224x224 ImageNet-1k. However, this one-sided approach is limited to a
+particular backbone, layer, and statistics, which limits the improvement of the
+generalization of a distilled dataset. We suggest that sufficient and various
+""local-match-global"" matching are more precise and effective than a single one
+and has the ability to create a distilled dataset with richer information and
+better generalization. We call this perspective ""generalized matching"" and
+propose Generalized Various Backbone and Statistical Matching (G-VBSM) in this
+work, which aims to create a synthetic dataset with densities, ensuring
+consistency with the complete dataset across various backbones, layers, and
+statistics. As experimentally demonstrated, G-VBSM is the first algorithm to
+obtain strong performance across both small-scale and large-scale datasets.
+Specifically, G-VBSM achieves a performance of 38.7% on CIFAR-100 with
+128-width ConvNet, 47.6% on Tiny-ImageNet with ResNet18, and 31.4% on the full
+224x224 ImageNet-1k with ResNet18, under images per class (IPC) 10, 50, and 10,
+respectively. These results surpass all SOTA methods by margins of 3.9%, 6.5%,
+and 10.1%, respectively.",cs.CV,"['cs.CV', 'cs.AI']"
+Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation,Haofeng Liu · Chenshu Xu · Yifei Yang · Lihua Zeng · Shengfeng He,https://github.com/haofengl/DragNoise,https://arxiv.org/abs/2404.01050,,2404.01050.pdf,Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation,"Point-based interactive editing serves as an essential tool to complement the
+controllability of existing generative models. A concurrent work,
+DragDiffusion, updates the diffusion latent map in response to user inputs,
+causing global latent map alterations. This results in imprecise preservation
+of the original content and unsuccessful editing due to gradient vanishing. In
+contrast, we present DragNoise, offering robust and accelerated editing without
+retracing the latent map. The core rationale of DragNoise lies in utilizing the
+predicted noise output of each U-Net as a semantic editor. This approach is
+grounded in two critical observations: firstly, the bottleneck features of
+U-Net inherently possess semantically rich features ideal for interactive
+editing; secondly, high-level semantics, established early in the denoising
+process, show minimal variation in subsequent stages. Leveraging these
+insights, DragNoise edits diffusion semantics in a single denoising step and
+efficiently propagates these changes, ensuring stability and efficiency in
+diffusion editing. Comparative experiments reveal that DragNoise achieves
+superior control and semantic retention, reducing the optimization time by over
+50% compared to DragDiffusion. Our codes are available at
+https://github.com/haofengl/DragNoise.",cs.CV,"['cs.CV', 'cs.GR', 'cs.HC', 'cs.LG']"
+CMA: A Chromaticity Map Adapter for Robust Detection of Screen-Recapture Document Images,Changsheng Chen · Liangwei Lin · Yongqi Chen · Bin Li · Jishen Zeng · Jiwu Huang,https://github.com/chenlewis/Chromaticity-Map-Adapter-for-DPAD,https://arxiv.org/abs/2404.06663,,2404.06663.pdf,Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement,"Document Presentation Attack Detection (DPAD) is an important measure in
+protecting the authenticity of a document image. However, recent DPAD methods
+demand additional resources, such as manual effort in collecting additional
+data or knowing the parameters of acquisition devices. This work proposes a
+DPAD method based on multi-modal disentangled traces (MMDT) without the above
+drawbacks. We first disentangle the recaptured traces by a self-supervised
+disentanglement and synthesis network to enhance the generalization capacity in
+document images with different contents and layouts. Then, unlike the existing
+DPAD approaches that rely only on data in the RGB domain, we propose to
+explicitly employ the disentangled recaptured traces as new modalities in the
+transformer backbone through adaptive multi-modal adapters to fuse RGB/trace
+features efficiently. Visualization of the disentangled traces confirms the
+effectiveness of the proposed method in different document contents. Extensive
+experiments on three benchmark datasets demonstrate the superiority of our MMDT
+method on representing forensic traces of recapturing distortion.",cs.CV,['cs.CV']
+Navigating Beyond Dropout: An Intriguing Solution towards Generalizable Image Super-Resolution,Hongjun Wang · Jiyuan Chen · Yinqiang Zheng · Tieyong Zeng, ,https://arxiv.org/abs/2402.18929,,2402.18929.pdf,Navigating Beyond Dropout: An Intriguing Solution Towards Generalizable Image Super Resolution,"Deep learning has led to a dramatic leap on Single Image Super-Resolution
+(SISR) performances in recent years. %Despite the substantial advancement%
+While most existing work assumes a simple and fixed degradation model (e.g.,
+bicubic downsampling), the research of Blind SR seeks to improve model
+generalization ability with unknown degradation. Recently, Kong et al pioneer
+the investigation of a more suitable training strategy for Blind SR using
+Dropout. Although such method indeed brings substantial generalization
+improvements via mitigating overfitting, we argue that Dropout simultaneously
+introduces undesirable side-effect that compromises model's capacity to
+faithfully reconstruct fine details. We show both the theoretical and
+experimental analyses in our paper, and furthermore, we present another easy
+yet effective training strategy that enhances the generalization ability of the
+model by simply modulating its first and second-order features statistics.
+Experimental results have shown that our method could serve as a model-agnostic
+regularization and outperforms Dropout on seven benchmark datasets including
+both synthetic and real-world scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement,Shiwei Jin · Zhen Wang · Lei Wang · Peng Liu · Ning Bi · Truong Nguyen, ,https://arxiv.org/abs/2404.05063,,2404.05063.pdf,AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement,"Facial action unit (AU) intensity plays a pivotal role in quantifying
+fine-grained expression behaviors, which is an effective condition for facial
+expression manipulation. However, publicly available datasets containing
+intensity annotations for multiple AUs remain severely limited, often featuring
+a restricted number of subjects. This limitation places challenges to the AU
+intensity manipulation in images due to disentanglement issues, leading
+researchers to resort to other large datasets with pretrained AU intensity
+estimators for pseudo labels. In addressing this constraint and fully
+leveraging manual annotations of AU intensities for precise manipulation, we
+introduce AUEditNet. Our proposed model achieves impressive intensity
+manipulation across 12 AUs, trained effectively with only 18 subjects.
+Utilizing a dual-branch architecture, our approach achieves comprehensive
+disentanglement of facial attributes and identity without necessitating
+additional loss functions or implementing with large batch sizes. This approach
+offers a potential solution to achieve desired facial attribute editing despite
+the dataset's limited subject count. Our experiments demonstrate AUEditNet's
+superior accuracy in editing AU intensities, affirming its capability in
+disentangling facial attributes and identity within a limited subject pool.
+AUEditNet allows conditioning by either intensity values or target images,
+eliminating the need for constructing AU combinations for specific facial
+expression synthesis. Moreover, AU intensity estimation, as a downstream task,
+validates the consistency between real and edited images, confirming the
+effectiveness of our proposed AU intensity manipulation method.",cs.CV,['cs.CV']
+Degree-of-Freedom Matters: Inferring Dynamics from Point Trajectories,Yan Zhang · Sergey Prokudin · Marko Mihajlovic · Qianli Ma · Siyu Tang, ,,https://www.nature.com/articles/s44172-024-00179-3,,,,,nan
+Structure-Guided Adversarial Training of Diffusion Models,Ling Yang · Haotian Qian · Zhilong Zhang · Jingwei Liu · Bin CUI, ,https://arxiv.org/abs/2402.17563v1,,2402.17563v1.pdf,Structure-Guided Adversarial Training of Diffusion Models,"Diffusion models have demonstrated exceptional efficacy in various generative
+applications. While existing models focus on minimizing a weighted sum of
+denoising score matching losses for data distribution modeling, their training
+primarily emphasizes instance-level optimization, overlooking valuable
+structural information within each mini-batch, indicative of pair-wise
+relationships among samples. To address this limitation, we introduce
+Structure-guided Adversarial training of Diffusion Models (SADM). In this
+pioneering approach, we compel the model to learn manifold structures between
+samples in each training batch. To ensure the model captures authentic manifold
+structures in the data distribution, we advocate adversarial training of the
+diffusion generator against a novel structure discriminator in a minimax game,
+distinguishing real manifold structures from the generated ones. SADM
+substantially improves existing diffusion transformers (DiT) and outperforms
+existing methods in image generation and cross-domain fine-tuning tasks across
+12 datasets, establishing a new state-of-the-art FID of 1.58 and 2.11 on
+ImageNet for class-conditional image generation at resolutions of 256x256 and
+512x512, respectively.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners,Chun Feng · Joy Hsu · Weiyu Liu · Jiajun Wu,https://chunfeng3364.github.io/projects/larc_website/project_page.html,https://arxiv.org/abs/2404.19696,,,Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners,"3D visual grounding is a challenging task that often requires direct and
+dense supervision, notably the semantic label for each object in the scene. In
+this paper, we instead study the naturally supervised setting that learns from
+only 3D scene and QA pairs, where prior works underperform. We propose the
+Language-Regularized Concept Learner (LARC), which uses constraints from
+language as regularization to significantly improve the accuracy of
+neuro-symbolic concept learners in the naturally supervised setting. Our
+approach is based on two core insights: the first is that language constraints
+(e.g., a word's relation to another) can serve as effective regularization for
+structured representations in neuro-symbolic models; the second is that we can
+query large language models to distill such constraints from language
+properties. We show that LARC improves performance of prior works in naturally
+supervised 3D visual grounding, and demonstrates a wide range of 3D visual
+reasoning capabilities-from zero-shot composition, to data efficiency and
+transferability. Our method represents a promising step towards regularizing
+structured visual reasoning frameworks with language-based priors, for learning
+in settings without dense supervision.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+SuperPrimitive: Scene Reconstruction at a Primitive Level,Kirill Mazur · Gwangbin Bae · Andrew J. Davison, ,https://arxiv.org/abs/2312.05889,,2312.05889.pdf,SuperPrimitive: Scene Reconstruction at a Primitive Level,"Joint camera pose and dense geometry estimation from a set of images or a
+monocular video remains a challenging problem due to its computational
+complexity and inherent visual ambiguities. Most dense incremental
+reconstruction systems operate directly on image pixels and solve for their 3D
+positions using multi-view geometry cues. Such pixel-level approaches suffer
+from ambiguities or violations of multi-view consistency (e.g. caused by
+textureless or specular surfaces).
+  We address this issue with a new image representation which we call a
+SuperPrimitive. SuperPrimitives are obtained by splitting images into
+semantically correlated local regions and enhancing them with estimated surface
+normal directions, both of which are predicted by state-of-the-art single image
+neural networks. This provides a local geometry estimate per SuperPrimitive,
+while their relative positions are adjusted based on multi-view observations.
+  We demonstrate the versatility of our new representation by addressing three
+3D reconstruction tasks: depth completion, few-view structure from motion, and
+monocular dense visual odometry.",cs.CV,['cs.CV']
+Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification,Chao Yi · Lu Ren · De-Chuan Zhan · Han-Jia Ye, ,https://arxiv.org/abs/2404.17753,,2404.17753.pdf,Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification,"CLIP showcases exceptional cross-modal matching capabilities due to its
+training on image-text contrastive learning tasks. However, without specific
+optimization for unimodal scenarios, its performance in single-modality feature
+extraction might be suboptimal. Despite this, some studies have directly used
+CLIP's image encoder for tasks like few-shot classification, introducing a
+misalignment between its pre-training objectives and feature extraction
+methods. This inconsistency can diminish the quality of the image's feature
+representation, adversely affecting CLIP's effectiveness in target tasks. In
+this paper, we view text features as precise neighbors of image features in
+CLIP's space and present a novel CrOss-moDal nEighbor Representation(CODER)
+based on the distance structure between images and their neighbor texts. This
+feature extraction method aligns better with CLIP's pre-training objectives,
+thereby fully leveraging CLIP's robust cross-modal capabilities. The key to
+construct a high-quality CODER lies in how to create a vast amount of
+high-quality and diverse texts to match with images. We introduce the Auto Text
+Generator(ATG) to automatically generate the required texts in a data-free and
+training-free manner. We apply CODER to CLIP's zero-shot and few-shot image
+classification tasks. Experiment results across various datasets and models
+confirm CODER's effectiveness. Code is available
+at:https://github.com/YCaigogogo/CVPR24-CODER.",cs.CV,"['cs.CV', 'cs.AI']"
+MicroDiffusion: Implicit Representation-Guided Diffusion for 3D Reconstruction from Limited 2D Microscopy Projections,mude hui · Zihao Wei · Hongru Zhu · Fei Xia · Yuyin Zhou,https://github.com/UCSC-VLAA/MicroDiffusion,https://arxiv.org/abs/2403.10815,,2403.10815.pdf,MicroDiffusion: Implicit Representation-Guided Diffusion for 3D Reconstruction from Limited 2D Microscopy Projections,"Volumetric optical microscopy using non-diffracting beams enables rapid
+imaging of 3D volumes by projecting them axially to 2D images but lacks crucial
+depth information. Addressing this, we introduce MicroDiffusion, a pioneering
+tool facilitating high-quality, depth-resolved 3D volume reconstruction from
+limited 2D projections. While existing Implicit Neural Representation (INR)
+models often yield incomplete outputs and Denoising Diffusion Probabilistic
+Models (DDPM) excel at capturing details, our method integrates INR's
+structural coherence with DDPM's fine-detail enhancement capabilities. We
+pretrain an INR model to transform 2D axially-projected images into a
+preliminary 3D volume. This pretrained INR acts as a global prior guiding
+DDPM's generative process through a linear interpolation between INR outputs
+and noise inputs. This strategy enriches the diffusion process with structured
+3D information, enhancing detail and reducing noise in localized 2D images. By
+conditioning the diffusion model on the closest 2D projection, MicroDiffusion
+substantially enhances fidelity in resulting 3D reconstructions, surpassing INR
+and standard DDPM outputs with unparalleled image quality and structural
+fidelity. Our code and dataset are available at
+https://github.com/UCSC-VLAA/MicroDiffusion.",eess.IV,"['eess.IV', 'cs.CV']"
+Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data,Yu Deng · Duomin Wang · Xiaohang Ren · Xingyu Chen · Baoyuan Wang,https://github.com/YuDeng/Portrait-4D,https://arxiv.org/abs/2311.18729,,2311.18729.pdf,Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data,"Existing one-shot 4D head synthesis methods usually learn from monocular
+videos with the aid of 3DMM reconstruction, yet the latter is evenly
+challenging which restricts them from reasonable 4D head synthesis. We present
+a method to learn one-shot 4D head synthesis via large-scale synthetic data.
+The key is to first learn a part-wise 4D generative model from monocular images
+via adversarial learning, to synthesize multi-view images of diverse identities
+and full motions as training data; then leverage a transformer-based animatable
+triplane reconstructor to learn 4D head reconstruction using the synthetic
+data. A novel learning strategy is enforced to enhance the generalizability to
+real images by disentangling the learning process of 3D reconstruction and
+reenactment. Experiments demonstrate our superiority over the prior art.",cs.CV,['cs.CV']
+Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes,Hmrishav Bandyopadhyay · Subhadeep Koley · Ayan Das · Ayan Kumar Bhunia · Aneeshan Sain · Pinaki Nath Chowdhury · Tao Xiang · Yi-Zhe Song,https://hmrishavbandy.github.io/doodle23d/,https://arxiv.org/abs/2312.04043,,2312.04043.pdf,Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes,"In this paper, we democratise 3D content creation, enabling precise
+generation of 3D shapes from abstract sketches while overcoming limitations
+tied to drawing skills. We introduce a novel part-level modelling and alignment
+framework that facilitates abstraction modelling and cross-modal
+correspondence. Leveraging the same part-level decoder, our approach seamlessly
+extends to sketch modelling by establishing correspondence between CLIPasso
+edgemaps and projected 3D part regions, eliminating the need for a dataset
+pairing human sketches and 3D shapes. Additionally, our method introduces a
+seamless in-position editing process as a byproduct of cross-modal part-aligned
+modelling. Operating in a low-dimensional implicit space, our approach
+significantly reduces computational demands and processing time.",cs.CV,"['cs.CV', 'cs.AI']"
+Blur-aware Spatio-temporal Sparse Transformer for Video Deblurring,Huicong Zhang · Haozhe Xie · Hongxun Yao,https://vilab.hit.edu.cn/projects/bsstnet,,https://github.com/huicongzhang/BSSTNet,,,,,nan
+Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications,Junyi Ma · Xieyuanli Chen · Jiawei Huang · Jingyi Xu · Zhen Luo · Jintao Xu · Weihao Gu · Rui Ai · Hesheng Wang,https://github.com/haomo-ai/Cam4DOcc,https://arxiv.org/abs/2311.17663,,2311.17663.pdf,Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications,"Understanding how the surrounding environment changes is crucial for
+performing downstream tasks safely and reliably in autonomous driving
+applications. Recent occupancy estimation techniques using only camera images
+as input can provide dense occupancy representations of large-scale scenes
+based on the current observation. However, they are mostly limited to
+representing the current 3D space and do not consider the future state of
+surrounding objects along the time axis. To extend camera-only occupancy
+estimation into spatiotemporal prediction, we propose Cam4DOcc, a new benchmark
+for camera-only 4D occupancy forecasting, evaluating the surrounding scene
+changes in a near future. We build our benchmark based on multiple publicly
+available datasets, including nuScenes, nuScenes-Occupancy, and Lyft-Level5,
+which provides sequential occupancy states of general movable and static
+objects, as well as their 3D backward centripetal flow. To establish this
+benchmark for future research with comprehensive comparisons, we introduce four
+baseline types from diverse camera-based perception and prediction
+implementations, including a static-world occupancy model, voxelization of
+point cloud prediction, 2D-3D instance-based prediction, and our proposed novel
+end-to-end 4D occupancy forecasting network. Furthermore, the standardized
+evaluation protocol for preset multiple tasks is also provided to compare the
+performance of all the proposed baselines on present and future occupancy
+estimation with respect to objects of interest in autonomous driving scenarios.
+The dataset and our implementation of all four baselines in the proposed
+Cam4DOcc benchmark will be released here: https://github.com/haomo-ai/Cam4DOcc.",cs.CV,['cs.CV']
+DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations,Tianhao Qi · Shancheng Fang · Yanze Wu · Hongtao Xie · Jiawei Liu · Lang chen · Qian HE · Yongdong Zhang,https://tianhao-qi.github.io/DEADiff/,https://arxiv.org/abs/2403.06951,,2403.06951.pdf,DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations,"The diffusion-based text-to-image model harbors immense potential in
+transferring reference style. However, current encoder-based approaches
+significantly impair the text controllability of text-to-image models while
+transferring styles. In this paper, we introduce DEADiff to address this issue
+using the following two strategies: 1) a mechanism to decouple the style and
+semantics of reference images. The decoupled feature representations are first
+extracted by Q-Formers which are instructed by different text descriptions.
+Then they are injected into mutually exclusive subsets of cross-attention
+layers for better disentanglement. 2) A non-reconstructive learning method. The
+Q-Formers are trained using paired images rather than the identical target, in
+which the reference image and the ground-truth image are with the same style or
+semantics. We show that DEADiff attains the best visual stylization results and
+optimal balance between the text controllability inherent in the text-to-image
+model and style similarity to the reference image, as demonstrated both
+quantitatively and qualitatively. Our project page is
+https://tianhao-qi.github.io/DEADiff/.",cs.CV,['cs.CV']
+What Sketch Explainability Really Means for Downstream Tasks ?,Hmrishav Bandyopadhyay · Pinaki Nath Chowdhury · Ayan Kumar Bhunia · Aneeshan Sain · Tao Xiang · Yi-Zhe Song, ,https://arxiv.org/abs/2403.09480,,2403.09480.pdf,What Sketch Explainability Really Means for Downstream Tasks,"In this paper, we explore the unique modality of sketch for explainability,
+emphasising the profound impact of human strokes compared to conventional
+pixel-oriented studies. Beyond explanations of network behavior, we discern the
+genuine implications of explainability across diverse downstream sketch-related
+tasks. We propose a lightweight and portable explainability solution -- a
+seamless plugin that integrates effortlessly with any pre-trained model,
+eliminating the need for re-training. Demonstrating its adaptability, we
+present four applications: highly studied retrieval and generation, and
+completely novel assisted drawing and sketch adversarial attacks. The
+centrepiece to our solution is a stroke-level attribution map that takes
+different forms when linked with downstream tasks. By addressing the inherent
+non-differentiability of rasterisation, we enable explanations at both coarse
+stroke level (SLA) and partial stroke level (P-SLA), each with its advantages
+for specific downstream tasks.",cs.CV,"['cs.CV', 'cs.AI']"
+OHTA: One-shot Hand Avatar via Data-driven Implicit Priors,Xiaozheng Zheng · Chao Wen · Zhuo Su · Zeran Xu · Zhaohu Li · Yang Zhao · Zhou Xue,https://zxz267.github.io/OHTA/,https://arxiv.org/abs/2402.18969,,2402.18969.pdf,OHTA: One-shot Hand Avatar via Data-driven Implicit Priors,"In this paper, we delve into the creation of one-shot hand avatars, attaining
+high-fidelity and drivable hand representations swiftly from a single image.
+With the burgeoning domains of the digital human, the need for quick and
+personalized hand avatar creation has become increasingly critical. Existing
+techniques typically require extensive input data and may prove cumbersome or
+even impractical in certain scenarios. To enhance accessibility, we present a
+novel method OHTA (One-shot Hand avaTAr) that enables the creation of detailed
+hand avatars from merely one image. OHTA tackles the inherent difficulties of
+this data-limited problem by learning and utilizing data-driven hand priors.
+Specifically, we design a hand prior model initially employed for 1) learning
+various hand priors with available data and subsequently for 2) the inversion
+and fitting of the target identity with prior knowledge. OHTA demonstrates the
+capability to create high-fidelity hand avatars with consistent animatable
+quality, solely relying on a single image. Furthermore, we illustrate the
+versatility of OHTA through diverse applications, encompassing text-to-avatar
+conversion, hand editing, and identity latent space manipulation.",cs.CV,['cs.CV']
+FedUV: Uniformity and Variance for Heterogeneous Federated Learning,Ha Min Son · Moon-Hyun Kim · Tai-Myoung Chung · Chao Huang · Xin Liu,https://github.com/sonhamin/FedUV,https://arxiv.org/abs/2402.18372,,2402.18372.pdf,FedUV: Uniformity and Variance for Heterogeneous Federated Learning,"Federated learning is a promising framework to train neural networks with
+widely distributed data. However, performance degrades heavily with
+heterogeneously distributed data. Recent work has shown this is due to the
+final layer of the network being most prone to local bias, some finding success
+freezing the final layer as an orthogonal classifier. We investigate the
+training dynamics of the classifier by applying SVD to the weights motivated by
+the observation that freezing weights results in constant singular values. We
+find that there are differences when training in IID and non-IID settings.
+Based on this finding, we introduce two regularization terms for local training
+to continuously emulate IID settings: (1) variance in the dimension-wise
+probability distribution of the classifier and (2) hyperspherical uniformity of
+representations of the encoder. These regularizations promote local models to
+act as if it were in an IID setting regardless of the local data distribution,
+thus offsetting proneness to bias while being flexible to the data. On
+extensive experiments in both label-shift and feature-shift settings, we verify
+that our method achieves highest performance by a large margin especially in
+highly non-IID cases in addition to being scalable to larger models and
+datasets.",cs.LG,"['cs.LG', 'cs.AI', 'cs.DC']"
+WinSyn: A High Resolution Testbed for Synthetic Data,Tom Kelly · John Femiani · Peter Wonka, ,https://arxiv.org/abs/2310.08471,,2310.08471.pdf,WinSyn: A High Resolution Testbed for Synthetic Data,"We present WinSyn, a unique dataset and testbed for creating high-quality
+synthetic data with procedural modeling techniques. The dataset contains
+high-resolution photographs of windows, selected from locations around the
+world, with 89,318 individual window crops showcasing diverse geometric and
+material characteristics. We evaluate a procedural model by training semantic
+segmentation networks on both synthetic and real images and then comparing
+their performances on a shared test set of real images. Specifically, we
+measure the difference in mean Intersection over Union (mIoU) and determine the
+effective number of real images to match synthetic data's training performance.
+We design a baseline procedural model as a benchmark and provide 21,290
+synthetically generated images. By tuning the procedural model, key factors are
+identified which significantly influence the model's fidelity in replicating
+real-world scenarios. Importantly, we highlight the challenge of procedural
+modeling using current techniques, especially in their ability to replicate the
+spatial semantics of real-world scenarios. This insight is critical because of
+the potential of procedural models to bridge to hidden scene aspects such as
+depth, reflectivity, material properties, and lighting conditions.",cs.CV,"['cs.CV', 'cs.GR']"
+Rethinking Inductive Biases for Surface Normal Estimation,Gwangbin Bae · Andrew J. Davison, ,https://arxiv.org/abs/2403.00712,,2403.00712.pdf,Rethinking Inductive Biases for Surface Normal Estimation,"Despite the growing demand for accurate surface normal estimation models,
+existing methods use general-purpose dense prediction models, adopting the same
+inductive biases as other tasks. In this paper, we discuss the inductive biases
+needed for surface normal estimation and propose to (1) utilize the per-pixel
+ray direction and (2) encode the relationship between neighboring surface
+normals by learning their relative rotation. The proposed method can generate
+crisp - yet, piecewise smooth - predictions for challenging in-the-wild images
+of arbitrary resolution and aspect ratio. Compared to a recent ViT-based
+state-of-the-art model, our method shows a stronger generalization ability,
+despite being trained on an orders of magnitude smaller dataset. The code is
+available at https://github.com/baegwangbin/DSINE.",cs.CV,['cs.CV']
+MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding,Xu Cao · Tong Zhou · Yunsheng Ma · Wenqian Ye · Can Cui · Kun Tang · Zhipeng Cao · Kaizhao Liang · Ziran Wang · James Rehg · chao zheng, ,,https://ysma.me/,,,,,nan
+In2SET: Intra-Inter Similarity Exploiting Transformer for  Dual-Camera Compressive Hyperspectral Imaging,Xin Wang · Lizhi Wang · Xiangtian Ma · Maoqing Zhang · Lin Zhu · Hua Huang,https://github.com/2JONAS/In2SET,https://arxiv.org/abs/2312.13319,,2312.13319.pdf,In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging,"Dual-Camera Compressed Hyperspectral Imaging (DCCHI) offers the capability to
+reconstruct 3D Hyperspectral Image (HSI) by fusing compressive and Panchromatic
+(PAN) image, which has shown great potential for snapshot hyperspectral imaging
+in practice. In this paper, we introduce a novel DCCHI reconstruction network,
+the Intra-Inter Similarity Exploiting Transformer (In2SET). Our key insight is
+to make full use of the PAN image to assist the reconstruction. To this end, we
+propose using the intra-similarity within the PAN image as a proxy for
+approximating the intra-similarity in the original HSI, thereby offering an
+enhanced content prior for more accurate HSI reconstruction. Furthermore, we
+aim to align the features from the underlying HSI with those of the PAN image,
+maintaining semantic consistency and introducing new contextual information for
+the reconstruction process. By integrating In2SET into a PAN-guided unrolling
+framework, our method substantially enhances the spatial-spectral fidelity and
+detail of the reconstructed images, providing a more comprehensive and accurate
+depiction of the scene. Extensive experiments conducted on both real and
+simulated datasets demonstrate that our approach consistently outperforms
+existing state-of-the-art methods in terms of reconstruction quality and
+computational complexity. Code will be released.",eess.IV,"['eess.IV', 'cs.CV']"
+Describing Differences in Image Sets with Natural Language,Lisa Dunlap · Yuhui Zhang · Xiaohan Wang · Ruiqi Zhong · Trevor Darrell · Jacob Steinhardt · Joseph Gonzalez · Serena Yeung,https://understanding-visual-datasets.github.io/VisDiff-website/,https://arxiv.org/abs/2312.02974,,2312.02974.pdf,Describing Differences in Image Sets with Natural Language,"How do two sets of images differ? Discerning set-level differences is crucial
+for understanding model behaviors and analyzing datasets, yet manually sifting
+through thousands of images is impractical. To aid in this discovery process,
+we explore the task of automatically describing the differences between two
+$\textbf{sets}$ of images, which we term Set Difference Captioning. This task
+takes in image sets $D_A$ and $D_B$, and outputs a description that is more
+often true on $D_A$ than $D_B$. We outline a two-stage approach that first
+proposes candidate difference descriptions from image sets and then re-ranks
+the candidates by checking how well they can differentiate the two sets. We
+introduce VisDiff, which first captions the images and prompts a language model
+to propose candidate descriptions, then re-ranks these descriptions using CLIP.
+To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
+sets with ground truth difference descriptions. We apply VisDiff to various
+domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
+classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
+model failure modes (supervised ResNet), characterizing differences between
+generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
+images memorable. Using VisDiff, we are able to find interesting and previously
+unknown differences in datasets and models, demonstrating its utility in
+revealing nuanced insights.",cs.CV,"['cs.CV', 'cs.CL', 'cs.CY', 'cs.LG']"
+SketchINR: A First Look into Sketches as Implicit Neural Representations,Hmrishav Bandyopadhyay · Ayan Kumar Bhunia · Pinaki Nath Chowdhury · Aneeshan Sain · Tao Xiang · Timothy Hospedales · Yi-Zhe Song,https://hmrishavbandy.github.io/sketchinr,https://arxiv.org/abs/2403.09344,,2403.09344.pdf,SketchINR: A First Look into Sketches as Implicit Neural Representations,"We propose SketchINR, to advance the representation of vector sketches with
+implicit neural models. A variable length vector sketch is compressed into a
+latent space of fixed dimension that implicitly encodes the underlying shape as
+a function of time and strokes. The learned function predicts the $xy$ point
+coordinates in a sketch at each time and stroke. Despite its simplicity,
+SketchINR outperforms existing representations at multiple tasks: (i) Encoding
+an entire sketch dataset into a fixed size latent vector, SketchINR gives
+$60\times$ and $10\times$ data compression over raster and vector sketches,
+respectively. (ii) SketchINR's auto-decoder provides a much higher-fidelity
+representation than other learned vector sketch representations, and is
+uniquely able to scale to complex vector sketches such as FS-COCO. (iii)
+SketchINR supports parallelisation that can decode/render $\sim$$100\times$
+faster than other learned vector representations such as SketchRNN. (iv)
+SketchINR, for the first time, emulates the human ability to reproduce a sketch
+with varying abstraction in terms of number and complexity of strokes. As a
+first look at implicit sketches, SketchINR's compact high-fidelity
+representation will support future work in modelling long and complex sketches.",cs.CV,"['cs.CV', 'cs.AI']"
+Commonsense Prototype for Outdoor Unsupervised 3D Object Detection,Hai Wu · Shijia Zhao · Xun Huang · Chenglu Wen · Xin Li · Cheng Wang,https://github.com/hailanyi/CPD,https://arxiv.org/abs/2404.16493,,2404.16493.pdf,Commonsense Prototype for Outdoor Unsupervised 3D Object Detection,"The prevalent approaches of unsupervised 3D object detection follow
+cluster-based pseudo-label generation and iterative self-training processes.
+However, the challenge arises due to the sparsity of LiDAR scans, which leads
+to pseudo-labels with erroneous size and position, resulting in subpar
+detection performance. To tackle this problem, this paper introduces a
+Commonsense Prototype-based Detector, termed CPD, for unsupervised 3D object
+detection. CPD first constructs Commonsense Prototype (CProto) characterized by
+high-quality bounding box and dense points, based on commonsense intuition.
+Subsequently, CPD refines the low-quality pseudo-labels by leveraging the size
+prior from CProto. Furthermore, CPD enhances the detection accuracy of sparsely
+scanned objects by the geometric knowledge from CProto. CPD outperforms
+state-of-the-art unsupervised 3D detectors on Waymo Open Dataset (WOD),
+PandaSet, and KITTI datasets by a large margin. Besides, by training CPD on WOD
+and testing on KITTI, CPD attains 90.85% and 81.01% 3D Average Precision on
+easy and moderate car classes, respectively. These achievements position CPD in
+close proximity to fully supervised detectors, highlighting the significance of
+our method. The code will be available at https://github.com/hailanyi/CPD.",cs.CV,['cs.CV']
+Global and Hierarchical Geometry Consistency Priors for Few-shot NeRFs in Indoor Scenes,Xiaotian Sun · Qingshan Xu · Xinjie Yang · Yu Zang · Cheng Wang, ,https://arxiv.org/html/2404.00992v1,,2404.00992v1.pdf,SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance,"Neural Radiance Field (NeRF) technology has made significant strides in
+creating novel viewpoints. However, its effectiveness is hampered when working
+with sparsely available views, often leading to performance dips due to
+overfitting. FreeNeRF attempts to overcome this limitation by integrating
+implicit geometry regularization, which incrementally improves both geometry
+and textures. Nonetheless, an initial low positional encoding bandwidth results
+in the exclusion of high-frequency elements. The quest for a holistic approach
+that simultaneously addresses overfitting and the preservation of
+high-frequency details remains ongoing. This study introduces a novel feature
+matching based sparse geometry regularization module. This module excels in
+pinpointing high-frequency keypoints, thereby safeguarding the integrity of
+fine details. Through progressive refinement of geometry and textures across
+NeRF iterations, we unveil an effective few-shot neural rendering architecture,
+designated as SGCNeRF, for enhanced novel view synthesis. Our experiments
+demonstrate that SGCNeRF not only achieves superior geometry-consistent
+outcomes but also surpasses FreeNeRF, with improvements of 0.7 dB and 0.6 dB in
+PSNR on the LLFF and DTU datasets, respectively.",cs.CV,['cs.CV']
+Segment Every Out-of-Distribution Object,Wenjie Zhao · Jia Li · Xin Dong · Yu Xiang · Yunhui Guo, ,https://arxiv.org/abs/2311.16516,,2311.16516.pdf,Segment Every Out-of-Distribution Object,"Semantic segmentation models, while effective for in-distribution categories,
+face challenges in real-world deployment due to encountering
+out-of-distribution (OoD) objects. Detecting these OoD objects is crucial for
+safety-critical applications. Existing methods rely on anomaly scores, but
+choosing a suitable threshold for generating masks presents difficulties and
+can lead to fragmentation and inaccuracy. This paper introduces a method to
+convert anomaly \textbf{S}core \textbf{T}o segmentation \textbf{M}ask, called
+S2M, a simple and effective framework for OoD detection in semantic
+segmentation. Unlike assigning anomaly scores to pixels, S2M directly segments
+the entire OoD object. By transforming anomaly scores into prompts for a
+promptable segmentation model, S2M eliminates the need for threshold selection.
+Extensive experiments demonstrate that S2M outperforms the state-of-the-art by
+approximately 20% in IoU and 40% in mean F1 score, on average, across various
+benchmarks including Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly
+datasets.",cs.CV,['cs.CV']
+Learning to Segment Referred Objects from Narrated Egocentric Videos,Yuhan Shen · Huiyu Wang · Xitong Yang · Matt Feiszli · Ehsan Elhamifar · Lorenzo Torresani · Effrosyni Mavroudi, ,https://arxiv.org/abs/2404.05206,,2404.05206.pdf,SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos,"We propose a novel self-supervised embedding to learn how actions sound from
+narrated in-the-wild egocentric videos. Whereas existing methods rely on
+curated data with known audio-visual correspondence, our multimodal
+contrastive-consensus coding (MC3) embedding reinforces the associations
+between audio, language, and vision when all modality pairs agree, while
+diminishing those associations when any one pair does not. We show our approach
+can successfully discover how the long tail of human actions sound from
+egocentric video, outperforming an array of recent multimodal embedding
+techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal
+tasks.",cs.CV,"['cs.CV', 'cs.MM', 'cs.SD', 'eess.AS']"
+Low-Resource Vision Challenges for Foundation Models,Yunhua Zhang · Hazel Doughty · Cees G. M. Snoek, ,https://arxiv.org/abs/2401.04716,,2401.04716.pdf,Low-Resource Vision Challenges for Foundation Models,"Low-resource settings are well-established in natural language processing,
+where many languages lack sufficient data for deep learning at scale. However,
+low-resource problems are under-explored in computer vision. In this paper, we
+address this gap and explore the challenges of low-resource image tasks with
+vision foundation models. We first collect a benchmark of genuinely
+low-resource image data, covering historic maps, circuit diagrams, and
+mechanical drawings. These low-resource settings all share three challenges:
+data scarcity, fine-grained differences, and the distribution shift from
+natural images to the specialized domain of interest. While existing foundation
+models have shown impressive generalizability, we find they cannot transfer
+well to our low-resource tasks. To begin to tackle the challenges of
+low-resource vision, we introduce one simple baseline per challenge.
+Specifically, we i) enlarge the data space by generative models, ii) adopt the
+best sub-kernels to encode local regions for fine-grained difference discovery
+and iii) learn attention for specialized domains. Experiments on our three
+low-resource tasks demonstrate our proposals already provide a better baseline
+than transfer learning, data augmentation, and fine-grained methods. This
+highlights the unique characteristics and challenges of low-resource vision for
+foundation models that warrant further investigation. Project page:
+https://xiaobai1217.github.io/Low-Resource-Vision/.",cs.CV,['cs.CV']
+SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors,Dave Zhenyu Chen · Haoxuan Li · Hsin-Ying Lee · Sergey Tulyakov · Matthias Nießner,https://daveredrum.github.io/SceneTex/,https://arxiv.org/abs/2311.17261,,2311.17261.pdf,SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors,"We propose SceneTex, a novel method for effectively generating high-quality
+and style-consistent textures for indoor scenes using depth-to-image diffusion
+priors. Unlike previous methods that either iteratively warp 2D views onto a
+mesh surface or distillate diffusion latent features without accurate geometric
+and style cues, SceneTex formulates the texture synthesis task as an
+optimization problem in the RGB space where style and geometry consistency are
+properly reflected. At its core, SceneTex proposes a multiresolution texture
+field to implicitly encode the mesh appearance. We optimize the target texture
+via a score-distillation-based objective function in respective RGB renderings.
+To further secure the style consistency across views, we introduce a
+cross-attention decoder to predict the RGB values by cross-attending to the
+pre-sampled reference locations in each instance. SceneTex enables various and
+accurate texture synthesis for 3D-FRONT scenes, demonstrating significant
+improvements in visual quality and prompt fidelity over the prior texture
+generation methods.",cs.CV,['cs.CV']
+TCP: Textual-based Class-aware Prompt tuning for Visual-Language Model,Hantao Yao · Rui Zhang · Changsheng Xu, ,https://arxiv.org/abs/2311.18231,,2311.18231.pdf,TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model,"Prompt tuning represents a valuable technique for adapting pre-trained
+visual-language models (VLM) to various downstream tasks. Recent advancements
+in CoOp-based methods propose a set of learnable domain-shared or
+image-conditional textual tokens to facilitate the generation of task-specific
+textual classifiers. However, those textual tokens have a limited
+generalization ability regarding unseen domains, as they cannot dynamically
+adjust to the distribution of testing classes. To tackle this issue, we present
+a novel Textual-based Class-aware Prompt tuning(TCP) that explicitly
+incorporates prior knowledge about classes to enhance their discriminability.
+The critical concept of TCP involves leveraging Textual Knowledge Embedding
+(TKE) to map the high generalizability of class-level textual knowledge into
+class-aware textual tokens. By seamlessly integrating these class-aware prompts
+into the Text Encoder, a dynamic class-aware classifier is generated to enhance
+discriminability for unseen domains. During inference, TKE dynamically
+generates class-aware prompts related to the unseen classes. Comprehensive
+evaluations demonstrate that TKE serves as a plug-and-play module effortlessly
+combinable with existing methods. Furthermore, TCP consistently achieves
+superior performance while demanding less training time.
+Code:https://github.com/htyao89/Textual-based_Class-aware_prompt_tuning/",cs.CV,['cs.CV']
+URHand: Universal Relightable Hands,Zhaoxi Chen · Gyeongsik Moon · Kaiwen Guo · Chen Cao · Stanislav Pidhorskyi · Tomas Simon · Rohan Joshi · Yuan Dong · Yichen Xu · Bernardo Pires · He Wen · Lucas Evans · Bo Peng · Julia Buffalini · Autumn Trimble · Kevyn McPhail · Melissa Schoeller · Shoou-I Yu · Javier Romero · Michael Zollhoefer · Yaser Sheikh · Ziwei Liu · Shunsuke Saito,https://frozenburning.github.io/projects/urhand/,http://export.arxiv.org/abs/2401.05334,,2401.05334.pdf,URHand: Universal Relightable Hands,"Existing photorealistic relightable hand models require extensive
+identity-specific observations in different views, poses, and illuminations,
+and face challenges in generalizing to natural illuminations and novel
+identities. To bridge this gap, we present URHand, the first universal
+relightable hand model that generalizes across viewpoints, poses,
+illuminations, and identities. Our model allows few-shot personalization using
+images captured with a mobile phone, and is ready to be photorealistically
+rendered under novel illuminations. To simplify the personalization process
+while retaining photorealism, we build a powerful universal relightable prior
+based on neural relighting from multi-view images of hands captured in a light
+stage with hundreds of identities. The key challenge is scaling the
+cross-identity training while maintaining personalized fidelity and sharp
+details without compromising generalization under natural illuminations. To
+this end, we propose a spatially varying linear lighting model as the neural
+renderer that takes physics-inspired shading as input feature. By removing
+non-linear activations and bias, our specifically designed lighting model
+explicitly keeps the linearity of light transport. This enables single-stage
+training from light-stage data while generalizing to real-time rendering under
+arbitrary continuous illuminations across diverse identities. In addition, we
+introduce the joint learning of a physically based model and our neural
+relighting model, which further improves fidelity and generalization. Extensive
+experiments show that our approach achieves superior performance over existing
+methods in terms of both quality and generalizability. We also demonstrate
+quick personalization of URHand from a short phone scan of an unseen identity.",cs.CV,"['cs.CV', 'cs.GR']"
+EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams,Christen Millerdurai · Hiroyasu Akada · Jian Wang · Diogo Luvizon · Christian Theobalt · Vladislav Golyanik,https://4dqv.mpi-inf.mpg.de/EventEgo3D/,https://arxiv.org/abs/2404.08640,,2404.08640.pdf,EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams,"Monocular egocentric 3D human motion capture is a challenging and actively
+researched problem. Existing methods use synchronously operating visual sensors
+(e.g. RGB cameras) and often fail under low lighting and fast motions, which
+can be restricting in many applications involving head-mounted devices. In
+response to the existing limitations, this paper 1) introduces a new problem,
+i.e., 3D human motion capture from an egocentric monocular event camera with a
+fisheye lens, and 2) proposes the first approach to it called EventEgo3D
+(EE3D). Event streams have high temporal resolution and provide reliable cues
+for 3D human motion capture under high-speed human motions and rapidly changing
+illumination. The proposed EE3D framework is specifically tailored for learning
+with event streams in the LNES representation, enabling high 3D reconstruction
+accuracy. We also design a prototype of a mobile head-mounted device with an
+event camera and record a real dataset with event observations and the
+ground-truth 3D human poses (in addition to the synthetic dataset). Our EE3D
+demonstrates robustness and superior 3D accuracy compared to existing solutions
+across various challenging experiments while supporting real-time 3D pose
+update rates of 140Hz.",cs.CV,['cs.CV']
+WateRF: Robust Watermarks in Radiance Fields for Protection of Copyrights,Youngdong Jang · Dong In Lee · MinHyuk Jang · Jong Wook Kim · Feng Yang · Sangpil Kim,https://kuai-lab.github.io/cvpr2024waterf/,https://arxiv.org/abs/2405.02066,,2405.02066.pdf,WateRF: Robust Watermarks in Radiance Fields for Protection of Copyrights,"The advances in the Neural Radiance Fields (NeRF) research offer extensive
+applications in diverse domains, but protecting their copyrights has not yet
+been researched in depth. Recently, NeRF watermarking has been considered one
+of the pivotal solutions for safely deploying NeRF-based 3D representations.
+However, existing methods are designed to apply only to implicit or explicit
+NeRF representations. In this work, we introduce an innovative watermarking
+method that can be employed in both representations of NeRF. This is achieved
+by fine-tuning NeRF to embed binary messages in the rendering process. In
+detail, we propose utilizing the discrete wavelet transform in the NeRF space
+for watermarking. Furthermore, we adopt a deferred back-propagation technique
+and introduce a combination with the patch-wise loss to improve rendering
+quality and bit accuracy with minimum trade-offs. We evaluate our method in
+three different aspects: capacity, invisibility, and robustness of the embedded
+watermarks in the 2D-rendered images. Our method achieves state-of-the-art
+performance with faster training speed over the compared state-of-the-art
+methods.",cs.CV,"['cs.CV', 'eess.IV']"
+ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association,Shuxiao Ding · Lukas Schneider · Marius Cordts · Jürgen Gall,https://github.com/dsx0511/ADA-Track,https://arxiv.org/abs/2405.08909,,2405.08909.pdf,ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association,"Many query-based approaches for 3D Multi-Object Tracking (MOT) adopt the
+tracking-by-attention paradigm, utilizing track queries for identity-consistent
+detection and object queries for identity-agnostic track spawning.
+Tracking-by-attention, however, entangles detection and tracking queries in one
+embedding for both the detection and tracking task, which is sub-optimal. Other
+approaches resemble the tracking-by-detection paradigm, detecting objects using
+decoupled track and detection queries followed by a subsequent association.
+These methods, however, do not leverage synergies between the detection and
+association task. Combining the strengths of both paradigms, we introduce
+ADA-Track, a novel end-to-end framework for 3D MOT from multi-view cameras. We
+introduce a learnable data association module based on edge-augmented
+cross-attention, leveraging appearance and geometric features. Furthermore, we
+integrate this association module into the decoder layer of a DETR-based 3D
+detector, enabling simultaneous DETR-like query-to-image cross-attention for
+detection and query-to-query cross-attention for data association. By stacking
+these decoder layers, queries are refined for the detection and association
+task alternately, effectively harnessing the task dependencies. We evaluate our
+method on the nuScenes dataset and demonstrate the advantage of our approach
+compared to the two previous paradigms. Code is available at
+https://github.com/dsx0511/ADA-Track.",cs.CV,['cs.CV']
+Scale Decoupled Distillation,Shicai Wei · Chunbo Luo · Yang Luo, ,https://arxiv.org/abs/2403.13512,,2403.13512.pdf,Scale Decoupled Distillation,"Logit knowledge distillation attracts increasing attention due to its
+practicality in recent studies. However, it often suffers inferior performance
+compared to the feature knowledge distillation. In this paper, we argue that
+existing logit-based methods may be sub-optimal since they only leverage the
+global logit output that couples multiple semantic knowledge. This may transfer
+ambiguous knowledge to the student and mislead its learning. To this end, we
+propose a simple but effective method, i.e., Scale Decoupled Distillation
+(SDD), for logit knowledge distillation. SDD decouples the global logit output
+into multiple local logit outputs and establishes distillation pipelines for
+them. This helps the student to mine and inherit fine-grained and unambiguous
+logit knowledge. Moreover, the decoupled knowledge can be further divided into
+consistent and complementary logit knowledge that transfers the semantic
+information and sample ambiguity, respectively. By increasing the weight of
+complementary parts, SDD can guide the student to focus more on ambiguous
+samples, improving its discrimination ability. Extensive experiments on several
+benchmark datasets demonstrate the effectiveness of SDD for wide
+teacher-student pairs, especially in the fine-grained classification task. Code
+is available at: https://github.com/shicaiwei123/SDD-CVPR2024",cs.CV,"['cs.CV', 'cs.AI']"
+SIRA: Scalable Inter-frame Relation and Association for Radar Perception,Ryoma Yataka · Pu (Perry) Wang · Petros Boufounos · Ryuhei Takahashi, ,,https://www.semanticscholar.org/paper/Radar-Perception-with-Scalable-Connective-Temporal-Yataka-Wang/78d83560c7e2aee39d8153bafc815482dcbd163e,,,,,nan
+Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning,Rui Li · Tobias Fischer · Mattia Segu · Marc Pollefeys · Luc Van Gool · Federico Tombari, ,https://arxiv.org/abs/2404.03658,,2404.03658.pdf,Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning,"Recovering the 3D scene geometry from a single view is a fundamental yet
+ill-posed problem in computer vision. While classical depth estimation methods
+infer only a 2.5D scene representation limited to the image plane, recent
+approaches based on radiance fields reconstruct a full 3D representation.
+However, these methods still struggle with occluded regions since inferring
+geometry without visual observation requires (i) semantic knowledge of the
+surroundings, and (ii) reasoning about spatial context. We propose KYN, a novel
+method for single-view scene reconstruction that reasons about semantic and
+spatial context to predict each point's density. We introduce a vision-language
+modulation module to enrich point features with fine-grained semantic
+information. We aggregate point representations across the scene through a
+language-guided spatial attention mechanism to yield per-point density
+predictions aware of the 3D semantic context. We show that KYN improves 3D
+shape recovery compared to predicting density for each 3D point in isolation.
+We achieve state-of-the-art results in scene and object reconstruction on
+KITTI-360, and show improved zero-shot generalization compared to prior work.
+Project page: https://ruili3.github.io/kyn.",cs.CV,['cs.CV']
+Rich Human Feedback for Text-to-Image Generation,Youwei Liang · Junfeng He · Gang Li · Peizhao Li · Arseniy Klimovskiy · Nicholas Carolan · Jiao Sun · Jordi Pont-Tuset · Sarah Young · Feng Yang · Junjie Ke · Krishnamurthy Dvijotham · Katherine Collins · Yiwen Luo · Yang Li · Kai Kohlhoff · Deepak Ramachandran · Vidhya Navalpakkam, ,https://arxiv.org/abs/2312.10240,,2312.10240.pdf,Rich Human Feedback for Text-to-Image Generation,"Recent Text-to-Image (T2I) generation models such as Stable Diffusion and
+Imagen have made significant progress in generating high-resolution images
+based on text descriptions. However, many generated images still suffer from
+issues such as artifacts/implausibility, misalignment with text descriptions,
+and low aesthetic quality. Inspired by the success of Reinforcement Learning
+with Human Feedback (RLHF) for large language models, prior works collected
+human-provided scores as feedback on generated images and trained a reward
+model to improve the T2I generation. In this paper, we enrich the feedback
+signal by (i) marking image regions that are implausible or misaligned with the
+text, and (ii) annotating which words in the text prompt are misrepresented or
+missing on the image. We collect such rich human feedback on 18K generated
+images (RichHF-18K) and train a multimodal transformer to predict the rich
+feedback automatically. We show that the predicted rich human feedback can be
+leveraged to improve image generation, for example, by selecting high-quality
+training data to finetune and improve the generative models, or by creating
+masks with predicted heatmaps to inpaint the problematic regions. Notably, the
+improvements generalize to models (Muse) beyond those used to generate the
+images on which human feedback data were collected (Stable Diffusion variants).
+The RichHF-18K data set will be released in our GitHub repository:
+https://github.com/google-research/google-research/tree/master/richhf_18k.",cs.CV,['cs.CV']
+"AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond",Zixiang Zhou · Yu Wan · Baoyuan Wang,https://github.com/zixiangzhou916/AvatarGPT,,https://www.semanticscholar.org/paper/AvatarGPT:-All-in-One-Framework-for-Motion-and-Zhou-Wan/b4e6f30ab07666dc7d485b24f072f2533609545c/figure/4,,,,,nan
+SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection,Gang Zhang · Chen Junnan · Guohuan Gao · Jianmin Li · Si Liu · Xiaolin Hu, ,https://arxiv.org/abs/2403.05817,,2403.05817.pdf,SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection,"LiDAR-based 3D object detection plays an essential role in autonomous
+driving. Existing high-performing 3D object detectors usually build dense
+feature maps in the backbone network and prediction head. However, the
+computational costs introduced by the dense feature maps grow quadratically as
+the perception range increases, making these models hard to scale up to
+long-range detection. Some recent works have attempted to construct fully
+sparse detectors to solve this issue; nevertheless, the resulting models either
+rely on a complex multi-stage pipeline or exhibit inferior performance. In this
+work, we propose SAFDNet, a straightforward yet highly effective architecture,
+tailored for fully sparse 3D object detection. In SAFDNet, an adaptive feature
+diffusion strategy is designed to address the center feature missing problem.
+We conducted extensive experiments on Waymo Open, nuScenes, and Argoverse2
+datasets. SAFDNet performed slightly better than the previous SOTA on the first
+two datasets but much better on the last dataset, which features long-range
+detection, verifying the efficacy of SAFDNet in scenarios where long-range
+detection is required. Notably, on Argoverse2, SAFDNet surpassed the previous
+best hybrid detector HEDNet by 2.6% mAP while being 2.1x faster, and yielded
+2.1% mAP gains over the previous best sparse detector FSDv2 while being 1.3x
+faster. The code will be available at https://github.com/zhanggang001/HEDNet.",cs.CV,['cs.CV']
+Neural Clustering based Visual Representation Learning,Guikun Chen · Xia Li · Yi Yang · Wenguan Wang,https://github.com/guikunchen/FEC,https://arxiv.org/abs/2403.17409,,2403.17409.pdf,Neural Clustering based Visual Representation Learning,"We investigate a fundamental aspect of machine vision: the measurement of
+features, by revisiting clustering, one of the most classic approaches in
+machine learning and data analysis. Existing visual feature extractors,
+including ConvNets, ViTs, and MLPs, represent an image as rectangular regions.
+Though prevalent, such a grid-style paradigm is built upon engineering practice
+and lacks explicit modeling of data distribution. In this work, we propose
+feature extraction with clustering (FEC), a conceptually elegant yet
+surprisingly ad-hoc interpretable neural clustering framework, which views
+feature extraction as a process of selecting representatives from data and thus
+automatically captures the underlying data distribution. Given an image, FEC
+alternates between grouping pixels into individual clusters to abstract
+representatives and updating the deep features of pixels with current
+representatives. Such an iterative working mechanism is implemented in the form
+of several neural layers and the final representatives can be used for
+downstream tasks. The cluster assignments across layers, which can be viewed
+and inspected by humans, make the forward process of FEC fully transparent and
+empower it with promising ad-hoc interpretability. Extensive experiments on
+various visual recognition models and tasks verify the effectiveness,
+generality, and interpretability of FEC. We expect this work will provoke a
+rethink of the current de facto grid-style paradigm.",cs.CV,['cs.CV']
+Neural Redshift: Random Networks are not Random Functions,Damien Teney · Armand Nicolicioiu · Valentin Hartmann · Ehsan Abbasnejad, ,https://arxiv.org/abs/2403.02241,,2403.02241.pdf,Neural Redshift: Random Networks are not Random Functions,"Our understanding of the generalization capabilities of neural networks (NNs)
+is still incomplete. Prevailing explanations are based on implicit biases of
+gradient descent (GD) but they cannot account for the capabilities of models
+from gradient-free methods nor the simplicity bias recently observed in
+untrained networks. This paper seeks other sources of generalization in NNs.
+  Findings. To understand the inductive biases provided by architectures
+independently from GD, we examine untrained, random-weight networks. Even
+simple MLPs show strong inductive biases: uniform sampling in weight space
+yields a very biased distribution of functions in terms of complexity. But
+unlike common wisdom, NNs do not have an inherent ""simplicity bias"". This
+property depends on components such as ReLUs, residual connections, and layer
+normalizations. Alternative architectures can be built with a bias for any
+level of complexity. Transformers also inherit all these properties from their
+building blocks.
+  Implications. We provide a fresh explanation for the success of deep learning
+independent from gradient-based training. It points at promising avenues for
+controlling the solutions implemented by trained models.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Segment Any Event Streams via Weighted Adaptation of Pivotal Tokens,Zhiwen Chen · Zhiyu Zhu · Yifan Zhang · Junhui Hou · Guangming Shi · Jinjian Wu,https://github.com/happychenpipi/EventSAM/,https://arxiv.org/abs/2312.16222,,2312.16222.pdf,Segment Any Events via Weighted Adaptation of Pivotal Tokens,"In this paper, we delve into the nuanced challenge of tailoring the Segment
+Anything Models (SAMs) for integration with event data, with the overarching
+objective of attaining robust and universal object segmentation within the
+event-centric domain. One pivotal issue at the heart of this endeavor is the
+precise alignment and calibration of embeddings derived from event-centric data
+such that they harmoniously coincide with those originating from RGB imagery.
+Capitalizing on the vast repositories of datasets with paired events and RGB
+images, our proposition is to harness and extrapolate the profound knowledge
+encapsulated within the pre-trained SAM framework. As a cornerstone to
+achieving this, we introduce a multi-scale feature distillation methodology.
+This methodology rigorously optimizes the alignment of token embeddings
+originating from event data with their RGB image counterparts, thereby
+preserving and enhancing the robustness of the overall architecture.
+Considering the distinct significance that token embeddings from intermediate
+layers hold for higher-level embeddings, our strategy is centered on accurately
+calibrating the pivotal token embeddings. This targeted calibration is aimed at
+effectively managing the discrepancies in high-level embeddings originating
+from both the event and image domains. Extensive experiments on different
+datasets demonstrate the effectiveness of the proposed distillation method.
+Code in http://github.com/happychenpipi/EventSAM.",cs.CV,['cs.CV']
+Continual Forgetting for Pre-trained Vision Models,Hongbo Zhao · Bolin Ni · Junsong Fan · Yuxi Wang · Yuntao Chen · Gaofeng Meng · Zhaoxiang Zhang,https://github.com/bjzhb666/GS-LoRA,https://arxiv.org/abs/2403.11530,,2403.11530.pdf,Continual Forgetting for Pre-trained Vision Models,"For privacy and security concerns, the need to erase unwanted information
+from pre-trained vision models is becoming evident nowadays. In real-world
+scenarios, erasure requests originate at any time from both users and model
+owners. These requests usually form a sequence. Therefore, under such a
+setting, selective information is expected to be continuously removed from a
+pre-trained model while maintaining the rest. We define this problem as
+continual forgetting and identify two key challenges. (i) For unwanted
+knowledge, efficient and effective deleting is crucial. (ii) For remaining
+knowledge, the impact brought by the forgetting procedure should be minimal. To
+address them, we propose Group Sparse LoRA (GS-LoRA). Specifically, towards
+(i), we use LoRA modules to fine-tune the FFN layers in Transformer blocks for
+each forgetting task independently, and towards (ii), a simple group sparse
+regularization is adopted, enabling automatic selection of specific LoRA groups
+and zeroing out the others. GS-LoRA is effective, parameter-efficient,
+data-efficient, and easy to implement. We conduct extensive experiments on face
+recognition, object detection and image classification and demonstrate that
+GS-LoRA manages to forget specific classes with minimal impact on other
+classes. Codes will be released on \url{https://github.com/bjzhb666/GS-LoRA}.",cs.CV,['cs.CV']
+Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling,Jianan Fan · Dongnan Liu · Hang Chang · Heng Huang · Mei Chen · Weidong Cai, ,https://arxiv.org/abs/2403.01053,,2403.01053.pdf,Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling,"Machine learning holds tremendous promise for transforming the fundamental
+practice of scientific discovery by virtue of its data-driven nature. With the
+ever-increasing stream of research data collection, it would be appealing to
+autonomously explore patterns and insights from observational data for
+discovering novel classes of phenotypes and concepts. However, in the
+biomedical domain, there are several challenges inherently presented in the
+cumulated data which hamper the progress of novel class discovery. The
+non-i.i.d. data distribution accompanied by the severe imbalance among
+different groups of classes essentially leads to ambiguous and biased semantic
+representations. In this work, we present a geometry-constrained probabilistic
+modeling treatment to resolve the identified issues. First, we propose to
+parameterize the approximated posterior of instance embedding as a marginal von
+MisesFisher distribution to account for the interference of distributional
+latent bias. Then, we incorporate a suite of critical geometric properties to
+impose proper constraints on the layout of constructed embedding space, which
+in turn minimizes the uncontrollable risk for unknown class learning and
+structuring. Furthermore, a spectral graph-theoretic method is devised to
+estimate the number of potential novel classes. It inherits two intriguing
+merits compared to existent approaches, namely high computational efficiency
+and flexibility for taxonomy-adaptive estimation. Extensive experiments across
+various biomedical scenarios substantiate the effectiveness and general
+applicability of our method.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection,Heng Zhang · Qiuyu Zhao · Linyu Zheng · Hao Zeng · Zhiwei Ge · Tianhao Li · Sulong Xu, ,https://arxiv.org/abs/2310.16667,,2310.16667.pdf,CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection,"Deriving reliable region-word alignment from image-text pairs is critical to
+learn object-level vision-language representations for open-vocabulary object
+detection. Existing methods typically rely on pre-trained or self-trained
+vision-language models for alignment, which are prone to limitations in
+localization accuracy or generalization capabilities. In this paper, we propose
+CoDet, a novel approach that overcomes the reliance on pre-aligned
+vision-language space by reformulating region-word alignment as a co-occurring
+object discovery problem. Intuitively, by grouping images that mention a shared
+concept in their captions, objects corresponding to the shared concept shall
+exhibit high co-occurrence among the group. CoDet then leverages visual
+similarities to discover the co-occurring objects and align them with the
+shared concept. Extensive experiments demonstrate that CoDet has superior
+performances and compelling scalability in open-vocabulary detection, e.g., by
+scaling up the visual backbone, CoDet achieves 37.0 $\text{AP}^m_{novel}$ and
+44.7 $\text{AP}^m_{all}$ on OV-LVIS, surpassing the previous SoTA by 4.2
+$\text{AP}^m_{novel}$ and 9.8 $\text{AP}^m_{all}$. Code is available at
+https://github.com/CVMI-Lab/CoDet.",cs.CV,['cs.CV']
+Blind Image Quality Assessment Based on Geometric Order Learning,Nyeong-Ho Shin · Seon-Ho Lee · Chang-Su Kim, ,https://arxiv.org/abs/2404.14949,,2404.14949.pdf,Multi-Modal Prompt Learning on Blind Image Quality Assessment,"Image Quality Assessment (IQA) models benefit significantly from semantic
+information, which allows them to treat different types of objects distinctly.
+Currently, leveraging semantic information to enhance IQA is a crucial research
+direction. Traditional methods, hindered by a lack of sufficiently annotated
+data, have employed the CLIP image-text pretraining model as their backbone to
+gain semantic awareness. However, the generalist nature of these pre-trained
+Vision-Language (VL) models often renders them suboptimal for IQA-specific
+tasks. Recent approaches have attempted to address this mismatch using prompt
+technology, but these solutions have shortcomings. Existing prompt-based VL
+models overly focus on incremental semantic information from text, neglecting
+the rich insights available from visual data analysis. This imbalance limits
+their performance improvements in IQA tasks. This paper introduces an
+innovative multi-modal prompt-based methodology for IQA. Our approach employs
+carefully crafted prompts that synergistically mine incremental semantic
+information from both visual and linguistic data. Specifically, in the visual
+branch, we introduce a multi-layer prompt structure to enhance the VL model's
+adaptability. In the text branch, we deploy a dual-prompt scheme that steers
+the model to recognize and differentiate between scene category and distortion
+type, thereby refining the model's capacity to assess image quality. Our
+experimental findings underscore the effectiveness of our method over existing
+Blind Image Quality Assessment (BIQA) approaches. Notably, it demonstrates
+competitive performance across various datasets. Our method achieves Spearman
+Rank Correlation Coefficient (SRCC) values of 0.961(surpassing 0.946 in CSIQ)
+and 0.941 (exceeding 0.930 in KADID), illustrating its robustness and accuracy
+in diverse contexts.",cs.CV,['cs.CV']
+MemoNav: Working Memory Model for Visual Navigation,Hongxin Li · Zeyu Wang · Xu Yang · yuran Yang · Shuqi Mei · Zhaoxiang Zhang,https://github.com/ZJULiHongxin/MemoNav,https://arxiv.org/abs/2402.19161v1,,2402.19161v1.pdf,MemoNav: Working Memory Model for Visual Navigation,"Image-goal navigation is a challenging task that requires an agent to
+navigate to a goal indicated by an image in unfamiliar environments. Existing
+methods utilizing diverse scene memories suffer from inefficient exploration
+since they use all historical observations for decision-making without
+considering the goal-relevant fraction. To address this limitation, we present
+MemoNav, a novel memory model for image-goal navigation, which utilizes a
+working memory-inspired pipeline to improve navigation performance.
+Specifically, we employ three types of navigation memory. The node features on
+a map are stored in the short-term memory (STM), as these features are
+dynamically updated. A forgetting module then retains the informative STM
+fraction to increase efficiency. We also introduce long-term memory (LTM) to
+learn global scene representations by progressively aggregating STM features.
+Subsequently, a graph attention module encodes the retained STM and the LTM to
+generate working memory (WM) which contains the scene features essential for
+efficient navigation. The synergy among these three memory types boosts
+navigation performance by enabling the agent to learn and leverage
+goal-relevant scene features within a topological map. Our evaluation on
+multi-goal tasks demonstrates that MemoNav significantly outperforms previous
+methods across all difficulty levels in both Gibson and Matterport3D scenes.
+Qualitative results further illustrate that MemoNav plans more efficient
+routes.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts,Fei Ni · Jianye Hao · Shiguang Wu · Longxin Kou · Jiashun Liu · YAN ZHENG · Bin Wang · Yuzheng Zhuang, ,,https://pub.towardsai.net/ai-robotics-breakthroughs-and-trends-at-cvpr-2024-d4a83b5f9564,,,,,nan
+RCBEVDet: Radar-camera Fusion in Bird’s Eye View for 3D Object Detection,Zhiwei Lin · Zhe Liu · Zhongyu Xia · Xinhao Wang · Yongtao Wang · Shengxiang Qi · Yang Dong · Nan Dong · Le Zhang · Ce Zhu, ,https://arxiv.org/abs/2403.16440,,2403.16440.pdf,RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection,"Three-dimensional object detection is one of the key tasks in autonomous
+driving. To reduce costs in practice, low-cost multi-view cameras for 3D object
+detection are proposed to replace the expansive LiDAR sensors. However, relying
+solely on cameras is difficult to achieve highly accurate and robust 3D object
+detection. An effective solution to this issue is combining multi-view cameras
+with the economical millimeter-wave radar sensor to achieve more reliable
+multi-modal 3D object detection. In this paper, we introduce RCBEVDet, a
+radar-camera fusion 3D object detection method in the bird's eye view (BEV).
+Specifically, we first design RadarBEVNet for radar BEV feature extraction.
+RadarBEVNet consists of a dual-stream radar backbone and a Radar Cross-Section
+(RCS) aware BEV encoder. In the dual-stream radar backbone, a point-based
+encoder and a transformer-based encoder are proposed to extract radar features,
+with an injection and extraction module to facilitate communication between the
+two encoders. The RCS-aware BEV encoder takes RCS as the object size prior to
+scattering the point feature in BEV. Besides, we present the Cross-Attention
+Multi-layer Fusion module to automatically align the multi-modal BEV feature
+from radar and camera with the deformable attention mechanism, and then fuse
+the feature with channel and spatial fusion layers. Experimental results show
+that RCBEVDet achieves new state-of-the-art radar-camera fusion results on
+nuScenes and view-of-delft (VoD) 3D object detection benchmarks. Furthermore,
+RCBEVDet achieves better 3D detection results than all real-time camera-only
+and radar-camera 3D object detectors with a faster inference speed at 21~28
+FPS. The source code will be released at https://github.com/VDIGPKU/RCBEVDet.",cs.CV,['cs.CV']
+Instance-based Max-margin for Practical Few-shot Recognition,Minghao Fu · Ke Zhu,https://github.com/heekhero/IbM2,https://arxiv.org/abs/2312.07856,,2312.07856.pdf,DTL: Disentangled Transfer Learning for Visual Recognition,"When pre-trained models become rapidly larger, the cost of fine-tuning on
+downstream tasks steadily increases, too. To economically fine-tune these
+models, parameter-efficient transfer learning (PETL) is proposed, which only
+tunes a tiny subset of trainable parameters to efficiently learn quality
+representations. However, current PETL methods are facing the dilemma that
+during training the GPU memory footprint is not effectively reduced as
+trainable parameters. PETL will likely fail, too, if the full fine-tuning
+encounters the out-of-GPU-memory issue. This phenomenon happens because
+trainable parameters from these methods are generally entangled with the
+backbone, such that a lot of intermediate states have to be stored in GPU
+memory for gradient propagation. To alleviate this problem, we introduce
+Disentangled Transfer Learning (DTL), which disentangles the trainable
+parameters from the backbone using a lightweight Compact Side Network (CSN). By
+progressively extracting task-specific information with a few low-rank linear
+mappings and appropriately adding the information back to the backbone, CSN
+effectively realizes knowledge transfer in various downstream tasks. We
+conducted extensive experiments to validate the effectiveness of our method.
+The proposed method not only reduces a large amount of GPU memory usage and
+trainable parameters, but also outperforms existing PETL methods by a
+significant margin in accuracy, achieving new state-of-the-art on several
+standard benchmarks. The code is available at https://github.com/heekhero/DTL.",cs.CV,"['cs.CV', 'cs.AI']"
+TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding,Shuhuai Ren · Linli Yao · Shicheng Li · Xu Sun · Lu Hou,https://github.com/RenShuhuai-Andy/TimeChat,https://arxiv.org/abs/2312.02051,,2312.02051.pdf,TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding,"This work proposes TimeChat, a time-sensitive multimodal large language model
+specifically designed for long video understanding. Our model incorporates two
+key architectural contributions: (1) a timestamp-aware frame encoder that binds
+visual content with the timestamp of each frame, and (2) a sliding video
+Q-Former that produces a video token sequence of varying lengths to accommodate
+videos of various durations. Additionally, we construct an instruction-tuning
+dataset, encompassing 6 tasks and a total of 125K instances, to further enhance
+TimeChat's instruction-following performance. Experiment results across various
+video understanding tasks, such as dense captioning, temporal grounding, and
+highlight detection, demonstrate TimeChat's strong zero-shot temporal
+localization and reasoning capabilities. For example, it achieves +9.2 F1 score
+and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (IoU=0.5)
+on Charades-STA, compared to state-of-the-art video large language models,
+holding the potential to serve as a versatile video assistant for long-form
+video comprehension tasks and satisfy realistic user requirements.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis,Yufei Ye · Abhinav Gupta · Kris Kitani · Shubham Tulsiani, ,https://arxiv.org/abs/2404.12383,,2404.12383.pdf,G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis,"We propose G-HOP, a denoising diffusion based generative prior for
+hand-object interactions that allows modeling both the 3D object and a human
+hand, conditioned on the object category. To learn a 3D spatial diffusion model
+that can capture this joint distribution, we represent the human hand via a
+skeletal distance field to obtain a representation aligned with the (latent)
+signed distance field for the object. We show that this hand-object prior can
+then serve as generic guidance to facilitate other tasks like reconstruction
+from interaction clip and human grasp synthesis. We believe that our model,
+trained by aggregating seven diverse real-world interaction datasets spanning
+across 155 categories, represents a first approach that allows jointly
+generating both hand and object. Our empirical evaluations demonstrate the
+benefit of this joint prior in video-based reconstruction and human grasp
+synthesis, outperforming current task-specific baselines.
+  Project website: https://judyye.github.io/ghop-www",cs.CV,['cs.CV']
+VTimeLLM: Empower LLM to Grasp Video Moments,Bin Huang · Xin Wang · Hong Chen · Zihan Song · Wenwu Zhu, ,https://arxiv.org/abs/2311.18445v1,,2311.18445v1.pdf,VTimeLLM: Empower LLM to Grasp Video Moments,"Large language models (LLMs) have shown remarkable text understanding
+capabilities, which have been extended as Video LLMs to handle video data for
+comprehending visual details. However, existing Video LLMs can only provide a
+coarse description of the entire video, failing to capture the precise start
+and end time boundary of specific events. In this paper, we solve this issue
+via proposing VTimeLLM, a novel Video LLM designed for fine-grained video
+moment understanding and reasoning with respect to time boundary. Specifically,
+our VTimeLLM adopts a boundary-aware three-stage training strategy, which
+respectively utilizes image-text pairs for feature alignment, multiple-event
+videos to increase temporal-boundary awareness, and high-quality
+video-instruction tuning to further improve temporal understanding ability as
+well as align with human intents. Extensive experiments demonstrate that in
+fine-grained time-related comprehension tasks for videos such as Temporal Video
+Grounding and Dense Video Captioning, VTimeLLM significantly outperforms
+existing Video LLMs. Besides, benefits from the fine-grained temporal
+understanding of the videos further enable VTimeLLM to beat existing Video LLMs
+in video dialogue benchmark, showing its superior cross-modal understanding and
+reasoning abilities.",cs.CV,['cs.CV']
+DITTO: Dual and Integrated Latent Topologies for Implicit 3D Reconstruction,Jaehyeok Shim · Kyungdon Joo, ,https://arxiv.org/abs/2403.05005,,2403.05005.pdf,DITTO: Dual and Integrated Latent Topologies for Implicit 3D Reconstruction,"We propose a novel concept of dual and integrated latent topologies (DITTO in
+short) for implicit 3D reconstruction from noisy and sparse point clouds. Most
+existing methods predominantly focus on single latent type, such as point or
+grid latents. In contrast, the proposed DITTO leverages both point and grid
+latents (i.e., dual latent) to enhance their strengths, the stability of grid
+latents and the detail-rich capability of point latents. Concretely, DITTO
+consists of dual latent encoder and integrated implicit decoder. In the dual
+latent encoder, a dual latent layer, which is the key module block composing
+the encoder, refines both latents in parallel, maintaining their distinct
+shapes and enabling recursive interaction. Notably, a newly proposed dynamic
+sparse point transformer within the dual latent layer effectively refines point
+latents. Then, the integrated implicit decoder systematically combines these
+refined latents, achieving high-fidelity 3D reconstruction and surpassing
+previous state-of-the-art methods on object- and scene-level datasets,
+especially in thin and detailed structures.",cs.CV,['cs.CV']
+Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimation,Lior Talker · Aviad Cohen · Erez Yosef · Alexandra Dana · Michael Dinerstein,https://github.com/liortalker/MindTheEdge,,https://www.youtube.com/watch?v=WPmbAnJk3rE,,,,,nan
+StyleCineGAN: Landscape Cinemagraph Generation using a Pre-trained StyleGAN,Jongwoo Choi · Kwanggyoon Seo · Amirsaman Ashtari · Junyong Noh,https://jeolpyeoni.github.io/stylecinegan_project/,https://arxiv.org/abs/2403.14186,,2403.14186.pdf,StyleCineGAN: Landscape Cinemagraph Generation using a Pre-trained StyleGAN,"We propose a method that can generate cinemagraphs automatically from a still
+landscape image using a pre-trained StyleGAN. Inspired by the success of recent
+unconditional video generation, we leverage a powerful pre-trained image
+generator to synthesize high-quality cinemagraphs. Unlike previous approaches
+that mainly utilize the latent space of a pre-trained StyleGAN, our approach
+utilizes its deep feature space for both GAN inversion and cinemagraph
+generation. Specifically, we propose multi-scale deep feature warping (MSDFW),
+which warps the intermediate features of a pre-trained StyleGAN at different
+resolutions. By using MSDFW, the generated cinemagraphs are of high resolution
+and exhibit plausible looping animation. We demonstrate the superiority of our
+method through user studies and quantitative comparisons with state-of-the-art
+cinemagraph generation methods and a video generation method that uses a
+pre-trained StyleGAN.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+Decoupled Pseudo-labeling in Semi-Supervised Monocular 3D Object Detection,Jiacheng Zhang · Jiaming Li · Xiangru Lin · Wei Zhang · Xiao Tan · Junyu Han · Errui Ding · Jingdong Wang · Guanbin Li, ,https://arxiv.org/abs/2403.17387,,2403.17387.pdf,Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection,"We delve into pseudo-labeling for semi-supervised monocular 3D object
+detection (SSM3OD) and discover two primary issues: a misalignment between the
+prediction quality of 3D and 2D attributes and the tendency of depth
+supervision derived from pseudo-labels to be noisy, leading to significant
+optimization conflicts with other reliable forms of supervision. We introduce a
+novel decoupled pseudo-labeling (DPL) approach for SSM3OD. Our approach
+features a Decoupled Pseudo-label Generation (DPG) module, designed to
+efficiently generate pseudo-labels by separately processing 2D and 3D
+attributes. This module incorporates a unique homography-based method for
+identifying dependable pseudo-labels in BEV space, specifically for 3D
+attributes. Additionally, we present a DepthGradient Projection (DGP) module to
+mitigate optimization conflicts caused by noisy depth supervision of
+pseudo-labels, effectively decoupling the depth gradient and removing
+conflicting gradients. This dual decoupling strategy-at both the pseudo-label
+generation and gradient levels-significantly improves the utilization of
+pseudo-labels in SSM3OD. Our comprehensive experiments on the KITTI benchmark
+demonstrate the superiority of our method over existing approaches.",cs.CV,['cs.CV']
+T-VSL: Text-Guided Visual Sound Source Localization in Mixtures,Tanvir Mahmud · Yapeng Tian · Diana Marculescu, ,https://arxiv.org/abs/2404.01751v1,,2404.01751v1.pdf,T-VSL: Text-Guided Visual Sound Source Localization in Mixtures,"Visual sound source localization poses a significant challenge in identifying
+the semantic region of each sounding source within a video. Existing
+self-supervised and weakly supervised source localization methods struggle to
+accurately distinguish the semantic regions of each sounding object,
+particularly in multi-source mixtures. These methods often rely on audio-visual
+correspondence as guidance, which can lead to substantial performance drops in
+complex multi-source localization scenarios. The lack of access to individual
+source sounds in multi-source mixtures during training exacerbates the
+difficulty of learning effective audio-visual correspondence for localization.
+To address this limitation, in this paper, we propose incorporating the text
+modality as an intermediate feature guide using tri-modal joint embedding
+models (e.g., AudioCLIP) to disentangle the semantic audio-visual source
+correspondence in multi-source mixtures. Our framework, dubbed T-VSL, begins by
+predicting the class of sounding entities in mixtures. Subsequently, the
+textual representation of each sounding source is employed as guidance to
+disentangle fine-grained audio-visual source correspondence from multi-source
+mixtures, leveraging the tri-modal AudioCLIP embedding. This approach enables
+our framework to handle a flexible number of sources and exhibits promising
+zero-shot transferability to unseen classes during test time. Extensive
+experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets
+demonstrate significant performance improvements over state-of-the-art methods.",cs.CV,"['cs.CV', 'cs.SD', 'eess.AS']"
+USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation,Xiaoqi Wang · Wenbin He · Xiwei Xuan · Clint Sebastian · Jorge Piazentin Ono · Xin Li · Sima Behpour · Thang Doan · Liang Gou · Shen · Liu Ren, ,http://export.arxiv.org/abs/2307.00764,,2307.00764.pdf,Hierarchical Open-vocabulary Universal Image Segmentation,"Open-vocabulary image segmentation aims to partition an image into semantic
+regions according to arbitrary text descriptions. However, complex visual
+scenes can be naturally decomposed into simpler parts and abstracted at
+multiple levels of granularity, introducing inherent segmentation ambiguity.
+Unlike existing methods that typically sidestep this ambiguity and treat it as
+an external factor, our approach actively incorporates a hierarchical
+representation encompassing different semantic-levels into the learning
+process. We propose a decoupled text-image fusion mechanism and representation
+learning modules for both ""things"" and ""stuff"". Additionally, we systematically
+examine the differences that exist in the textual and visual features between
+these types of categories. Our resulting model, named HIPIE, tackles
+HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a
+unified framework. Benchmarked on over 40 datasets, e.g., ADE20K, COCO,
+Pascal-VOC Part, RefCOCO/RefCOCOg, ODinW and SeginW, HIPIE achieves the
+state-of-the-art results at various levels of image comprehension, including
+semantic-level (e.g., semantic segmentation), instance-level (e.g.,
+panoptic/referring segmentation and object detection), as well as part-level
+(e.g., part/subpart segmentation) tasks. Our code is released at
+https://github.com/berkeley-hipie/HIPIE.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+DVMNet: Computing Relative Pose for Unseen Objects Beyond Hypotheses,Chen Zhao · Tong Zhang · Zheng Dang · Mathieu Salzmann, ,https://arxiv.org/abs/2403.13683,,2403.13683.pdf,DVMNet: Computing Relative Pose for Unseen Objects Beyond Hypotheses,"Determining the relative pose of an object between two images is pivotal to
+the success of generalizable object pose estimation. Existing approaches
+typically approximate the continuous pose representation with a large number of
+discrete pose hypotheses, which incurs a computationally expensive process of
+scoring each hypothesis at test time. By contrast, we present a Deep Voxel
+Matching Network (DVMNet) that eliminates the need for pose hypotheses and
+computes the relative object pose in a single pass. To this end, we map the two
+input RGB images, reference and query, to their respective voxelized 3D
+representations. We then pass the resulting voxels through a pose estimation
+module, where the voxels are aligned and the pose is computed in an end-to-end
+fashion by solving a least-squares problem. To enhance robustness, we introduce
+a weighted closest voxel algorithm capable of mitigating the impact of noisy
+voxels. We conduct extensive experiments on the CO3D, LINEMOD, and Objaverse
+datasets, demonstrating that our method delivers more accurate relative pose
+estimates for novel objects at a lower computational cost compared to
+state-of-the-art methods. Our code is released at:
+https://github.com/sailor-z/DVMNet/.",cs.CV,"['cs.CV', 'cs.RO']"
+pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction,David Charatan · Sizhe Lester Li · Andrea Tagliasacchi · Vincent Sitzmann, ,https://arxiv.org/abs/2312.12337,,2312.12337.pdf,pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction,"We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D
+radiance fields parameterized by 3D Gaussian primitives from pairs of images.
+Our model features real-time and memory-efficient rendering for scalable
+training as well as fast 3D reconstruction at inference time. To overcome local
+minima inherent to sparse and locally supported representations, we predict a
+dense probability distribution over 3D and sample Gaussian means from that
+probability distribution. We make this sampling operation differentiable via a
+reparameterization trick, allowing us to back-propagate gradients through the
+Gaussian splatting representation. We benchmark our method on wide-baseline
+novel view synthesis on the real-world RealEstate10k and ACID datasets, where
+we outperform state-of-the-art light field transformers and accelerate
+rendering by 2.5 orders of magnitude while reconstructing an interpretable and
+editable 3D radiance field.",cs.CV,"['cs.CV', 'cs.LG']"
+MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation,Yanhui Wang · Jianmin Bao · Wenming Weng · Ruoyu Feng · Dacheng Yin · Tao Yang · Jingxu Zhang · Qi Dai · Zhiyuan Zhao · Chunyu Wang · Kai Qiu · Yuhui Yuan · Xiaoyan Sun · Chong Luo · Baining Guo, ,https://arxiv.org/abs/2311.18829,,2311.18829.pdf,MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation,"We present MicroCinema, a straightforward yet effective framework for
+high-quality and coherent text-to-video generation. Unlike existing approaches
+that align text prompts with video directly, MicroCinema introduces a
+Divide-and-Conquer strategy which divides the text-to-video into a two-stage
+process: text-to-image generation and image\&text-to-video generation. This
+strategy offers two significant advantages. a) It allows us to take full
+advantage of the recent advances in text-to-image models, such as Stable
+Diffusion, Midjourney, and DALLE, to generate photorealistic and highly
+detailed images. b) Leveraging the generated image, the model can allocate less
+focus to fine-grained appearance details, prioritizing the efficient learning
+of motion dynamics. To implement this strategy effectively, we introduce two
+core designs. First, we propose the Appearance Injection Network, enhancing the
+preservation of the appearance of the given image. Second, we introduce the
+Appearance Noise Prior, a novel mechanism aimed at maintaining the capabilities
+of pre-trained 2D diffusion models. These design elements empower MicroCinema
+to generate high-quality videos with precise motion, guided by the provided
+text prompts. Extensive experiments demonstrate the superiority of the proposed
+framework. Concretely, MicroCinema achieves SOTA zero-shot FVD of 342.86 on
+UCF-101 and 377.40 on MSR-VTT. See
+https://wangyanhui666.github.io/MicroCinema.github.io/ for video samples.",cs.CV,['cs.CV']
+Domain Prompt Learning with Quaternion Networks,Qinglong Cao · Zhengqin Xu · Yuntian Chen · Chao Ma · Xiaokang Yang, ,https://arxiv.org/abs/2312.08878,,2312.08878.pdf,Domain Prompt Learning with Quaternion Networks,"Prompt learning has emerged as an effective and data-efficient technique in
+large Vision-Language Models (VLMs). However, when adapting VLMs to specialized
+domains such as remote sensing and medical imaging, domain prompt learning
+remains underexplored. While large-scale domain-specific foundation models can
+help tackle this challenge, their concentration on a single vision level makes
+it challenging to prompt both vision and language modalities. To overcome this,
+we propose to leverage domain-specific knowledge from domain-specific
+foundation models to transfer the robust recognition ability of VLMs from
+generalized to specialized domains, using quaternion networks. Specifically,
+the proposed method involves using domain-specific vision features from
+domain-specific foundation models to guide the transformation of generalized
+contextual embeddings from the language branch into a specialized space within
+the quaternion networks. Moreover, we present a hierarchical approach that
+generates vision prompt features by analyzing intermodal relationships between
+hierarchical language prompt features and domain-specific vision features. In
+this way, quaternion networks can effectively mine the intermodal relationships
+in the specific domain, facilitating domain-specific vision-language
+contrastive learning. Extensive experiments on domain-specific datasets show
+that our proposed method achieves new state-of-the-art results in prompt
+learning.",cs.CV,"['cs.CV', 'cs.LG', 'stat.AP']"
+MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints,Pengfei Xie · Wenqiang Xu · Tutian Tang · Zhenjun Yu · Cewu Lu, ,https://arxiv.org/abs/2404.10227,,2404.10227.pdf,MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints,"This work proposes a novel learning framework for visual hand dynamics
+analysis that takes into account the physiological aspects of hand motion. The
+existing models, which are simplified joint-actuated systems, often produce
+unnatural motions. To address this, we integrate a musculoskeletal system with
+a learnable parametric hand model, MANO, to create a new model, MS-MANO. This
+model emulates the dynamics of muscles and tendons to drive the skeletal
+system, imposing physiologically realistic constraints on the resulting torque
+trajectories. We further propose a simulation-in-the-loop pose refinement
+framework, BioPR, that refines the initial estimated pose through a multi-layer
+perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the
+efficacy of the BioPR is conducted in two separate parts. The accuracy of
+MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked
+against two large-scale public datasets and two recent state-of-the-art
+methods. The results demonstrate that our approach consistently improves the
+baseline methods both quantitatively and qualitatively.",cs.CV,"['cs.CV', 'cs.RO']"
+JDEC: JPEG Decoding via Enhanced Continuous Cosine Coefficients,Woo Kyoung Han · Sunghoon Im · Jaedeok Kim · Kyong Hwan Jin,https://wookyounghan.github.io/JDEC/,https://arxiv.org/abs/2404.05558,,2404.05558.pdf,JDEC: JPEG Decoding via Enhanced Continuous Cosine Coefficients,"We propose a practical approach to JPEG image decoding, utilizing a local
+implicit neural representation with continuous cosine formulation. The JPEG
+algorithm significantly quantizes discrete cosine transform (DCT) spectra to
+achieve a high compression rate, inevitably resulting in quality degradation
+while encoding an image. We have designed a continuous cosine spectrum
+estimator to address the quality degradation issue that restores the distorted
+spectrum. By leveraging local DCT formulations, our network has the privilege
+to exploit dequantization and upsampling simultaneously. Our proposed model
+enables decoding compressed images directly across different quality factors
+using a single pre-trained model without relying on a conventional JPEG
+decoder. As a result, our proposed network achieves state-of-the-art
+performance in flexible color image JPEG artifact removal tasks. Our source
+code is available at https://github.com/WooKyoungHan/JDEC.",eess.IV,"['eess.IV', 'cs.CV']"
+Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation,Sihan liu · Yiwei Ma · Xiaoqing Zhang · Haowei Wang · Jiayi Ji · Xiaoshuai Sun · Rongrong Ji, ,https://arxiv.org/abs/2312.12470,,2312.12470.pdf,Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation,"Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that
+combines computer vision and natural language processing, delineating specific
+regions in aerial images as described by textual queries. Traditional Referring
+Image Segmentation (RIS) approaches have been impeded by the complex spatial
+scales and orientations found in aerial imagery, leading to suboptimal
+segmentation results. To address these challenges, we introduce the Rotated
+Multi-Scale Interaction Network (RMSIN), an innovative approach designed for
+the unique demands of RRSIS. RMSIN incorporates an Intra-scale Interaction
+Module (IIM) to effectively address the fine-grained detail required at
+multiple scales and a Cross-scale Interaction Module (CIM) for integrating
+these details coherently across the network. Furthermore, RMSIN employs an
+Adaptive Rotated Convolution (ARC) to account for the diverse orientations of
+objects, a novel contribution that significantly enhances segmentation
+accuracy. To assess the efficacy of RMSIN, we have curated an expansive dataset
+comprising 17,402 image-caption-mask triplets, which is unparalleled in terms
+of scale and variety. This dataset not only presents the model with a wide
+range of spatial and rotational scenarios but also establishes a stringent
+benchmark for the RRSIS task, ensuring a rigorous evaluation of performance.
+Our experimental evaluations demonstrate the exceptional performance of RMSIN,
+surpassing existing state-of-the-art models by a significant margin. All
+datasets and code are made available at https://github.com/Lsan2401/RMSIN.",cs.CV,['cs.CV']
+"IDGuard: Robust, General, Identity-centric POI Proactive Defense Against Face Editing Abuse",Yunshu Dai · Jianwei Fei · Fangjun Huang, ,https://arxiv.org/abs/2311.01357,,2311.01357.pdf,Robust Identity Perceptual Watermark Against Deepfake Face Swapping,"Notwithstanding offering convenience and entertainment to society, Deepfake
+face swapping has caused critical privacy issues with the rapid development of
+deep generative models. Due to imperceptible artifacts in high-quality
+synthetic images, passive detection models against face swapping in recent
+years usually suffer performance damping regarding the generalizability issue.
+Therefore, several studies have been attempted to proactively protect the
+original images against malicious manipulations by inserting invisible signals
+in advance. However, the existing proactive defense approaches demonstrate
+unsatisfactory results with respect to visual quality, detection accuracy, and
+source tracing ability. In this study, to fulfill the research gap, we propose
+the first robust identity perceptual watermarking framework that concurrently
+performs detection and source tracing against Deepfake face swapping
+proactively. We assign identity semantics regarding the image contents to the
+watermarks and devise an unpredictable and nonreversible chaotic encryption
+system to ensure watermark confidentiality. The watermarks are encoded and
+recovered by jointly training an encoder-decoder framework along with
+adversarial image manipulations. Falsification and source tracing are
+accomplished by justifying the consistency between the content-matched identity
+perceptual watermark and the recovered robust watermark from the image.
+Extensive experiments demonstrate state-of-the-art detection performance on
+Deepfake face swapping under both cross-dataset and cross-manipulation
+settings.",cs.CV,['cs.CV']
+Tri-Perspective View Decomposition for Geometry-Aware Depth Completion,Zhiqiang Yan · Yuankai Lin · Kun Wang · Yupeng Zheng · Yufei Wang · Zhenyu Zhang · Jun Li · Jian Yang, ,https://arxiv.org/abs/2403.15008,,2403.15008.pdf,Tri-Perspective View Decomposition for Geometry-Aware Depth Completion,"Depth completion is a vital task for autonomous driving, as it involves
+reconstructing the precise 3D geometry of a scene from sparse and noisy depth
+measurements. However, most existing methods either rely only on 2D depth
+representations or directly incorporate raw 3D point clouds for compensation,
+which are still insufficient to capture the fine-grained 3D geometry of the
+scene. To address this challenge, we introduce Tri-Perspective view
+Decomposition (TPVD), a novel framework that can explicitly model 3D geometry.
+In particular, (1) TPVD ingeniously decomposes the original point cloud into
+three 2D views, one of which corresponds to the sparse depth input. (2) We
+design TPV Fusion to update the 2D TPV features through recurrent 2D-3D-2D
+aggregation, where a Distance-Aware Spherical Convolution (DASC) is applied.
+(3) By adaptively choosing TPV affinitive neighbors, the newly proposed
+Geometric Spatial Propagation Network (GSPN) further improves the geometric
+consistency. As a result, our TPVD outperforms existing methods on KITTI,
+NYUv2, and SUN RGBD. Furthermore, we build a novel depth completion dataset
+named TOFDC, which is acquired by the time-of-flight (TOF) sensor and the color
+camera on smartphones. Project page:
+https://yanzq95.github.io/projectpage/TOFDC/index.html",cs.CV,['cs.CV']
+Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing,Bingyan Liu · Chengyu Wang · Tingfeng Cao · Kui Jia · Jun Huang, ,https://arxiv.org/abs/2403.03431,,2403.03431.pdf,Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing,"Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have
+recently gained significant popularity for creative Text-to-image generation.
+Yet, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE)
+is of greater importance for application developers, which modify objects or
+object properties in images by manipulating feature components in attention
+layers during the generation process. However, little is known about what
+semantic meanings these attention layers have learned and which parts of the
+attention maps contribute to the success of image editing. In this paper, we
+conduct an in-depth probing analysis and demonstrate that cross-attention maps
+in Stable Diffusion often contain object attribution information that can
+result in editing failures. In contrast, self-attention maps play a crucial
+role in preserving the geometric and shape details of the source image during
+the transformation to the target image. Our analysis offers valuable insights
+into understanding cross and self-attention maps in diffusion models. Moreover,
+based on our findings, we simplify popular image editing methods and propose a
+more straightforward yet more stable and efficient tuning-free procedure that
+only modifies self-attention maps of the specified attention layers during the
+denoising process. Experimental results show that our simplified method
+consistently surpasses the performance of popular approaches on multiple
+datasets.",cs.CV,['cs.CV']
+RoHM: Robust Human Motion Reconstruction via Diffusion,Siwei Zhang · Bharat Lal Bhatnagar · Yuanlu Xu · Alexander Winkler · Petr Kadlecek · Siyu Tang · Federica Bogo,https://sanweiliti.github.io/ROHM/ROHM.html,https://arxiv.org/abs/2401.08570,,2401.08570.pdf,RoHM: Robust Human Motion Reconstruction via Diffusion,"We propose RoHM, an approach for robust 3D human motion reconstruction from
+monocular RGB(-D) videos in the presence of noise and occlusions. Most previous
+approaches either train neural networks to directly regress motion in 3D or
+learn data-driven motion priors and combine them with optimization at test
+time. The former do not recover globally coherent motion and fail under
+occlusions; the latter are time-consuming, prone to local minima, and require
+manual tuning. To overcome these shortcomings, we exploit the iterative,
+denoising nature of diffusion models. RoHM is a novel diffusion-based motion
+model that, conditioned on noisy and occluded input data, reconstructs
+complete, plausible motions in consistent global coordinates. Given the
+complexity of the problem -- requiring one to address different tasks
+(denoising and infilling) in different solution spaces (local and global
+motion) -- we decompose it into two sub-tasks and learn two models, one for
+global trajectory and one for local motion. To capture the correlations between
+the two, we then introduce a novel conditioning module, combining it with an
+iterative inference scheme. We apply RoHM to a variety of tasks -- from motion
+reconstruction and denoising to spatial and temporal infilling. Extensive
+experiments on three popular datasets show that our method outperforms
+state-of-the-art approaches qualitatively and quantitatively, while being
+faster at test time. The code is available at
+https://sanweiliti.github.io/ROHM/ROHM.html.",cs.CV,['cs.CV']
+Abductive Ego-View Accident Video Understanding for Safe Driving Perception,Jianwu Fang · Lei-lei Li · Junfei Zhou · Junbin Xiao · Hongkai Yu · Chen Lv · Jianru Xue · Tat-seng Chua,www.lotvsmmau.net,https://arxiv.org/abs/2403.00436,,2403.00436.pdf,Abductive Ego-View Accident Video Understanding for Safe Driving Perception,"We present MM-AU, a novel dataset for Multi-Modal Accident video
+Understanding. MM-AU contains 11,727 in-the-wild ego-view accident videos, each
+with temporally aligned text descriptions. We annotate over 2.23 million object
+boxes and 58,650 pairs of video-based accident reasons, covering 58 accident
+categories. MM-AU supports various accident understanding tasks, particularly
+multimodal video diffusion to understand accident cause-effect chains for safe
+driving. With MM-AU, we present an Abductive accident Video understanding
+framework for Safe Driving perception (AdVersa-SD). AdVersa-SD performs video
+diffusion via an Object-Centric Video Diffusion (OAVD) method which is driven
+by an abductive CLIP model. This model involves a contrastive interaction loss
+to learn the pair co-occurrence of normal, near-accident, accident frames with
+the corresponding text descriptions, such as accident reasons, prevention
+advice, and accident categories. OAVD enforces the causal region learning while
+fixing the content of the original frame background in video generation, to
+find the dominant cause-effect chain for certain accidents. Extensive
+experiments verify the abductive ability of AdVersa-SD and the superiority of
+OAVD against the state-of-the-art diffusion models. Additionally, we provide
+careful benchmark evaluations for object detection and accident reason
+answering since AdVersa-SD relies on precise object and accident reason
+information.",cs.CV,"['cs.CV', 'cs.AI']"
+Towards Language-Driven Video Inpainting via Multimodal Large Language Models,Jianzong Wu · Xiangtai Li · Chenyang Si · Shangchen Zhou · Jingkang Yang · Jiangning Zhang · Yining Li · Kai Chen · Yunhai Tong · Ziwei Liu · Chen Change Loy,https://jianzongwu.github.io/projects/rovi/,https://arxiv.org/abs/2401.10226,,2401.10226.pdf,Towards Language-Driven Video Inpainting via Multimodal Large Language Models,"We introduce a new task -- language-driven video inpainting, which uses
+natural language instructions to guide the inpainting process. This approach
+overcomes the limitations of traditional video inpainting methods that depend
+on manually labeled binary masks, a process often tedious and labor-intensive.
+We present the Remove Objects from Videos by Instructions (ROVI) dataset,
+containing 5,650 videos and 9,091 inpainting results, to support training and
+evaluation for this task. We also propose a novel diffusion-based
+language-driven video inpainting framework, the first end-to-end baseline for
+this task, integrating Multimodal Large Language Models to understand and
+execute complex language-based inpainting requests effectively. Our
+comprehensive results showcase the dataset's versatility and the model's
+effectiveness in various language-instructed inpainting scenarios. We will make
+datasets, code, and models publicly available.",cs.CV,['cs.CV']
+Self-Supervised Facial Representation Learning with Facial Region Awareness,Zheng Gao · Ioannis Patras, ,https://arxiv.org/abs/2403.02138,,2403.02138.pdf,Self-Supervised Facial Representation Learning with Facial Region Awareness,"Self-supervised pre-training has been proved to be effective in learning
+transferable representations that benefit various visual tasks. This paper asks
+this question: can self-supervised pre-training learn general facial
+representations for various facial analysis tasks? Recent efforts toward this
+goal are limited to treating each face image as a whole, i.e., learning
+consistent facial representations at the image-level, which overlooks the
+consistency of local facial representations (i.e., facial regions like eyes,
+nose, etc). In this work, we make a first attempt to propose a novel
+self-supervised facial representation learning framework to learn consistent
+global and local facial representations, Facial Region Awareness (FRA).
+Specifically, we explicitly enforce the consistency of facial regions by
+matching the local facial representations across views, which are extracted
+with learned heatmaps highlighting the facial regions. Inspired by the mask
+prediction in supervised semantic segmentation, we obtain the heatmaps via
+cosine similarity between the per-pixel projection of feature maps and facial
+mask embeddings computed from learnable positional embeddings, which leverage
+the attention mechanism to globally look up the facial image for facial
+regions. To learn such heatmaps, we formulate the learning of facial mask
+embeddings as a deep clustering problem by assigning the pixel features from
+the feature maps to them. The transfer learning results on facial
+classification and regression tasks show that our FRA outperforms previous
+pre-trained models and more importantly, using ResNet as the unified backbone
+for various tasks, our FRA achieves comparable or even better performance
+compared with SOTA methods in facial analysis tasks.",cs.CV,['cs.CV']
+Visual Anagrams: Synthesizing Multi-View Optical Illusions with Diffusion Models,Daniel Geng · Inbum Park · Andrew Owens, ,https://arxiv.org/abs/2311.17919,,2311.17919.pdf,Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models,"We address the problem of synthesizing multi-view optical illusions: images
+that change appearance upon a transformation, such as a flip or rotation. We
+propose a simple, zero-shot method for obtaining these illusions from
+off-the-shelf text-to-image diffusion models. During the reverse diffusion
+process, we estimate the noise from different views of a noisy image, and then
+combine these noise estimates together and denoise the image. A theoretical
+analysis suggests that this method works precisely for views that can be
+written as orthogonal transformations, of which permutations are a subset. This
+leads to the idea of a visual anagram--an image that changes appearance under
+some rearrangement of pixels. This includes rotations and flips, but also more
+exotic pixel permutations such as a jigsaw rearrangement. Our approach also
+naturally extends to illusions with more than two views. We provide both
+qualitative and quantitative results demonstrating the effectiveness and
+flexibility of our method. Please see our project webpage for additional
+visualizations and results: https://dangeng.github.io/visual_anagrams/",cs.CV,['cs.CV']
+AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving,Mingfu Liang · Jong-Chyi Su · Samuel Schulter · Sparsh Garg · Shiyu Zhao · Ying Wu · Manmohan Chandraker, ,https://arxiv.org/abs/2403.17373,,2403.17373.pdf,AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving,"Autonomous vehicle (AV) systems rely on robust perception models as a
+cornerstone of safety assurance. However, objects encountered on the road
+exhibit a long-tailed distribution, with rare or unseen categories posing
+challenges to a deployed perception model. This necessitates an expensive
+process of continuously curating and annotating data with significant human
+effort. We propose to leverage recent advances in vision-language and large
+language models to design an Automatic Data Engine (AIDE) that automatically
+identifies issues, efficiently curates data, improves the model through
+auto-labeling, and verifies the model through generation of diverse scenarios.
+This process operates iteratively, allowing for continuous self-improvement of
+the model. We further establish a benchmark for open-world detection on AV
+datasets to comprehensively evaluate various learning paradigms, demonstrating
+our method's superior performance at a reduced cost.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Hierarchical Intra-modal Correlation Learning for Label-free 3D Semantic Segmentation,Xin Kang · Lei Chu · Jiahao Li · Xuejin Chen · Yan Lu, ,https://arxiv.org/abs/2309.10649,,2309.10649.pdf,Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation,"Current state-of-the-art point cloud-based perception methods usually rely on
+large-scale labeled data, which requires expensive manual annotations. A
+natural option is to explore the unsupervised methodology for 3D perception
+tasks. However, such methods often face substantial performance-drop
+difficulties. Fortunately, we found that there exist amounts of image-based
+datasets and an alternative can be proposed, i.e., transferring the knowledge
+in the 2D images to 3D point clouds. Specifically, we propose a novel approach
+for the challenging cross-modal and cross-domain adaptation task by fully
+exploring the relationship between images and point clouds and designing
+effective feature alignment strategies. Without any 3D labels, our method
+achieves state-of-the-art performance for 3D point cloud semantic segmentation
+on SemanticKITTI by using the knowledge of KITTI360 and GTA5, compared to
+existing unsupervised and weakly-supervised baselines.",cs.CV,['cs.CV']
+Rethinking Multi-domain Generalization with A General Learning Objective,Zhaorui Tan · Xi Yang · Kaizhu Huang, ,https://arxiv.org/abs/2402.18853,,2402.18853.pdf,Rethinking Multi-domain Generalization with A General Learning Objective,"Multi-domain generalization (mDG) is universally aimed to minimize the
+discrepancy between training and testing distributions to enhance
+marginal-to-label distribution mapping. However, existing mDG literature lacks
+a general learning objective paradigm and often imposes constraints on static
+target marginal distributions. In this paper, we propose to leverage a
+$Y$-mapping to relax the constraint. We rethink the learning objective for mDG
+and design a new \textbf{general learning objective} to interpret and analyze
+most existing mDG wisdom. This general objective is bifurcated into two
+synergistic amis: learning domain-independent conditional features and
+maximizing a posterior. Explorations also extend to two effective
+regularization terms that incorporate prior information and suppress invalid
+causality, alleviating the issues that come with relaxed constraints. We
+theoretically contribute an upper bound for the domain alignment of
+domain-independent conditional features, disclosing that many previous mDG
+endeavors actually \textbf{optimize partially the objective} and thus lead to
+limited performance. As such, our study distills a general learning objective
+into four practical components, providing a general, robust, and flexible
+mechanism to handle complex domain shifts. Extensive empirical results indicate
+that the proposed objective with $Y$-mapping leads to substantially better mDG
+performance in various downstream tasks, including regression, segmentation,
+and classification.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+VideoMAC: Video Masked Autoencoders Meet ConvNets,Gensheng Pei · Tao Chen · Xiruo Jiang · 刘华峰 Liu · Zeren Sun · Yazhou Yao, ,https://arxiv.org/abs/2402.19082,,2402.19082.pdf,VideoMAC: Video Masked Autoencoders Meet ConvNets,"Recently, the advancement of self-supervised learning techniques, like masked
+autoencoders (MAE), has greatly influenced visual representation learning for
+images and videos. Nevertheless, it is worth noting that the predominant
+approaches in existing masked image / video modeling rely excessively on
+resource-intensive vision transformers (ViTs) as the feature encoder. In this
+paper, we propose a new approach termed as \textbf{VideoMAC}, which combines
+video masked autoencoders with resource-friendly ConvNets. Specifically,
+VideoMAC employs symmetric masking on randomly sampled pairs of video frames.
+To prevent the issue of mask pattern dissipation, we utilize ConvNets which are
+implemented with sparse convolutional operators as encoders. Simultaneously, we
+present a simple yet effective masked video modeling (MVM) approach, a dual
+encoder architecture comprising an online encoder and an exponential moving
+average target encoder, aimed to facilitate inter-frame reconstruction
+consistency in videos. Additionally, we demonstrate that VideoMAC, empowering
+classical (ResNet) / modern (ConvNeXt) convolutional encoders to harness the
+benefits of MVM, outperforms ViT-based approaches on downstream tasks,
+including video object segmentation (+\textbf{5.2\%} / \textbf{6.4\%}
+$\mathcal{J}\&\mathcal{F}$), body part propagation (+\textbf{6.3\%} /
+\textbf{3.1\%} mIoU), and human pose tracking (+\textbf{10.2\%} /
+\textbf{11.1\%} PCK@0.1).",cs.CV,['cs.CV']
+EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection,Xuanyu Zhang · Runyi Li · Jiwen Yu · Youmin Xu · Weiqi Li · Jian Zhang,https://xuanyuzhang21.github.io/project/editguard/,https://arxiv.org/abs/2312.08883,,2312.08883.pdf,EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection,"In the era where AI-generated content (AIGC) models can produce stunning and
+lifelike images, the lingering shadow of unauthorized reproductions and
+malicious tampering poses imminent threats to copyright integrity and
+information security. Current image watermarking methods, while widely accepted
+for safeguarding visual content, can only protect copyright and ensure
+traceability. They fall short in localizing increasingly realistic image
+tampering, potentially leading to trust crises, privacy violations, and legal
+disputes. To solve this challenge, we propose an innovative proactive forensics
+framework EditGuard, to unify copyright protection and tamper-agnostic
+localization, especially for AIGC-based editing methods. It can offer a
+meticulous embedding of imperceptible watermarks and precise decoding of
+tampered areas and copyright information. Leveraging our observed fragility and
+locality of image-into-image steganography, the realization of EditGuard can be
+converted into a united image-bit steganography issue, thus completely
+decoupling the training process from the tampering types. Extensive experiments
+demonstrate that our EditGuard balances the tamper localization accuracy,
+copyright recovery precision, and generalizability to various AIGC-based
+tampering methods, especially for image forgery that is difficult for the naked
+eye to detect. The project page is available at
+https://xuanyuzhang21.github.io/project/editguard/.",cs.CV,['cs.CV']
+Re-thinking Data Availability Attacks Against Deep Neural Networks,Bin Fang · Bo Li · Shuang Wu · Shouhong Ding · Ran Yi · Lizhuang Ma, ,https://arxiv.org/abs/2401.09740,,2401.09740.pdf,Hijacking Attacks against Neural Networks by Analyzing Training Data,"Backdoors and adversarial examples are the two primary threats currently
+faced by deep neural networks (DNNs). Both attacks attempt to hijack the model
+behaviors with unintended outputs by introducing (small) perturbations to the
+inputs. Backdoor attacks, despite the high success rates, often require a
+strong assumption, which is not always easy to achieve in reality. Adversarial
+example attacks, which put relatively weaker assumptions on attackers, often
+demand high computational resources, yet do not always yield satisfactory
+success rates when attacking mainstream black-box models in the real world.
+These limitations motivate the following research question: can model hijacking
+be achieved more simply, with a higher attack success rate and more reasonable
+assumptions? In this paper, we propose CleanSheet, a new model hijacking attack
+that obtains the high performance of backdoor attacks without requiring the
+adversary to tamper with the model training process. CleanSheet exploits
+vulnerabilities in DNNs stemming from the training data. Specifically, our key
+idea is to treat part of the clean training data of the target model as
+""poisoned data,"" and capture the characteristics of these data that are more
+sensitive to the model (typically called robust features) to construct
+""triggers."" These triggers can be added to any input example to mislead the
+target model, similar to backdoor attacks. We validate the effectiveness of
+CleanSheet through extensive experiments on 5 datasets, 79 normally trained
+models, 68 pruned models, and 39 defensive models. Results show that CleanSheet
+exhibits performance comparable to state-of-the-art backdoor attacks, achieving
+an average attack success rate (ASR) of 97.5% on CIFAR-100 and 92.4% on GTSRB,
+respectively. Furthermore, CleanSheet consistently maintains a high ASR, when
+confronted with various mainstream backdoor defenses.",cs.CR,['cs.CR']
+Multi-View Attentive Contextualization for Multi-View 3D Object Detection,Xianpeng Liu · Ce Zheng · Ming Qian · Nan Xue · Chen Chen · Zhebin Zhang · Chen Li · Tianfu Wu,https://xianpeng919.github.io/mvacon/,https://arxiv.org/abs/2405.12200,,2405.12200.pdf,Multi-View Attentive Contextualization for Multi-View 3D Object Detection,"We present Multi-View Attentive Contextualization (MvACon), a simple yet
+effective method for improving 2D-to-3D feature lifting in query-based
+multi-view 3D (MV3D) object detection. Despite remarkable progress witnessed in
+the field of query-based MV3D object detection, prior art often suffers from
+either the lack of exploiting high-resolution 2D features in dense
+attention-based lifting, due to high computational costs, or from
+insufficiently dense grounding of 3D queries to multi-scale 2D features in
+sparse attention-based lifting. Our proposed MvACon hits the two birds with one
+stone using a representationally dense yet computationally sparse attentive
+feature contextualization scheme that is agnostic to specific 2D-to-3D feature
+lifting approaches. In experiments, the proposed MvACon is thoroughly tested on
+the nuScenes benchmark, using both the BEVFormer and its recent 3D deformable
+attention (DFA3D) variant, as well as the PETR, showing consistent detection
+performance improvement, especially in enhancing performance in location,
+orientation, and velocity prediction. It is also tested on the Waymo-mini
+benchmark using BEVFormer with similar improvement. We qualitatively and
+quantitatively show that global cluster-based contexts effectively encode dense
+scene-level contexts for MV3D object detection. The promising results of our
+proposed MvACon reinforces the adage in computer vision -- ``(contextualized)
+feature matters"".",cs.CV,['cs.CV']
+RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection,Ximiao Zhang · Min Xu · Xiuzhuang Zhou,https://github.com/cnulab/RealNet,https://arxiv.org/abs/2403.05897,,2403.05897.pdf,RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection,"Self-supervised feature reconstruction methods have shown promising advances
+in industrial image anomaly detection and localization. Despite this progress,
+these methods still face challenges in synthesizing realistic and diverse
+anomaly samples, as well as addressing the feature redundancy and pre-training
+bias of pre-trained feature. In this work, we introduce RealNet, a feature
+reconstruction network with realistic synthetic anomaly and adaptive feature
+selection. It is incorporated with three key innovations: First, we propose
+Strength-controllable Diffusion Anomaly Synthesis (SDAS), a diffusion
+process-based synthesis strategy capable of generating samples with varying
+anomaly strengths that mimic the distribution of real anomalous samples.
+Second, we develop Anomaly-aware Features Selection (AFS), a method for
+selecting representative and discriminative pre-trained feature subsets to
+improve anomaly detection performance while controlling computational costs.
+Third, we introduce Reconstruction Residuals Selection (RRS), a strategy that
+adaptively selects discriminative residuals for comprehensive identification of
+anomalous regions across multiple levels of granularity. We assess RealNet on
+four benchmark datasets, and our results demonstrate significant improvements
+in both Image AUROC and Pixel AUROC compared to the current state-o-the-art
+methods. The code, data, and models are available at
+https://github.com/cnulab/RealNet.",cs.CV,['cs.CV']
+CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation,Jun Wang · Yuzhe Qin · Kaiming Kuang · Yigit Korkmaz · Akhilan Gurumoorthy · Hao Su · Xiaolong Wang, ,https://arxiv.org/abs/2402.14795,,2402.14795.pdf,CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation,"We introduce CyberDemo, a novel approach to robotic imitation learning that
+leverages simulated human demonstrations for real-world tasks. By incorporating
+extensive data augmentation in a simulated environment, CyberDemo outperforms
+traditional in-domain real-world demonstrations when transferred to the real
+world, handling diverse physical and visual conditions. Regardless of its
+affordability and convenience in data collection, CyberDemo outperforms
+baseline methods in terms of success rates across various tasks and exhibits
+generalizability with previously unseen objects. For example, it can rotate
+novel tetra-valve and penta-valve, despite human demonstrations only involving
+tri-valves. Our research demonstrates the significant potential of simulated
+human demonstrations for real-world dexterous manipulation tasks. More details
+can be found at https://cyber-demo.github.io",cs.RO,"['cs.RO', 'cs.CV']"
+InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion,Jihyun Lee · Shunsuke Saito · Giljoo Nam · Minhyuk Sung · Tae-Kyun Kim,https://jyunlee.github.io/projects/interhandgen/,https://arxiv.org/abs/2403.17422,,2403.17422.pdf,InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion,"We present InterHandGen, a novel framework that learns the generative prior
+of two-hand interaction. Sampling from our model yields plausible and diverse
+two-hand shapes in close interaction with or without an object. Our prior can
+be incorporated into any optimization or learning methods to reduce ambiguity
+in an ill-posed setup. Our key observation is that directly modeling the joint
+distribution of multiple instances imposes high learning complexity due to its
+combinatorial nature. Thus, we propose to decompose the modeling of joint
+distribution into the modeling of factored unconditional and conditional single
+instance distribution. In particular, we introduce a diffusion model that
+learns the single-hand distribution unconditional and conditional to another
+hand via conditioning dropout. For sampling, we combine anti-penetration and
+classifier-free guidance to enable plausible generation. Furthermore, we
+establish the rigorous evaluation protocol of two-hand synthesis, where our
+method significantly outperforms baseline generative models in terms of
+plausibility and diversity. We also demonstrate that our diffusion prior can
+boost the performance of two-hand reconstruction from monocular in-the-wild
+images, achieving new state-of-the-art accuracy.",cs.CV,['cs.CV']
+DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model,Lirui Zhao · Yue Yang · Kaipeng Zhang · Wenqi Shao · Yuxin Zhang · Yu Qiao · Ping Luo · Rongrong Ji, ,https://arxiv.org/abs/2404.01342,,2404.01342.pdf,DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model,"Text-to-image (T2I) generative models have attracted significant attention
+and found extensive applications within and beyond academic research. For
+example, the Civitai community, a platform for T2I innovation, currently hosts
+an impressive array of 74,492 distinct models. However, this diversity presents
+a formidable challenge in selecting the most appropriate model and parameters,
+a process that typically requires numerous trials. Drawing inspiration from the
+tool usage research of large language models (LLMs), we introduce DiffAgent, an
+LLM agent designed to screen the accurate selection in seconds via API calls.
+DiffAgent leverages a novel two-stage training framework, SFTA, enabling it to
+accurately align T2I API responses with user input in accordance with human
+preferences. To train and evaluate DiffAgent's capabilities, we present
+DABench, a comprehensive dataset encompassing an extensive range of T2I APIs
+from the community. Our evaluations reveal that DiffAgent not only excels in
+identifying the appropriate T2I API but also underscores the effectiveness of
+the SFTA training framework. Codes are available at
+https://github.com/OpenGVLab/DiffAgent.",cs.CL,"['cs.CL', 'cs.AI']"
+Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation,Mukul Khanna · Yongsen Mao · Hanxiao Jiang · Sanjay Haresh · Brennan Shacklett · Dhruv Batra · Alexander William Clegg · Eric Undersander · Angel Xuan Chang · Manolis Savva, ,https://arxiv.org/abs/2306.11290,,2306.11290.pdf,Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation,"We contribute the Habitat Synthetic Scene Dataset, a dataset of 211
+high-quality 3D scenes, and use it to test navigation agent generalization to
+realistic 3D environments. Our dataset represents real interiors and contains a
+diverse set of 18,656 models of real-world objects. We investigate the impact
+of synthetic 3D scene dataset scale and realism on the task of training
+embodied agents to find and navigate to objects (ObjectGoal navigation). By
+comparing to synthetic 3D scene datasets from prior work, we find that scale
+helps in generalization, but the benefits quickly saturate, making visual
+fidelity and correlation to real-world scenes more important. Our experiments
+show that agents trained on our smaller-scale dataset can match or outperform
+agents trained on much larger datasets. Surprisingly, we observe that agents
+trained on just 122 scenes from our dataset outperform agents trained on 10,000
+scenes from the ProcTHOR-10K dataset in terms of zero-shot generalization in
+real-world scanned environments.",cs.CV,['cs.CV']
+Objects as volumes: A stochastic geometry view of opaque solids,Bailey Miller · Hanyu Chen · Alice Lai · Ioannis Gkioulekas,https://imaging.cs.cmu.edu/volumetric_opaque_solids/,https://arxiv.org/abs/2312.15406,,2312.15406.pdf,Objects as volumes: A stochastic geometry view of opaque solids,"We develop a theory for the representation of opaque solids as volumes.
+Starting from a stochastic representation of opaque solids as random indicator
+functions, we prove the conditions under which such solids can be modeled using
+exponential volumetric transport. We also derive expressions for the volumetric
+attenuation coefficient as a functional of the probability distributions of the
+underlying indicator functions. We generalize our theory to account for
+isotropic and anisotropic scattering at different parts of the solid, and for
+representations of opaque solids as stochastic implicit surfaces. We derive our
+volumetric representation from first principles, which ensures that it
+satisfies physical constraints such as reciprocity and reversibility. We use
+our theory to explain, compare, and correct previous volumetric
+representations, as well as propose meaningful extensions that lead to improved
+performance in 3D reconstruction tasks.",cs.CV,"['cs.CV', 'cs.GR']"
+MeaCap: Memory-Augmented Zero-shot Image Captioning,Zequn Zeng · Yan Xie · Hao Zhang · Chiyu Chen · Zhengjue Wang · Bo Chen,https://github.com/joeyz0z/MeaCap,https://arxiv.org/abs/2403.03715,,2403.03715.pdf,MeaCap: Memory-Augmented Zero-shot Image Captioning,"Zero-shot image captioning (IC) without well-paired image-text data can be
+divided into two categories, training-free and text-only-training. Generally,
+these two types of methods realize zero-shot IC by integrating pretrained
+vision-language models like CLIP for image-text similarity evaluation and a
+pre-trained language model (LM) for caption generation. The main difference
+between them is whether using a textual corpus to train the LM. Though
+achieving attractive performance w.r.t. some metrics, existing methods often
+exhibit some common drawbacks. Training-free methods tend to produce
+hallucinations, while text-only-training often lose generalization capability.
+To move forward, in this paper, we propose a novel Memory-Augmented zero-shot
+image Captioning framework (MeaCap). Specifically, equipped with a textual
+memory, we introduce a retrieve-then-filter module to get key concepts that are
+highly related to the image. By deploying our proposed memory-augmented
+visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate
+concept-centered captions that keep high consistency with the image with fewer
+hallucinations and more world-knowledge. The framework of MeaCap achieves the
+state-of-the-art performance on a series of zero-shot IC settings. Our code is
+available at https://github.com/joeyz0z/MeaCap.",cs.CV,['cs.CV']
+Weakly Supervised Monocular 3D Detection with a Single-View Image,Xueying Jiang · Sheng Jin · Lewei Lu · Xiaoqin Zhang · Shijian Lu, ,https://arxiv.org/abs/2402.19144,,2402.19144.pdf,Weakly Supervised Monocular 3D Detection with a Single-View Image,"Monocular 3D detection (M3D) aims for precise 3D object localization from a
+single-view image which usually involves labor-intensive annotation of 3D
+detection boxes. Weakly supervised M3D has recently been studied to obviate the
+3D annotation process by leveraging many existing 2D annotations, but it often
+requires extra training data such as LiDAR point clouds or multi-view images
+which greatly degrades its applicability and usability in various applications.
+We propose SKD-WM3D, a weakly supervised monocular 3D detection framework that
+exploits depth information to achieve M3D with a single-view image exclusively
+without any 3D annotations or other training data. One key design in SKD-WM3D
+is a self-knowledge distillation framework, which transforms image features
+into 3D-like representations by fusing depth information and effectively
+mitigates the inherent depth ambiguity in monocular scenarios with little
+computational overhead in inference. In addition, we design an
+uncertainty-aware distillation loss and a gradient-targeted transfer modulation
+strategy which facilitate knowledge acquisition and knowledge transfer,
+respectively. Extensive experiments show that SKD-WM3D surpasses the
+state-of-the-art clearly and is even on par with many fully supervised methods.",cs.CV,['cs.CV']
+SemCity: Semantic Scene Generation with Triplane Diffusion,Jumin Lee · Sebin Lee · Changho Jo · Woobin Im · Ju-hyeong Seon · Sung-Eui Yoon, ,https://arxiv.org/abs/2403.07773,,2403.07773.pdf,SemCity: Semantic Scene Generation with Triplane Diffusion,"We present ""SemCity,"" a 3D diffusion model for semantic scene generation in
+real-world outdoor environments. Most 3D diffusion models focus on generating a
+single object, synthetic indoor scenes, or synthetic outdoor scenes, while the
+generation of real-world outdoor scenes is rarely addressed. In this paper, we
+concentrate on generating a real-outdoor scene through learning a diffusion
+model on a real-world outdoor dataset. In contrast to synthetic data,
+real-outdoor datasets often contain more empty spaces due to sensor
+limitations, causing challenges in learning real-outdoor distributions. To
+address this issue, we exploit a triplane representation as a proxy form of
+scene distributions to be learned by our diffusion model. Furthermore, we
+propose a triplane manipulation that integrates seamlessly with our triplane
+diffusion model. The manipulation improves our diffusion model's applicability
+in a variety of downstream tasks related to outdoor scene generation such as
+scene inpainting, scene outpainting, and semantic scene completion refinements.
+In experimental results, we demonstrate that our triplane diffusion model shows
+meaningful generation results compared with existing work in a real-outdoor
+dataset, SemanticKITTI. We also show our triplane manipulation facilitates
+seamlessly adding, removing, or modifying objects within a scene. Further, it
+also enables the expansion of scenes toward a city-level scale. Finally, we
+evaluate our method on semantic scene completion refinements where our
+diffusion model enhances predictions of semantic scene completion networks by
+learning scene distribution. Our code is available at
+https://github.com/zoomin-lee/SemCity.",cs.CV,['cs.CV']
+SD2Event: Self-supervised Learning of Dynamic Detectors and Contextual Descriptors for Event Cameras,Yuan Gao · Yuqing Zhu · Xinjun Li · Yimin Du · Tianzhu Zhang, ,https://arxiv.org/abs/2401.01042,,2401.01042.pdf,Relating Events and Frames Based on Self-Supervised Learning and Uncorrelated Conditioning for Unsupervised Domain Adaptation,"Event-based cameras provide accurate and high temporal resolution
+measurements for performing computer vision tasks in challenging scenarios,
+such as high-dynamic range environments and fast-motion maneuvers. Despite
+their advantages, utilizing deep learning for event-based vision encounters a
+significant obstacle due to the scarcity of annotated data caused by the
+relatively recent emergence of event-based cameras. To overcome this
+limitation, leveraging the knowledge available from annotated data obtained
+with conventional frame-based cameras presents an effective solution based on
+unsupervised domain adaptation. We propose a new algorithm tailored for
+adapting a deep neural network trained on annotated frame-based data to
+generalize well on event-based unannotated data. Our approach incorporates
+uncorrelated conditioning and self-supervised learning in an adversarial
+learning scheme to close the gap between the two source and target domains. By
+applying self-supervised learning, the algorithm learns to align the
+representations of event-based data with those from frame-based camera data,
+thereby facilitating knowledge transfer.Furthermore, the inclusion of
+uncorrelated conditioning ensures that the adapted model effectively
+distinguishes between event-based and conventional data, enhancing its ability
+to classify event-based images accurately.Through empirical experimentation and
+evaluation, we demonstrate that our algorithm surpasses existing approaches
+designed for the same purpose using two benchmarks. The superior performance of
+our solution is attributed to its ability to effectively utilize annotated data
+from frame-based cameras and transfer the acquired knowledge to the event-based
+vision domain.",cs.CV,['cs.CV']
+Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation,Ming Xu · Stephen Gould, ,https://arxiv.org/abs/2404.01518,,2404.01518.pdf,Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation,"We propose a novel approach to the action segmentation task for long,
+untrimmed videos, based on solving an optimal transport problem. By encoding a
+temporal consistency prior into a Gromov-Wasserstein problem, we are able to
+decode a temporally consistent segmentation from a noisy affinity/matching cost
+matrix between video frames and action classes. Unlike previous approaches, our
+method does not require knowing the action order for a video to attain temporal
+consistency. Furthermore, our resulting (fused) Gromov-Wasserstein problem can
+be efficiently solved on GPUs using a few iterations of projected mirror
+descent. We demonstrate the effectiveness of our method in an unsupervised
+learning setting, where our method is used to generate pseudo-labels for
+self-training. We evaluate our segmentation approach and unsupervised learning
+pipeline on the Breakfast, 50-Salads, YouTube Instructions and Desktop Assembly
+datasets, yielding state-of-the-art results for the unsupervised video action
+segmentation task.",cs.CV,"['cs.CV', 'cs.LG', 'eess.IV']"
+Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes,YuJie Lu · Long Wan · Nayu Ding · Yulong Wang · Shuhan Shen · Shen Cai · Lin Gao,http://www.cscvlab.com/research/UODFs/index.html,https://arxiv.org/abs/2403.01414,,2403.01414.pdf,Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes,"Neural implicit representation of geometric shapes has witnessed considerable
+advancements in recent years. However, common distance field based implicit
+representations, specifically signed distance field (SDF) for watertight shapes
+or unsigned distance field (UDF) for arbitrary shapes, routinely suffer from
+degradation of reconstruction accuracy when converting to explicit surface
+points and meshes. In this paper, we introduce a novel neural implicit
+representation based on unsigned orthogonal distance fields (UODFs). In UODFs,
+the minimal unsigned distance from any spatial point to the shape surface is
+defined solely in one orthogonal direction, contrasting with the
+multi-directional determination made by SDF and UDF. Consequently, every point
+in the 3D UODFs can directly access its closest surface points along three
+orthogonal directions. This distinctive feature leverages the accurate
+reconstruction of surface points without interpolation errors. We verify the
+effectiveness of UODFs through a range of reconstruction examples, extending
+from simple watertight or non-watertight shapes to complex shapes that include
+hollows, internal or assembling structures.",cs.CV,['cs.CV']
+Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transfomers,Sheng Yang · Jiawang Bai · Kuofeng Gao · Yong Yang · Yiming Li · Shu-Tao Xia,https://github.com/20000yshust/SWARM,https://arxiv.org/abs/2405.10612,,2405.10612.pdf,Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers,"Given the power of vision transformers, a new learning paradigm, pre-training
+and then prompting, makes it more efficient and effective to address downstream
+visual recognition tasks. In this paper, we identify a novel security threat
+towards such a paradigm from the perspective of backdoor attacks. Specifically,
+an extra prompt token, called the switch token in this work, can turn the
+backdoor mode on, i.e., converting a benign model into a backdoored one. Once
+under the backdoor mode, a specific trigger can force the model to predict a
+target class. It poses a severe risk to the users of cloud API, since the
+malicious behavior can not be activated and detected under the benign mode,
+thus making the attack very stealthy. To attack a pre-trained model, our
+proposed attack, named SWARM, learns a trigger and prompt tokens including a
+switch token. They are optimized with the clean loss which encourages the model
+always behaves normally even the trigger presents, and the backdoor loss that
+ensures the backdoor can be activated by the trigger when the switch is on.
+Besides, we utilize the cross-mode feature distillation to reduce the effect of
+the switch token on clean samples. The experiments on diverse visual
+recognition tasks confirm the success of our switchable backdoor attack, i.e.,
+achieving 95%+ attack success rate, and also being hard to be detected and
+removed. Our code is available at https://github.com/20000yshust/SWARM.",cs.CV,"['cs.CV', 'cs.CR', 'cs.LG']"
+Unmixing before Fusion: A Generalized Paradigm for Multi-Source-based Hyperspectral Image Synthesis,Yang Yu · Erting Pan · Xinya Wang · Yuheng Wu · Xiaoguang Mei · Jiayi Ma,https://hsi-synthesis.github.io/,,https://ieeexplore.ieee.org/document/10414148,,,,,nan
+Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling,Liwen Wu · Sai Bi · Zexiang Xu · Fujun Luan · Kai Zhang · Iliyan Georgiev · Kalyan Sunkavalli · Ravi Ramamoorthi,https://lwwu2.github.io/nde/,https://arxiv.org/abs/2405.14847,,2405.14847.pdf,Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling,"Novel-view synthesis of specular objects like shiny metals or glossy paints
+remains a significant challenge. Not only the glossy appearance but also global
+illumination effects, including reflections of other objects in the
+environment, are critical components to faithfully reproduce a scene. In this
+paper, we present Neural Directional Encoding (NDE), a view-dependent
+appearance encoding of neural radiance fields (NeRF) for rendering specular
+objects. NDE transfers the concept of feature-grid-based spatial encoding to
+the angular domain, significantly improving the ability to model high-frequency
+angular signals. In contrast to previous methods that use encoding functions
+with only angular input, we additionally cone-trace spatial features to obtain
+a spatially varying directional encoding, which addresses the challenging
+interreflection effects. Extensive experiments on both synthetic and real
+datasets show that a NeRF model with NDE (1) outperforms the state of the art
+on view synthesis of specular objects, and (2) works with small networks to
+allow fast (real-time) inference. The project webpage and source code are
+available at: \url{https://lwwu2.github.io/nde/}.",cs.CV,['cs.CV']
+ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models,Jeong-gi Kwak · Erqun Dong · Yuhe Jin · Hanseok Ko · Shweta Mahajan · Kwang Moo Yi,https://ubc-vision.github.io/vivid123/,https://arxiv.org/abs/2312.01305,,2312.01305.pdf,ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models,"Generating novel views of an object from a single image is a challenging
+task. It requires an understanding of the underlying 3D structure of the object
+from an image and rendering high-quality, spatially consistent new views. While
+recent methods for view synthesis based on diffusion have shown great progress,
+achieving consistency among various view estimates and at the same time abiding
+by the desired camera pose remains a critical problem yet to be solved. In this
+work, we demonstrate a strikingly simple method, where we utilize a pre-trained
+video diffusion model to solve this problem. Our key idea is that synthesizing
+a novel view could be reformulated as synthesizing a video of a camera going
+around the object of interest -- a scanning video -- which then allows us to
+leverage the powerful priors that a video diffusion model would have learned.
+Thus, to perform novel-view synthesis, we create a smooth camera trajectory to
+the target view that we wish to render, and denoise using both a
+view-conditioned diffusion model and a video diffusion model. By doing so, we
+obtain a highly consistent novel view synthesis, outperforming the state of the
+art.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+Self-Adaptive Reality-Guided Diffusion for Artifact-Free Super-Resolution,Qingping Zheng · Ling Zheng · Yuanfan Guo · Ying Li · Songcen Xu · Jiankang Deng · Hang Xu, ,https://arxiv.org/abs/2403.16643v1,,2403.16643v1.pdf,Self-Adaptive Reality-Guided Diffusion for Artifact-Free Super-Resolution,"Artifact-free super-resolution (SR) aims to translate low-resolution images
+into their high-resolution counterparts with a strict integrity of the original
+content, eliminating any distortions or synthetic details. While traditional
+diffusion-based SR techniques have demonstrated remarkable abilities to enhance
+image detail, they are prone to artifact introduction during iterative
+procedures. Such artifacts, ranging from trivial noise to unauthentic textures,
+deviate from the true structure of the source image, thus challenging the
+integrity of the super-resolution process. In this work, we propose
+Self-Adaptive Reality-Guided Diffusion (SARGD), a training-free method that
+delves into the latent space to effectively identify and mitigate the
+propagation of artifacts. Our SARGD begins by using an artifact detector to
+identify implausible pixels, creating a binary mask that highlights artifacts.
+Following this, the Reality Guidance Refinement (RGR) process refines artifacts
+by integrating this mask with realistic latent representations, improving
+alignment with the original image. Nonetheless, initial realistic-latent
+representations from lower-quality images result in over-smoothing in the final
+output. To address this, we introduce a Self-Adaptive Guidance (SAG) mechanism.
+It dynamically computes a reality score, enhancing the sharpness of the
+realistic latent. These alternating mechanisms collectively achieve
+artifact-free super-resolution. Extensive experiments demonstrate the
+superiority of our method, delivering detailed artifact-free high-resolution
+images while reducing sampling steps by 2X. We release our code at
+https://github.com/ProAirVerse/Self-Adaptive-Guidance-Diffusion.git.",eess.IV,"['eess.IV', 'cs.CV']"
+SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes,Yihua Huang · Yangtian Sun · Ziyi Yang · Xiaoyang Lyu · Yan-Pei Cao · Xiaojuan Qi,https://yihua7.github.io/SC-GS-web/,https://arxiv.org/abs/2312.14937,,2312.14937.pdf,SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes,"Novel view synthesis for dynamic scenes is still a challenging problem in
+computer vision and graphics. Recently, Gaussian splatting has emerged as a
+robust technique to represent static scenes and enable high-quality and
+real-time novel view synthesis. Building upon this technique, we propose a new
+representation that explicitly decomposes the motion and appearance of dynamic
+scenes into sparse control points and dense Gaussians, respectively. Our key
+idea is to use sparse control points, significantly fewer in number than the
+Gaussians, to learn compact 6 DoF transformation bases, which can be locally
+interpolated through learned interpolation weights to yield the motion field of
+3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF
+transformations for each control point, which reduces learning complexities,
+enhances learning abilities, and facilitates obtaining temporal and spatial
+coherent motion patterns. Then, we jointly learn the 3D Gaussians, the
+canonical space locations of control points, and the deformation MLP to
+reconstruct the appearance, geometry, and dynamics of 3D scenes. During
+learning, the location and number of control points are adaptively adjusted to
+accommodate varying motion complexities in different regions, and an ARAP loss
+following the principle of as rigid as possible is developed to enforce spatial
+continuity and local rigidity of learned motions. Finally, thanks to the
+explicit sparse motion representation and its decomposition from appearance,
+our method can enable user-controlled motion editing while retaining
+high-fidelity appearances. Extensive experiments demonstrate that our approach
+outperforms existing approaches on novel view synthesis with a high rendering
+speed and enables novel appearance-preserved motion editing applications.
+Project page: https://yihua7.github.io/SC-GS-web/",cs.CV,"['cs.CV', 'cs.GR']"
+PHYSCENE: Physically Interactable 3D Scene Synthesis for Embodied AI,Yandan Yang · Baoxiong Jia · Peiyuan Zhi · Siyuan Huang, ,https://arxiv.org/abs/2404.09465,,2404.09465.pdf,PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI,"With recent developments in Embodied Artificial Intelligence (EAI) research,
+there has been a growing demand for high-quality, large-scale interactive scene
+generation. While prior methods in scene synthesis have prioritized the
+naturalness and realism of the generated scenes, the physical plausibility and
+interactivity of scenes have been largely left unexplored. To address this
+disparity, we introduce PhyScene, a novel method dedicated to generating
+interactive 3D scenes characterized by realistic layouts, articulated objects,
+and rich physical interactivity tailored for embodied agents. Based on a
+conditional diffusion model for capturing scene layouts, we devise novel
+physics- and interactivity-based guidance mechanisms that integrate constraints
+from object collision, room layout, and object reachability. Through extensive
+experiments, we demonstrate that PhyScene effectively leverages these guidance
+functions for physically interactable scene synthesis, outperforming existing
+state-of-the-art scene synthesis methods by a large margin. Our findings
+suggest that the scenes generated by PhyScene hold considerable potential for
+facilitating diverse skill acquisition among agents within interactive
+environments, thereby catalyzing further advancements in embodied AI research.
+Project website: http://physcene.github.io.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+PlatoNeRF: 3D Reconstruction in Plato’s Cave via Single-View Two-Bounce Lidar,Tzofi Klinghoffer · Xiaoyu Xiang · Siddharth Somasundaram · Yuchen Fan · Christian Richardt · Ramesh Raskar · Rakesh Ranjan,https://platonerf.github.io/,https://arxiv.org/abs/2312.14239,,2312.14239.pdf,PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar,"3D reconstruction from a single-view is challenging because of the ambiguity
+from monocular cues and lack of information about occluded regions. Neural
+radiance fields (NeRF), while popular for view synthesis and 3D reconstruction,
+are typically reliant on multi-view images. Existing methods for single-view 3D
+reconstruction with NeRF rely on either data priors to hallucinate views of
+occluded regions, which may not be physically accurate, or shadows observed by
+RGB cameras, which are difficult to detect in ambient light and low albedo
+backgrounds. We propose using time-of-flight data captured by a single-photon
+avalanche diode to overcome these limitations. Our method models two-bounce
+optical paths with NeRF, using lidar transient data for supervision. By
+leveraging the advantages of both NeRF and two-bounce light measured by lidar,
+we demonstrate that we can reconstruct visible and occluded geometry without
+data priors or reliance on controlled ambient lighting or scene albedo. In
+addition, we demonstrate improved generalization under practical constraints on
+sensor spatial- and temporal-resolution. We believe our method is a promising
+direction as single-photon lidars become ubiquitous on consumer devices, such
+as phones, tablets, and headsets.",cs.CV,"['cs.CV', 'eess.IV']"
+Learning to Rank Patches for Unbiased Image Redundancy Reduction,Yang Luo · Zhineng Chen · Peng Zhou · Zuxuan Wu · Xieping Gao · Yu-Gang Jiang, ,https://arxiv.org/abs/2404.00680,,2404.00680.pdf,Learning to Rank Patches for Unbiased Image Redundancy Reduction,"Images suffer from heavy spatial redundancy because pixels in neighboring
+regions are spatially correlated. Existing approaches strive to overcome this
+limitation by reducing less meaningful image regions. However, current leading
+methods rely on supervisory signals. They may compel models to preserve content
+that aligns with labeled categories and discard content belonging to unlabeled
+categories. This categorical inductive bias makes these methods less effective
+in real-world scenarios. To address this issue, we propose a self-supervised
+framework for image redundancy reduction called Learning to Rank Patches
+(LTRP). We observe that image reconstruction of masked image modeling models is
+sensitive to the removal of visible patches when the masking ratio is high
+(e.g., 90\%). Building upon it, we implement LTRP via two steps: inferring the
+semantic density score of each patch by quantifying variation between
+reconstructions with and without this patch, and learning to rank the patches
+with the pseudo score. The entire process is self-supervised, thus getting out
+of the dilemma of categorical inductive bias. We design extensive experiments
+on different datasets and tasks. The results demonstrate that LTRP outperforms
+both supervised and other self-supervised methods due to the fair assessment of
+image content.",cs.CV,['cs.CV']
+Domain-Rectifying Adapter for Cross-Domain Few-Shot Segmentation,Jiapeng Su · Qi Fan · Wenjie Pei · Guangming Lu · Fanglin Chen, ,https://arxiv.org/abs/2404.10322v1,,2404.10322v1.pdf,Domain-Rectifying Adapter for Cross-Domain Few-Shot Segmentation,"Few-shot semantic segmentation (FSS) has achieved great success on segmenting
+objects of novel classes, supported by only a few annotated samples. However,
+existing FSS methods often underperform in the presence of domain shifts,
+especially when encountering new domain styles that are unseen during training.
+It is suboptimal to directly adapt or generalize the entire model to new
+domains in the few-shot scenario. Instead, our key idea is to adapt a small
+adapter for rectifying diverse target domain styles to the source domain.
+Consequently, the rectified target domain features can fittingly benefit from
+the well-optimized source domain segmentation model, which is intently trained
+on sufficient source domain data. Training domain-rectifying adapter requires
+sufficiently diverse target domains. We thus propose a novel local-global style
+perturbation method to simulate diverse potential target domains by
+perturbating the feature channel statistics of the individual images and
+collective statistics of the entire source domain, respectively. Additionally,
+we propose a cyclic domain alignment module to facilitate the adapter
+effectively rectifying domains using a reverse domain rectification
+supervision. The adapter is trained to rectify the image features from diverse
+synthesized target domains to align with the source domain. During testing on
+target domains, we start by rectifying the image features and then conduct
+few-shot segmentation on the domain-rectified features. Extensive experiments
+demonstrate the effectiveness of our method, achieving promising results on
+cross-domain few-shot semantic segmentation tasks. Our code is available at
+https://github.com/Matt-Su/DR-Adapter.",cs.CV,['cs.CV']
+GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence,Van Nguyen Nguyen · Thibault Groueix · Mathieu Salzmann · Vincent Lepetit, ,https://arxiv.org/abs/2311.14155,,2311.14155.pdf,GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence,"We present GigaPose, a fast, robust, and accurate method for CAD-based novel
+object pose estimation in RGB images. GigaPose first leverages discriminative
+""templates"", rendered images of the CAD models, to recover the out-of-plane
+rotation and then uses patch correspondences to estimate the four remaining
+parameters. Our approach samples templates in only a two-degrees-of-freedom
+space instead of the usual three and matches the input image to the templates
+using fast nearest-neighbor search in feature space, results in a speedup
+factor of 35x compared to the state of the art. Moreover, GigaPose is
+significantly more robust to segmentation errors. Our extensive evaluation on
+the seven core datasets of the BOP challenge demonstrates that it achieves
+state-of-the-art accuracy and can be seamlessly integrated with existing
+refinement methods. Additionally, we show the potential of GigaPose with 3D
+models predicted by recent work on 3D reconstruction from a single image,
+relaxing the need for CAD models and making 6D pose object estimation much more
+convenient. Our source code and trained models are publicly available at
+https://github.com/nv-nguyen/gigaPose",cs.CV,['cs.CV']
+Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval,Fan Zhang · Xian-Sheng Hua · Chong Chen · Xiao Luo, ,,https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4774118,,,,,nan
+SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction,Yang Zhou · Hao Shao · Letian Wang · Steven L. Waslander · Hongsheng Li · Yu Liu, ,https://arxiv.org/abs/2403.11492,,2403.11492.pdf,SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction,"Predicting the future motion of surrounding agents is essential for
+autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed
+environments. Context information, such as road maps and surrounding agents'
+states, provides crucial geometric and semantic information for motion behavior
+prediction. To this end, recent works explore two-stage prediction frameworks
+where coarse trajectories are first proposed, and then used to select critical
+context information for trajectory refinement. However, they either incur a
+large amount of computation or bring limited improvement, if not both. In this
+paper, we introduce a novel scenario-adaptive refinement strategy, named
+SmartRefine, to refine prediction with minimal additional computation.
+Specifically, SmartRefine can comprehensively adapt refinement configurations
+based on each scenario's properties, and smartly chooses the number of
+refinement iterations by introducing a quality score to measure the prediction
+quality and remaining refinement potential of each scenario. SmartRefine is
+designed as a generic and flexible approach that can be seamlessly integrated
+into most state-of-the-art motion prediction models. Experiments on Argoverse
+(1 & 2) show that our method consistently improves the prediction accuracy of
+multiple state-of-the-art prediction models. Specifically, by adding
+SmartRefine to QCNet, we outperform all published ensemble-free works on the
+Argoverse 2 leaderboard (single agent track) at submission. Comprehensive
+studies are also conducted to ablate design choices and explore the mechanism
+behind multi-iteration refinement. Codes are available at
+https://github.com/opendilab/SmartRefine/",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+GenTron: Diffusion Transformers for Image and Video Generation,Shoufa Chen · Mengmeng Xu · Jiawei Ren · Yuren Cong · Sen He · Yanping Xie · Animesh Sinha · Ping Luo · Tao Xiang · Juan-Manuel Pérez-Rúa, ,https://arxiv.org/abs/2312.04557,,2312.04557.pdf,GenTron: Diffusion Transformers for Image and Video Generation,"In this study, we explore Transformer-based diffusion models for image and
+video generation. Despite the dominance of Transformer architectures in various
+fields due to their flexibility and scalability, the visual generative domain
+primarily utilizes CNN-based U-Net architectures, particularly in
+diffusion-based models. We introduce GenTron, a family of Generative models
+employing Transformer-based diffusion, to address this gap. Our initial step
+was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
+process involving thorough empirical exploration of the conditioning mechanism.
+We then scale GenTron from approximately 900M to over 3B parameters, observing
+significant improvements in visual quality. Furthermore, we extend GenTron to
+text-to-video generation, incorporating novel motion-free guidance to enhance
+video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
+rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
+alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
+underscoring its strengths in compositional generation. We believe this work
+will provide meaningful insights and serve as a valuable reference for future
+research.",cs.CV,['cs.CV']
+Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch,Xidong Wu · Shangqian Gao · Zeyu Zhang · Zhenzhen Li · Runxue Bao · Yanfu Zhang · Xiaoqian Wang · Heng Huang, ,https://arxiv.org/abs/2403.14729,,2403.14729.pdf,Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch,"Current techniques for deep neural network (DNN) pruning often involve
+intricate multi-step processes that require domain-specific expertise, making
+their widespread adoption challenging. To address the limitation, the
+Only-Train-Once (OTO) and OTOv2 are proposed to eliminate the need for
+additional fine-tuning steps by directly training and compressing a general DNN
+from scratch. Nevertheless, the static design of optimizers (in OTO) can lead
+to convergence issues of local optima. In this paper, we proposed the
+Auto-Train-Once (ATO), an innovative network pruning algorithm designed to
+automatically reduce the computational and storage costs of DNNs. During the
+model training phase, our approach not only trains the target model but also
+leverages a controller network as an architecture generator to guide the
+learning of target model weights. Furthermore, we developed a novel stochastic
+gradient algorithm that enhances the coordination between model training and
+controller network training, thereby improving pruning performance. We provide
+a comprehensive convergence analysis as well as extensive experiments, and the
+results show that our approach achieves state-of-the-art performance across
+various model architectures (including ResNet18, ResNet34, ResNet50, ResNet56,
+and MobileNetv2) on standard benchmark datasets (CIFAR-10, CIFAR-100, and
+ImageNet).",cs.CV,"['cs.CV', 'cs.LG']"
+NOPE: Novel Object Pose Estimation from a Single Image,Van Nguyen Nguyen · Thibault Groueix · Georgy Ponimatkin · Yinlin Hu · Renaud Marlet · Mathieu Salzmann · Vincent Lepetit, ,https://arxiv.org/abs/2311.14155,,,GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence,"We present GigaPose, a fast, robust, and accurate method for CAD-based novel
+object pose estimation in RGB images. GigaPose first leverages discriminative
+""templates"", rendered images of the CAD models, to recover the out-of-plane
+rotation and then uses patch correspondences to estimate the four remaining
+parameters. Our approach samples templates in only a two-degrees-of-freedom
+space instead of the usual three and matches the input image to the templates
+using fast nearest-neighbor search in feature space, results in a speedup
+factor of 35x compared to the state of the art. Moreover, GigaPose is
+significantly more robust to segmentation errors. Our extensive evaluation on
+the seven core datasets of the BOP challenge demonstrates that it achieves
+state-of-the-art accuracy and can be seamlessly integrated with existing
+refinement methods. Additionally, we show the potential of GigaPose with 3D
+models predicted by recent work on 3D reconstruction from a single image,
+relaxing the need for CAD models and making 6D pose object estimation much more
+convenient. Our source code and trained models are publicly available at
+https://github.com/nv-nguyen/gigaPose",cs.CV,['cs.CV']
+Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction,Hao Li · Ying Chen · Yifei Chen · Rongshan Yu · Wenxian Yang · Liansheng Wang · Bowen Ding · Yuchen Han, ,https://arxiv.org/abs/2402.19326,,2402.19326.pdf,Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction,"Whole Slide Image (WSI) classification is often formulated as a Multiple
+Instance Learning (MIL) problem. Recently, Vision-Language Models (VLMs) have
+demonstrated remarkable performance in WSI classification. However, existing
+methods leverage coarse-grained pathogenetic descriptions for visual
+representation supervision, which are insufficient to capture the complex
+visual appearance of pathogenetic images, hindering the generalizability of
+models on diverse downstream tasks. Additionally, processing high-resolution
+WSIs can be computationally expensive. In this paper, we propose a novel
+""Fine-grained Visual-Semantic Interaction"" (FiVE) framework for WSI
+classification. It is designed to enhance the model's generalizability by
+leveraging the interaction between localized visual patterns and fine-grained
+pathological semantics. Specifically, with meticulously designed queries, we
+start by utilizing a large language model to extract fine-grained pathological
+descriptions from various non-standardized raw reports. The output descriptions
+are then reconstructed into fine-grained labels used for training. By
+introducing a Task-specific Fine-grained Semantics (TFS) module, we enable
+prompts to capture crucial visual information in WSIs, which enhances
+representation learning and augments generalization capabilities significantly.
+Furthermore, given that pathological visual patterns are redundantly
+distributed across tissue slices, we sample a subset of visual instances during
+training. Our method demonstrates robust generalizability and strong
+transferability, dominantly outperforming the counterparts on the TCGA Lung
+Cancer dataset with at least 9.19% higher accuracy in few-shot experiments. The
+code is available at: https://github.com/ls1rius/WSI_FiVE.",cs.CV,['cs.CV']
+In-distribution Public Data Synthesis with Diffusion Models for Differentially Private Image Classification,Jinseong Park · Yujin Choi · Jaewook Lee,https://jinseongp.github.io/2024/05/28/cvpr2024.html,,https://jinseongp.github.io/2024/05/28/cvpr2024.html,,,,,nan
+CapHuman: Capture Your Moments in Parallel Universes,Chao Liang · Fan Ma · Linchao Zhu · Yingying Deng · Yi Yang,https://caphuman.github.io/,https://arxiv.org/abs/2402.00627,,2402.00627.pdf,CapHuman: Capture Your Moments in Parallel Universes,"We concentrate on a novel human-centric image synthesis task, that is, given
+only one reference facial photograph, it is expected to generate specific
+individual images with diverse head positions, poses, facial expressions, and
+illuminations in different contexts. To accomplish this goal, we argue that our
+generative model should be capable of the following favorable characteristics:
+(1) a strong visual and semantic understanding of our world and human society
+for basic object and human image generation. (2) generalizable identity
+preservation ability. (3) flexible and fine-grained head control. Recently,
+large pre-trained text-to-image diffusion models have shown remarkable results,
+serving as a powerful generative foundation. As a basis, we aim to unleash the
+above two capabilities of the pre-trained model. In this work, we present a new
+framework named CapHuman. We embrace the ""encode then learn to align"" paradigm,
+which enables generalizable identity preservation for new individuals without
+cumbersome tuning at inference. CapHuman encodes identity features and then
+learns to align them into the latent space. Moreover, we introduce the 3D
+facial prior to equip our model with control over the human head in a flexible
+and 3D-consistent manner. Extensive qualitative and quantitative analyses
+demonstrate our CapHuman can produce well-identity-preserved, photo-realistic,
+and high-fidelity portraits with content-rich representations and various head
+renditions, superior to established baselines. Code and checkpoint will be
+released at https://github.com/VamosC/CapHuman.",cs.CV,"['cs.CV', 'cs.AI']"
+CuVLER: Enhanced Unsupervised Object Discoveries through Exhaustive Self-Supervised Transformers,Shahaf Arica · Or Rubin · Sapir Gershov · Shlomi Laufer,https://github.com/shahaf-arica/cuvler,https://arxiv.org/abs/2403.07700,,2403.07700.pdf,CuVLER: Enhanced Unsupervised Object Discoveries through Exhaustive Self-Supervised Transformers,"In this paper, we introduce VoteCut, an innovative method for unsupervised
+object discovery that leverages feature representations from multiple
+self-supervised models. VoteCut employs normalized-cut based graph
+partitioning, clustering and a pixel voting approach. Additionally, We present
+CuVLER (Cut-Vote-and-LEaRn), a zero-shot model, trained using pseudo-labels,
+generated by VoteCut, and a novel soft target loss to refine segmentation
+accuracy. Through rigorous evaluations across multiple datasets and several
+unsupervised setups, our methods demonstrate significant improvements in
+comparison to previous state-of-the-art models. Our ablation studies further
+highlight the contributions of each component, revealing the robustness and
+efficacy of our approach. Collectively, VoteCut and CuVLER pave the way for
+future advancements in image segmentation.",cs.CV,['cs.CV']
+LEDITS++: Limitless Image Editing using Text-to-Image Models,Manuel Brack · Felix Friedrich · Katharina Kornmeier · Linoy Tsaban · Patrick Schramowski · Kristian Kersting · Apolinário Passos, ,https://arxiv.org/abs/2311.16711,,2311.16711.pdf,LEDITS++: Limitless Image Editing using Text-to-Image Models,"Text-to-image diffusion models have recently received increasing interest for
+their astonishing ability to produce high-fidelity images from solely text
+inputs. Subsequent research efforts aim to exploit and apply their capabilities
+to real image editing. However, existing image-to-image methods are often
+inefficient, imprecise, and of limited versatility. They either require
+time-consuming fine-tuning, deviate unnecessarily strongly from the input
+image, and/or lack support for multiple, simultaneous edits. To address these
+issues, we introduce LEDITS++, an efficient yet versatile and precise textual
+image manipulation technique. LEDITS++'s novel inversion approach requires no
+tuning nor optimization and produces high-fidelity results with a few diffusion
+steps. Second, our methodology supports multiple simultaneous edits and is
+architecture-agnostic. Third, we use a novel implicit masking technique that
+limits changes to relevant image regions. We propose the novel TEdBench++
+benchmark as part of our exhaustive evaluation. Our results demonstrate the
+capabilities of LEDITS++ and its improvements over previous methods. The
+project page is available at https://leditsplusplus-project.static.hf.space .",cs.CV,"['cs.CV', 'cs.AI', 'cs.HC', 'cs.LG']"
+Are Conventional SNNs Really Efficient? A Perspective from Network Quantization,Guobin Shen · Dongcheng Zhao · Tenglong Li · Jindong Li · Yi Zeng, ,https://arxiv.org/abs/2311.10802,,2311.10802.pdf,Is Conventional SNN Really Efficient? A Perspective from Network Quantization,"Spiking Neural Networks (SNNs) have been widely praised for their high energy
+efficiency and immense potential. However, comprehensive research that
+critically contrasts and correlates SNNs with quantized Artificial Neural
+Networks (ANNs) remains scant, often leading to skewed comparisons lacking
+fairness towards ANNs. This paper introduces a unified perspective,
+illustrating that the time steps in SNNs and quantized bit-widths of activation
+values present analogous representations. Building on this, we present a more
+pragmatic and rational approach to estimating the energy consumption of SNNs.
+Diverging from the conventional Synaptic Operations (SynOps), we champion the
+""Bit Budget"" concept. This notion permits an intricate discourse on
+strategically allocating computational and storage resources between weights,
+activation values, and temporal steps under stringent hardware constraints.
+Guided by the Bit Budget paradigm, we discern that pivoting efforts towards
+spike patterns and weight quantization, rather than temporal attributes,
+elicits profound implications for model performance. Utilizing the Bit Budget
+for holistic design consideration of SNNs elevates model performance across
+diverse data types, encompassing static imagery and neuromorphic datasets. Our
+revelations bridge the theoretical chasm between SNNs and quantized ANNs and
+illuminate a pragmatic trajectory for future endeavors in energy-efficient
+neural computations.",cs.NE,['cs.NE']
+Task-conditioned adaptation of visual features in multi-task policy learning,Pierre Marza · Laetitia Matignon · Olivier Simonin · Christian Wolf,https://pierremarza.github.io/projects/task_conditioned_adaptation/,https://arxiv.org/abs/2402.07739v1,,2402.07739v1.pdf,Task-conditioned adaptation of visual features in multi-task policy learning,"Successfully addressing a wide variety of tasks is a core ability of
+autonomous agents, which requires flexibly adapting the underlying
+decision-making strategies and, as we argue in this work, also adapting the
+underlying perception modules. An analogical argument would be the human visual
+system, which uses top-down signals to focus attention determined by the
+current task. Similarly, in this work, we adapt pre-trained large vision models
+conditioned on specific downstream tasks in the context of multi-task policy
+learning. We introduce task-conditioned adapters that do not require finetuning
+any pre-trained weights, combined with a single policy trained with behavior
+cloning and capable of addressing multiple tasks. We condition the policy and
+visual adapters on task embeddings, which can be selected at inference if the
+task is known, or alternatively inferred from a set of example demonstrations.
+To this end, we propose a new optimization-based estimator. We evaluate the
+method on a wide variety of tasks of the CortexBench benchmark and show that,
+compared to existing work, it can be addressed with a single policy. In
+particular, we demonstrate that adapting visual features is a key design choice
+and that the method generalizes to unseen tasks given visual demonstrations.",cs.CV,"['cs.CV', 'cs.LG', 'cs.RO']"
+Open-Vocabulary Video Anomaly Detection,Peng Wu · Xuerong Zhou · Guansong Pang · Yujia Sun · Jing Liu · Peng Wang · Yanning Zhang, ,https://arxiv.org/abs/2311.07042,,2311.07042.pdf,Open-Vocabulary Video Anomaly Detection,"Video anomaly detection (VAD) with weak supervision has achieved remarkable
+performance in utilizing video-level labels to discriminate whether a video
+frame is normal or abnormal. However, current approaches are inherently limited
+to a closed-set setting and may struggle in open-world applications where there
+can be anomaly categories in the test data unseen during training. A few recent
+studies attempt to tackle a more realistic setting, open-set VAD, which aims to
+detect unseen anomalies given seen anomalies and normal videos. However, such a
+setting focuses on predicting frame anomaly scores, having no ability to
+recognize the specific categories of anomalies, despite the fact that this
+ability is essential for building more informed video surveillance systems.
+This paper takes a step further and explores open-vocabulary video anomaly
+detection (OVVAD), in which we aim to leverage pre-trained large models to
+detect and categorize seen and unseen anomalies. To this end, we propose a
+model that decouples OVVAD into two mutually complementary tasks --
+class-agnostic detection and class-specific classification -- and jointly
+optimizes both tasks. Particularly, we devise a semantic knowledge injection
+module to introduce semantic knowledge from large language models for the
+detection task, and design a novel anomaly synthesis module to generate pseudo
+unseen anomaly videos with the help of large vision generation models for the
+classification task. These semantic knowledge and synthesis anomalies
+substantially extend our model's capability in detecting and categorizing a
+variety of seen and unseen anomalies. Extensive experiments on three
+widely-used benchmarks demonstrate our model achieves state-of-the-art
+performance on OVVAD task.",cs.CV,['cs.CV']
+Hierarchical Histogram Threshold Segmentation – Auto-terminating High-detail Oversegmentation,Thomas Chang · Simon Seibt · Bartosz von Rymon Lipinski,https://changtvs.github.io/hierarchical-histogram-threshold-segmentation/,,https://www.nature.com/articles/s41598-023-36066-8,,,,,nan
+ManiFPT: Defining and Analyzing Fingerprints of Generative Models,Hae Jin Song · Mahyar Khayatkhoei · Wael AbdAlmageed, ,https://arxiv.org/abs/2402.10401,,2402.10401.pdf,ManiFPT: Defining and Analyzing Fingerprints of Generative Models,"Recent works have shown that generative models leave traces of their
+underlying generative process on the generated samples, broadly referred to as
+fingerprints of a generative model, and have studied their utility in detecting
+synthetic images from real ones. However, the extend to which these
+fingerprints can distinguish between various types of synthetic image and help
+identify the underlying generative process remain under-explored. In
+particular, the very definition of a fingerprint remains unclear, to our
+knowledge. To that end, in this work, we formalize the definition of artifact
+and fingerprint in generative models, propose an algorithm for computing them
+in practice, and finally study its effectiveness in distinguishing a large
+array of different generative models. We find that using our proposed
+definition can significantly improve the performance on the task of identifying
+the underlying generative process from samples (model attribution) compared to
+existing methods. Additionally, we study the structure of the fingerprints, and
+observe that it is very predictive of the effect of different design choices on
+the generative process.",cs.LG,"['cs.LG', 'cs.CV']"
+Beyond Text: Frozen Large Language Models in Visual Signal Comprehension,Lei Zhu · Fangyun Wei · Yanye Lu, ,https://arxiv.org/abs/2403.07874,,2403.07874.pdf,Beyond Text: Frozen Large Language Models in Visual Signal Comprehension,"In this work, we investigate the potential of a large language model (LLM) to
+directly comprehend visual signals without the necessity of fine-tuning on
+multi-modal datasets. The foundational concept of our method views an image as
+a linguistic entity, and translates it to a set of discrete words derived from
+the LLM's vocabulary. To achieve this, we present the Vision-to-Language
+Tokenizer, abbreviated as V2T Tokenizer, which transforms an image into a
+``foreign language'' with the combined aid of an encoder-decoder, the LLM
+vocabulary, and a CLIP model. With this innovative image encoding, the LLM
+gains the ability not only for visual comprehension but also for image
+denoising and restoration in an auto-regressive fashion-crucially, without any
+fine-tuning. We undertake rigorous experiments to validate our method,
+encompassing understanding tasks like image recognition, image captioning, and
+visual question answering, as well as image denoising tasks like inpainting,
+outpainting, deblurring, and shift restoration. Code and models are available
+at https://github.com/zh460045050/V2L-Tokenizer.",cs.CV,['cs.CV']
+Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation,Philipp Schröppel · Christopher Wewer · Jan Lenssen · Eddy Ilg · Thomas Brox,https://neural-point-cloud-diffusion.github.io/,https://arxiv.org/abs/2312.14124,,2312.14124.pdf,Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation,"Controllable generation of 3D assets is important for many practical
+applications like content creation in movies, games and engineering, as well as
+in AR/VR. Recently, diffusion models have shown remarkable results in
+generation quality of 3D objects. However, none of the existing models enable
+disentangled generation to control the shape and appearance separately. For the
+first time, we present a suitable representation for 3D diffusion models to
+enable such disentanglement by introducing a hybrid point cloud and neural
+radiance field approach. We model a diffusion process over point positions
+jointly with a high-dimensional feature space for a local density and radiance
+decoder. While the point positions represent the coarse shape of the object,
+the point features allow modeling the geometry and appearance details. This
+disentanglement enables us to sample both independently and therefore to
+control both separately. Our approach sets a new state of the art in generation
+compared to previous disentanglement-capable methods by reduced FID scores of
+30-90% and is on-par with other non disentanglement-capable state-of-the art
+methods.",cs.CV,['cs.CV']
+SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos,Tao Wu · Runyu He · Gangshan Wu · Limin Wang,https://github.com/MCG-NJU/SportsHHI,https://arxiv.org/abs/2404.04565,,2404.04565.pdf,SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos,"Video-based visual relation detection tasks, such as video scene graph
+generation, play important roles in fine-grained video understanding. However,
+current video visual relation detection datasets have two main limitations that
+hinder the progress of research in this area. First, they do not explore
+complex human-human interactions in multi-person scenarios. Second, the
+relation types of existing datasets have relatively low-level semantics and can
+be often recognized by appearance or simple prior information, without the need
+for detailed spatio-temporal context reasoning. Nevertheless, comprehending
+high-level interactions between humans is crucial for understanding complex
+multi-person videos, such as sports and surveillance videos. To address this
+issue, we propose a new video visual relation detection task: video human-human
+interaction detection, and build a dataset named SportsHHI for it. SportsHHI
+contains 34 high-level interaction classes from basketball and volleyball
+sports. 118,075 human bounding boxes and 50,649 interaction instances are
+annotated on 11,398 keyframes. To benchmark this, we propose a two-stage
+baseline method and conduct extensive experiments to reveal the key factors for
+a successful human-human interaction detector. We hope that SportsHHI can
+stimulate research on human interaction understanding in videos and promote the
+development of spatio-temporal context modeling techniques in video visual
+relation detection.",cs.CV,['cs.CV']
+Time-Efficient Light-Field Acquisition Using Coded Aperture and Events,Shuji Habuchi · Keita Takahashi · Chihiro Tsutake · Toshiaki Fujii · Hajime Nagahara,https://www.fujii.nuee.nagoya-u.ac.jp/Research/EventLF/,https://arxiv.org/abs/2403.07244,,2403.07244.pdf,Time-Efficient Light-Field Acquisition Using Coded Aperture and Events,"We propose a computational imaging method for time-efficient light-field
+acquisition that combines a coded aperture with an event-based camera.
+Different from the conventional coded-aperture imaging method, our method
+applies a sequence of coding patterns during a single exposure for an image
+frame. The parallax information, which is related to the differences in coding
+patterns, is recorded as events. The image frame and events, all of which are
+measured in a single exposure, are jointly used to computationally reconstruct
+a light field. We also designed an algorithm pipeline for our method that is
+end-to-end trainable on the basis of deep optics and compatible with real
+camera hardware. We experimentally showed that our method can achieve more
+accurate reconstruction than several other imaging methods with a single
+exposure. We also developed a hardware prototype with the potential to complete
+the measurement on the camera within 22 msec and demonstrated that light fields
+from real 3-D scenes can be obtained with convincing visual quality. Our
+software and supplementary video are available from our project website.",cs.CV,"['cs.CV', 'eess.IV']"
+Rapid Motor Adaptation for Robotic Manipulator Arms,Yichao Liang · Kevin Ellis · João F. Henriques, ,https://arxiv.org/abs/2312.04670v1,,2312.04670v1.pdf,Rapid Motor Adaptation for Robotic Manipulator Arms,"Developing generalizable manipulation skills is a core challenge in embodied
+AI. This includes generalization across diverse task configurations,
+encompassing variations in object shape, density, friction coefficient, and
+external disturbances such as forces applied to the robot. Rapid Motor
+Adaptation (RMA) offers a promising solution to this challenge. It posits that
+essential hidden variables influencing an agent's task performance, such as
+object mass and shape, can be effectively inferred from the agent's action and
+proprioceptive history. Drawing inspiration from RMA in locomotion and in-hand
+rotation, we use depth perception to develop agents tailored for rapid motor
+adaptation in a variety of manipulation tasks. We evaluated our agents on four
+challenging tasks from the Maniskill2 benchmark, namely pick-and-place
+operations with hundreds of objects from the YCB and EGAD datasets, peg
+insertion with precise position and orientation, and operating a variety of
+faucets and handles, with customized environment variations. Empirical results
+demonstrate that our agents surpass state-of-the-art methods like automatic
+domain randomization and vision-based policies, obtaining better generalization
+performance and sample efficiency.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV', 'cs.LG']"
+Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation,Jin Wang · Bingfeng Zhang · Jian Pang · Honglong Chen · Weifeng Liu, ,https://arxiv.org/abs/2405.08458,,2405.08458.pdf,Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation,"Few-shot segmentation remains challenging due to the limitations of its
+labeling information for unseen classes. Most previous approaches rely on
+extracting high-level feature maps from the frozen visual encoder to compute
+the pixel-wise similarity as a key prior guidance for the decoder. However,
+such a prior representation suffers from coarse granularity and poor
+generalization to new classes since these high-level feature maps have obvious
+category bias. In this work, we propose to replace the visual prior
+representation with the visual-text alignment capacity to capture more reliable
+guidance and enhance the model generalization. Specifically, we design two
+kinds of training-free prior information generation strategy that attempts to
+utilize the semantic alignment capability of the Contrastive Language-Image
+Pre-training model (CLIP) to locate the target class. Besides, to acquire more
+accurate prior guidance, we build a high-order relationship of attention maps
+and utilize it to refine the initial prior information. Experiments on both the
+PASCAL-5{i} and COCO-20{i} datasets show that our method obtains a clearly
+substantial improvement and reaches the new state-of-the-art performance.",cs.CV,['cs.CV']
+A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning,Yuelin Zhang · Pengyu Zheng · Wanquan Yan · Chengyu Fang · Shing Shin Cheng, ,https://arxiv.org/abs/2403.02611,,2403.02611.pdf,A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning,"Defocus blur is a persistent problem in microscope imaging that poses harm to
+pathology interpretation and medical intervention in cell microscopy and
+microscope surgery. To address this problem, a unified framework including the
+multi-pyramid transformer (MPT) and extended frequency contrastive
+regularization (EFCR) is proposed to tackle two outstanding challenges in
+microscopy deblur: longer attention span and data deficiency. The MPT employs
+an explicit pyramid structure at each network stage that integrates the
+cross-scale window attention (CSWA), the intra-scale channel attention (ISCA),
+and the feature-enhancing feed-forward network (FEFN) to capture long-range
+cross-scale spatial interaction and global channel context. The EFCR addresses
+the data deficiency problem by exploring latent deblur signals from different
+frequency bands. It also enables deblur knowledge transfer to learn
+cross-domain information from extra data, improving deblur performance for
+labeled and unlabeled data. Extensive experiments and downstream task
+validation show the framework achieves state-of-the-art performance across
+multiple datasets. Project page: https://github.com/PieceZhang/MPT-CataBlur.",cs.CV,"['cs.CV', 'cs.AI']"
+Rotation-Agnostic Image Representation Learning for Digital Pathology,Saghir Alfasly · Abubakr Shafique · Peyman Nejat · Jibran Khan · Areej Alsaafin · Ghazal Alabtah · Hamid Tizhoosh,https://kimialabmayo.github.io/PathDino-Page/,https://arxiv.org/abs/2311.08359,,2311.08359.pdf,Rotation-Agnostic Image Representation Learning for Digital Pathology,"This paper addresses complex challenges in histopathological image analysis
+through three key contributions. Firstly, it introduces a fast patch selection
+method, FPS, for whole-slide image (WSI) analysis, significantly reducing
+computational cost while maintaining accuracy. Secondly, it presents PathDino,
+a lightweight histopathology feature extractor with a minimal configuration of
+five Transformer blocks and only 9 million parameters, markedly fewer than
+alternatives. Thirdly, it introduces a rotation-agnostic representation
+learning paradigm using self-supervised learning, effectively mitigating
+overfitting. We also show that our compact model outperforms existing
+state-of-the-art histopathology-specific vision transformers on 12 diverse
+datasets, including both internal datasets spanning four sites (breast, liver,
+skin, and colorectal) and seven public datasets (PANDA, CAMELYON16, BRACS,
+DigestPath, Kather, PanNuke, and WSSS4LUAD). Notably, even with a training
+dataset of 6 million histopathology patches from The Cancer Genome Atlas
+(TCGA), our approach demonstrates an average 8.5% improvement in patch-level
+majority vote performance. These contributions provide a robust framework for
+enhancing image analysis in digital pathology, rigorously validated through
+extensive evaluation. Project Page:
+https://kimialabmayo.github.io/PathDino-Page/",cs.CV,['cs.CV']
+Weakly Misalignment-free Adaptive Feature Alignment for UAVs-based Multimodal Object Detection,Chen Chen · Jiahao Qi · Xingyue Liu · Kangcheng Bin · Ruigang Fu · Xikun Hu · Ping Zhong, ,https://arxiv.org/abs/2405.16873,,2405.16873.pdf,ContrastAlign: Toward Robust BEV Feature Alignment via Contrastive Learning for Multi-Modal 3D Object Detection,"In the field of 3D object detection tasks, fusing heterogeneous features from
+LiDAR and camera sensors into a unified Bird's Eye View (BEV) representation is
+a widely adopted paradigm. However, existing methods are often compromised by
+imprecise sensor calibration, resulting in feature misalignment in LiDAR-camera
+BEV fusion. Moreover, such inaccuracies result in errors in depth estimation
+for the camera branch, ultimately causing misalignment between LiDAR and camera
+BEV features. In this work, we propose a novel ContrastAlign approach that
+utilizes contrastive learning to enhance the alignment of heterogeneous
+modalities, thereby improving the robustness of the fusion process.
+Specifically, our approach includes the L-Instance module, which directly
+outputs LiDAR instance features within LiDAR BEV features. Then, we introduce
+the C-Instance module, which predicts camera instance features through RoI
+(Region of Interest) pooling on the camera BEV features. We propose the
+InstanceFusion module, which utilizes contrastive learning to generate similar
+instance features across heterogeneous modalities. We then use graph matching
+to calculate the similarity between the neighboring camera instance features
+and the similarity instance features to complete the alignment of instance
+features. Our method achieves state-of-the-art performance, with an mAP of
+70.3%, surpassing BEVFusion by 1.8% on the nuScenes validation set.
+Importantly, our method outperforms BEVFusion by 7.3% under conditions with
+misalignment noise.",cs.CV,['cs.CV']
+Learning with Structural Labels for Learning with Noisy Labels,Noo-ri Kim · Jin-Seop Lee · Jee-Hyong Lee, ,https://arxiv.org/abs/2401.04390,,2401.04390.pdf,Learning with Noisy Labels: Interconnection of Two Expectation-Maximizations,"Labor-intensive labeling becomes a bottleneck in developing computer vision
+algorithms based on deep learning. For this reason, dealing with imperfect
+labels has increasingly gained attention and has become an active field of
+study. We address learning with noisy labels (LNL) problem, which is formalized
+as a task of finding a structured manifold in the midst of noisy data. In this
+framework, we provide a proper objective function and an optimization algorithm
+based on two expectation-maximization (EM) cycles. The separate networks
+associated with the two EM cycles collaborate to optimize the objective
+function, where one model is for distinguishing clean labels from corrupted
+ones while the other is for refurbishing the corrupted labels. This approach
+results in a non-collapsing LNL-flywheel model in the end. Experiments show
+that our algorithm achieves state-of-the-art performance in multiple standard
+benchmarks with substantial margins under various types of label noise.",cs.CV,['cs.CV']
+Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training,Runze He · Shaofei Huang · Xuecheng Nie · Tianrui Hui · Luoqi Liu · Jiao Dai · Jizhong Han · Guanbin Li · Si Liu,https://customnerf.github.io/,https://arxiv.org/abs/2312.01663,,2312.01663.pdf,Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training,"In this paper, we target the adaptive source driven 3D scene editing task by
+proposing a CustomNeRF model that unifies a text description or a reference
+image as the editing prompt. However, obtaining desired editing results
+conformed with the editing prompt is nontrivial since there exist two
+significant challenges, including accurate editing of only foreground regions
+and multi-view consistency given a single-view reference image. To tackle the
+first challenge, we propose a Local-Global Iterative Editing (LGIE) training
+scheme that alternates between foreground region editing and full-image
+editing, aimed at foreground-only manipulation while preserving the background.
+For the second challenge, we also design a class-guided regularization that
+exploits class priors within the generation model to alleviate the
+inconsistency problem among different views in image-driven editing. Extensive
+experiments show that our CustomNeRF produces precise editing results under
+various real scenes for both text- and image-driven settings.",cs.CV,"['cs.CV', 'cs.AI']"
+Training-free Pretrained Model Merging,Zhengqi Xu · Ke Yuan · Huiqiong Wang · Yong Wang · Mingli Song · Jie Song,https://github.com/zju-vipa/training_free_model_merging,https://arxiv.org/abs/2403.01753,,2403.01753.pdf,Training-Free Pretrained Model Merging,"Recently, model merging techniques have surfaced as a solution to combine
+multiple single-talent models into a single multi-talent model. However,
+previous endeavors in this field have either necessitated additional training
+or fine-tuning processes, or require that the models possess the same
+pre-trained initialization. In this work, we identify a common drawback in
+prior works w.r.t. the inconsistency of unit similarity in the weight space and
+the activation space. To address this inconsistency, we propose an innovative
+model merging framework, coined as merging under dual-space constraints
+(MuDSC). Specifically, instead of solely maximizing the objective of a single
+space, we advocate for the exploration of permutation matrices situated in a
+region with a unified high similarity in the dual space, achieved through the
+linear combination of activation and weight similarity matrices. In order to
+enhance usability, we have also incorporated adaptations for group structure,
+including Multi-Head Attention and Group Normalization. Comprehensive
+experimental comparisons demonstrate that MuDSC can significantly boost the
+performance of merged models with various task combinations and architectures.
+Furthermore, the visualization of the merged model within the multi-task loss
+landscape reveals that MuDSC enables the merged model to reside in the
+overlapping segment, featuring a unified lower loss for each task. Our code is
+publicly available at https://github.com/zju-vipa/training_free_model_merging.",cs.CV,['cs.CV']
+SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model,Zhengang Li · Yan Kang · Yuchen Liu · Difan Liu · Tobias Hinz · Feng Liu · Yanzhi Wang, ,https://ar5iv.labs.arxiv.org/html/2211.11018,,2211.11018.pdf,MagicVideo: Efficient Video Generation With Latent Diffusion Models,"We present an efficient text-to-video generation framework based on latent
+diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips
+that are concordant with the given text descriptions. Due to a novel and
+efficient 3D U-Net design and modeling video distributions in a low-dimensional
+space, MagicVideo can synthesize video clips with 256x256 spatial resolution on
+a single GPU card, which takes around 64x fewer computations than the Video
+Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works
+that directly train video models in the RGB space, we use a pre-trained VAE to
+map video clips into a low-dimensional latent space and learn the distribution
+of videos' latent codes via a diffusion model. Besides, we introduce two new
+designs to adapt the U-Net denoiser trained on image tasks to video data: a
+frame-wise lightweight adaptor for the image-to-video distribution adjustment
+and a directed temporal attention module to capture temporal dependencies
+across frames. Thus, we can exploit the informative weights of convolution
+operators from a text-to-image model for accelerating video training. To
+ameliorate the pixel dithering in the generated videos, we also propose a novel
+VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive
+experiments and demonstrate that MagicVideo can generate high-quality video
+clips with either realistic or imaginary content. Refer to
+\url{https://magicvideo.github.io/#} for more examples.",cs.CV,['cs.CV']
+GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects,Sungphill Moon · Hyeontae Son · Dongcheol Hur · Sangwook Kim, ,https://arxiv.org/abs/2403.11510,,2403.11510.pdf,GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects,"Despite the progress of learning-based methods for 6D object pose estimation,
+the trade-off between accuracy and scalability for novel objects still exists.
+Specifically, previous methods for novel objects do not make good use of the
+target object's 3D shape information since they focus on generalization by
+processing the shape indirectly, making them less effective. We present
+GenFlow, an approach that enables both accuracy and generalization to novel
+objects with the guidance of the target object's shape. Our method predicts
+optical flow between the rendered image and the observed image and refines the
+6D pose iteratively. It boosts the performance by a constraint of the 3D shape
+and the generalizable geometric knowledge learned from an end-to-end
+differentiable system. We further improve our model by designing a cascade
+network architecture to exploit the multi-scale correlations and coarse-to-fine
+refinement. GenFlow ranked first on the unseen object pose estimation
+benchmarks in both the RGB and RGB-D cases. It also achieves performance
+competitive with existing state-of-the-art methods for the seen object pose
+estimation without any fine-tuning.",cs.CV,['cs.CV']
+Day-Night Cross-domain Vehicle Re-identification,Hongchao Li · Jingong Chen · AIHUA ZHENG · Yong Wu · YongLong Luo, ,,https://www.mdpi.com/2079-9292/13/10/1823,,,,,nan
+Making Visual Sense of Oracle Bones for You and Me,Runqi Qiao · LAN YANG · Kaiyue Pang · Honggang Zhang, ,https://arxiv.org/abs/2311.15421,,2311.15421.pdf,Wired Perspectives: Multi-View Wire Art Embraces Generative AI,"Creating multi-view wire art (MVWA), a static 3D sculpture with diverse
+interpretations from different viewpoints, is a complex task even for skilled
+artists. In response, we present DreamWire, an AI system enabling everyone to
+craft MVWA easily. Users express their vision through text prompts or
+scribbles, freeing them from intricate 3D wire organisation. Our approach
+synergises 3D B\'ezier curves, Prim's algorithm, and knowledge distillation
+from diffusion models or their variants (e.g., ControlNet). This blend enables
+the system to represent 3D wire art, ensuring spatial continuity and overcoming
+data scarcity. Extensive evaluation and analysis are conducted to shed insight
+on the inner workings of the proposed system, including the trade-off between
+connectivity and visual aesthetics.",cs.CV,"['cs.CV', 'cs.AI']"
+EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI,Tai Wang · Xiaohan Mao · Chenming Zhu · Runsen Xu · Ruiyuan Lyu · Peisen Li · Xiao Chen · Wenwei Zhang · Kai Chen · Tianfan Xue · Xihui Liu · Cewu Lu · Dahua Lin · Jiangmiao Pang, ,https://arxiv.org/abs/2312.16170v1,,2312.16170v1.pdf,EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI,"In the realm of computer vision and robotics, embodied agents are expected to
+explore their environment and carry out human instructions. This necessitates
+the ability to fully understand 3D scenes given their first-person observations
+and contextualize them into language for interaction. However, traditional
+research focuses more on scene-level input and output setups from a global
+view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric
+3D perception dataset and benchmark for holistic 3D scene understanding. It
+encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language
+prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which
+partially align with LVIS, and dense semantic occupancy with 80 common
+categories. Building upon this database, we introduce a baseline framework
+named Embodied Perceptron. It is capable of processing an arbitrary number of
+multi-modal inputs and demonstrates remarkable 3D perception capabilities, both
+within the two series of benchmarks we set up, i.e., fundamental 3D perception
+tasks and language-grounded tasks, and in the wild. Codes, datasets, and
+benchmarks will be available at https://github.com/OpenRobotLab/EmbodiedScan.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+HouseCat6D - A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios,HyunJun Jung · Shun-Cheng Wu · Patrick Ruhkamp · Guangyao Zhai · Hannah Schieber · Giulia Rizzoli · Pengyuan Wang · Hongcheng Zhao · Lorenzo Garattoni · Sven Meier · Daniel Roth · Nassir Navab · Benjamin Busam,https://sites.google.com/view/housecat6d,https://ar5iv.labs.arxiv.org/html/2308.10627,,2308.10627.pdf,Polarimetric Information for Multi-Modal 6D Pose Estimation of Photometrically Challenging Objects with Limited Data,"6D pose estimation pipelines that rely on RGB-only or RGB-D data show
+limitations for photometrically challenging objects with e.g. textureless
+surfaces, reflections or transparency. A supervised learning-based method
+utilising complementary polarisation information as input modality is proposed
+to overcome such limitations. This supervised approach is then extended to a
+self-supervised paradigm by leveraging physical characteristics of polarised
+light, thus eliminating the need for annotated real data. The methods achieve
+significant advancements in pose estimation by leveraging geometric information
+from polarised light and incorporating shape priors and invertible physical
+constraints.",cs.CV,['cs.CV']
+SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image,Yunhao Li · Xiaodong Wang · Ping Wang · Xin Yuan · Peidong Liu, ,https://arxiv.org/abs/2403.20018,,2403.20018.pdf,SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image,"In this paper, we explore the potential of Snapshot Compressive Imaging (SCI)
+technique for recovering the underlying 3D scene representation from a single
+temporal compressed image. SCI is a cost-effective method that enables the
+recording of high-dimensional data, such as hyperspectral or temporal
+information, into a single image using low-cost 2D imaging sensors. To achieve
+this, a series of specially designed 2D masks are usually employed, which not
+only reduces storage requirements but also offers potential privacy protection.
+Inspired by this, to take one step further, our approach builds upon the
+powerful 3D scene representation capabilities of neural radiance fields (NeRF).
+Specifically, we formulate the physical imaging process of SCI as part of the
+training of NeRF, allowing us to exploit its impressive performance in
+capturing complex scene structures. To assess the effectiveness of our method,
+we conduct extensive evaluations using both synthetic data and real data
+captured by our SCI system. Extensive experimental results demonstrate that our
+proposed approach surpasses the state-of-the-art methods in terms of image
+reconstruction and novel view image synthesis. Moreover, our method also
+exhibits the ability to restore high frame-rate multi-view consistent images by
+leveraging SCI and the rendering capabilities of NeRF. The code is available at
+https://github.com/WU-CVGL/SCINeRF.",eess.IV,"['eess.IV', 'cs.CV']"
+Source-Free Domain Adaptation with Frozen Multimodal Foundation Model,Song Tang · Wenxin Su · Mao Ye · Xiatian Zhu,https://www.taulab.cc/proj/sfda/cvpr24/difo/index.html,https://arxiv.org/abs/2311.16510,,2311.16510.pdf,Source-Free Domain Adaptation with Frozen Multimodal Foundation Model,"Source-Free Domain Adaptation (SFDA) aims to adapt a source model for a
+target domain, with only access to unlabeled target training data and the
+source model pre-trained on a supervised source domain. Relying on pseudo
+labeling and/or auxiliary supervision, conventional methods are inevitably
+error-prone. To mitigate this limitation, in this work we for the first time
+explore the potentials of off-the-shelf vision-language (ViL) multimodal models
+(e.g.,CLIP) with rich whilst heterogeneous knowledge. We find that directly
+applying the ViL model to the target domain in a zero-shot fashion is
+unsatisfactory, as it is not specialized for this particular task but largely
+generic. To make it task specific, we propose a novel Distilling multimodal
+Foundation model(DIFO)approach. Specifically, DIFO alternates between two steps
+during adaptation: (i) Customizing the ViL model by maximizing the mutual
+information with the target model in a prompt learning manner, (ii) Distilling
+the knowledge of this customized ViL model to the target model. For more
+fine-grained and reliable distillation, we further introduce two effective
+regularization terms, namely most-likely category encouragement and predictive
+consistency. Extensive experiments show that DIFO significantly outperforms the
+state-of-the-art alternatives. Code is here",cs.CV,['cs.CV']
+InstructVideo: Instructing Video Diffusion Models with Human Feedback,Hangjie Yuan · Shiwei Zhang · Xiang Wang · Yujie Wei · Tao Feng · Yining Pan · Yingya Zhang · Ziwei Liu · Samuel Albanie · Dong Ni, ,https://arxiv.org/abs/2312.12490,,2312.12490.pdf,InstructVideo: Instructing Video Diffusion Models with Human Feedback,"Diffusion models have emerged as the de facto paradigm for video generation.
+However, their reliance on web-scale data of varied quality often yields
+results that are visually unappealing and misaligned with the textual prompts.
+To tackle this problem, we propose InstructVideo to instruct text-to-video
+diffusion models with human feedback by reward fine-tuning. InstructVideo has
+two key ingredients: 1) To ameliorate the cost of reward fine-tuning induced by
+generating through the full DDIM sampling chain, we recast reward fine-tuning
+as editing. By leveraging the diffusion process to corrupt a sampled video,
+InstructVideo requires only partial inference of the DDIM sampling chain,
+reducing fine-tuning cost while improving fine-tuning efficiency. 2) To
+mitigate the absence of a dedicated video reward model for human preferences,
+we repurpose established image reward models, e.g., HPSv2. To this end, we
+propose Segmental Video Reward, a mechanism to provide reward signals based on
+segmental sparse sampling, and Temporally Attenuated Reward, a method that
+mitigates temporal modeling degradation during fine-tuning. Extensive
+experiments, both qualitative and quantitative, validate the practicality and
+efficacy of using image reward models in InstructVideo, significantly enhancing
+the visual quality of generated videos without compromising generalization
+capabilities. Code and models will be made publicly available.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+Stable Neighbor Denoising for Source-free Domain Adaptive Segmentation.,Dong Zhao · Shuang Wang · Qi Zang · Licheng Jiao · Nicu Sebe · Zhun Zhong, ,,,,,,,nan
+FREE: Faster and Better Data-Free Meta-Learning,Yongxian Wei · Zixuan Hu · Zhenyi Wang · Li Shen · Chun Yuan · Dacheng Tao, ,https://arxiv.org/abs/2405.00984,,2405.00984.pdf,FREE: Faster and Better Data-Free Meta-Learning,"Data-Free Meta-Learning (DFML) aims to extract knowledge from a collection of
+pre-trained models without requiring the original data, presenting practical
+benefits in contexts constrained by data privacy concerns. Current DFML methods
+primarily focus on the data recovery from these pre-trained models. However,
+they suffer from slow recovery speed and overlook gaps inherent in
+heterogeneous pre-trained models. In response to these challenges, we introduce
+the Faster and Better Data-Free Meta-Learning (FREE) framework, which contains:
+(i) a meta-generator for rapidly recovering training tasks from pre-trained
+models; and (ii) a meta-learner for generalizing to new unseen tasks.
+Specifically, within the module Faster Inversion via Meta-Generator, each
+pre-trained model is perceived as a distinct task. The meta-generator can
+rapidly adapt to a specific task in just five steps, significantly accelerating
+the data recovery. Furthermore, we propose Better Generalization via
+Meta-Learner and introduce an implicit gradient alignment algorithm to optimize
+the meta-learner. This is achieved as aligned gradient directions alleviate
+potential conflicts among tasks from heterogeneous pre-trained models.
+Empirical experiments on multiple benchmarks affirm the superiority of our
+approach, marking a notable speed-up (20$\times$) and performance enhancement
+(1.42\% $\sim$ 4.78\%) in comparison to the state-of-the-art.",cs.LG,"['cs.LG', 'cs.CV']"
+HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation,Ce Zhang · Simon Stepputtis · Joseph Campbell · Katia Sycara · Yaqi Xie,https://zhangce01.github.io/HiKER-SGG/,https://arxiv.org/abs/2403.12033,,2403.12033.pdf,HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation,"Being able to understand visual scenes is a precursor for many downstream
+tasks, including autonomous driving, robotics, and other vision-based
+approaches. A common approach enabling the ability to reason over visual data
+is Scene Graph Generation (SGG); however, many existing approaches assume
+undisturbed vision, i.e., the absence of real-world corruptions such as fog,
+snow, smoke, as well as non-uniform perturbations like sun glare or water
+drops. In this work, we propose a novel SGG benchmark containing procedurally
+generated weather corruptions and other transformations over the Visual Genome
+dataset. Further, we introduce a corresponding approach, Hierarchical Knowledge
+Enhanced Robust Scene Graph Generation (HiKER-SGG), providing a strong baseline
+for scene graph generation under such challenging setting. At its core,
+HiKER-SGG utilizes a hierarchical knowledge graph in order to refine its
+predictions from coarse initial estimates to detailed predictions. In our
+extensive experiments, we show that HiKER-SGG does not only demonstrate
+superior performance on corrupted images in a zero-shot manner, but also
+outperforms current state-of-the-art methods on uncorrupted SGG tasks. Code is
+available at https://github.com/zhangce01/HiKER-SGG.",cs.CV,['cs.CV']
+Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models,Xin Li · Yunfei Wu · Xinghua Jiang · ZhiHao Guo · Mingming Gong · Haoyu Cao · Yinsong Liu · Deqiang Jiang · Xing Sun, ,https://arxiv.org/abs/2402.19014,,2402.19014.pdf,Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models,"Recently, the advent of Large Visual-Language Models (LVLMs) has received
+increasing attention across various domains, particularly in the field of
+visual document understanding (VDU). Different from conventional
+vision-language tasks, VDU is specifically concerned with text-rich scenarios
+containing abundant document elements. Nevertheless, the importance of
+fine-grained features remains largely unexplored within the community of LVLMs,
+leading to suboptimal performance in text-rich scenarios. In this paper, we
+abbreviate it as the fine-grained feature collapse issue. With the aim of
+filling this gap, we propose a contrastive learning framework, termed Document
+Object COntrastive learning (DoCo), specifically tailored for the downstream
+tasks of VDU. DoCo leverages an auxiliary multimodal encoder to obtain the
+features of document objects and align them to the visual features generated by
+the vision encoder of LVLM, which enhances visual representation in text-rich
+scenarios. It can represent that the contrastive learning between the visual
+holistic representations and the multimodal fine-grained features of document
+objects can assist the vision encoder in acquiring more effective visual cues,
+thereby enhancing the comprehension of text-rich documents in LVLMs. We also
+demonstrate that the proposed DoCo serves as a plug-and-play pre-training
+method, which can be employed in the pre-training of various LVLMs without
+inducing any increase in computational complexity during the inference process.
+Extensive experimental results on multiple benchmarks of VDU reveal that LVLMs
+equipped with our proposed DoCo can achieve superior performance and mitigate
+the gap between VDU and generic vision-language tasks.",cs.CV,['cs.CV']
+PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF,Yutao Feng · Yintong Shang · Xuan Li · Tianjia Shao · Chenfanfu Jiang · Yin Yang,https://fytalon.github.io/pienerf/,https://arxiv.org/abs/2311.13099,,2311.13099.pdf,PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF,"We show that physics-based simulations can be seamlessly integrated with NeRF
+to generate high-quality elastodynamics of real-world objects. Unlike existing
+methods, we discretize nonlinear hyperelasticity in a meshless way, obviating
+the necessity for intermediate auxiliary shape proxies like a tetrahedral mesh
+or voxel grid. A quadratic generalized moving least square (Q-GMLS) is employed
+to capture nonlinear dynamics and large deformation on the implicit model. Such
+meshless integration enables versatile simulations of complex and codimensional
+shapes. We adaptively place the least-square kernels according to the NeRF
+density field to significantly reduce the complexity of the nonlinear
+simulation. As a result, physically realistic animations can be conveniently
+synthesized using our method for a wide range of hyperelastic materials at an
+interactive rate. For more information, please visit our project page at
+https://fytalon.github.io/pienerf/.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling,Linqi Zhou · Andy Shih · Chenlin Meng · Stefano Ermon, ,https://arxiv.org/abs/2311.17082,,2311.17082.pdf,DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling,"Recent methods such as Score Distillation Sampling (SDS) and Variational
+Score Distillation (VSD) using 2D diffusion models for text-to-3D generation
+have demonstrated impressive generation quality. However, the long generation
+time of such algorithms significantly degrades the user experience. To tackle
+this problem, we propose DreamPropeller, a drop-in acceleration algorithm that
+can be wrapped around any existing text-to-3D generation pipeline based on
+score distillation. Our framework generalizes Picard iterations, a classical
+algorithm for parallel sampling an ODE path, and can account for non-ODE paths
+such as momentum-based gradient updates and changes in dimensions during the
+optimization process as in many cases of 3D generation. We show that our
+algorithm trades parallel compute for wallclock time and empirically achieves
+up to 4.7x speedup with a negligible drop in generation quality for all tested
+frameworks.",cs.CV,"['cs.CV', 'stat.ML']"
+RepViT: Revisiting Mobile CNN From ViT Perspective,Ao Wang · Hui Chen · Zijia Lin · Jungong Han · Guiguang Ding,https://github.com/THU-MIG/RepViT,https://arxiv.org/abs/2307.09283,,2307.09283.pdf,RepViT: Revisiting Mobile CNN From ViT Perspective,"Recently, lightweight Vision Transformers (ViTs) demonstrate superior
+performance and lower latency, compared with lightweight Convolutional Neural
+Networks (CNNs), on resource-constrained mobile devices. Researchers have
+discovered many structural connections between lightweight ViTs and lightweight
+CNNs. However, the notable architectural disparities in the block structure,
+macro, and micro designs between them have not been adequately examined. In
+this study, we revisit the efficient design of lightweight CNNs from ViT
+perspective and emphasize their promising prospect for mobile devices.
+Specifically, we incrementally enhance the mobile-friendliness of a standard
+lightweight CNN, \ie, MobileNetV3, by integrating the efficient architectural
+designs of lightweight ViTs. This ends up with a new family of pure lightweight
+CNNs, namely RepViT. Extensive experiments show that RepViT outperforms
+existing state-of-the-art lightweight ViTs and exhibits favorable latency in
+various vision tasks. Notably, on ImageNet, RepViT achieves over 80\% top-1
+accuracy with 1.0 ms latency on an iPhone 12, which is the first time for a
+lightweight model, to the best of our knowledge. Besides, when RepViT meets
+SAM, our RepViT-SAM can achieve nearly 10$\times$ faster inference than the
+advanced MobileSAM. Codes and models are available at
+\url{https://github.com/THU-MIG/RepViT}.",cs.CV,['cs.CV']
+Neural Video Compression with Feature Modulation,Jiahao Li · Bin Li · Yan Lu, ,https://arxiv.org/abs/2402.17414v1,,2402.17414v1.pdf,Neural Video Compression with Feature Modulation,"The emerging conditional coding-based neural video codec (NVC) shows
+superiority over commonly-used residual coding-based codec and the latest NVC
+already claims to outperform the best traditional codec. However, there still
+exist critical problems blocking the practicality of NVC. In this paper, we
+propose a powerful conditional coding-based NVC that solves two critical
+problems via feature modulation. The first is how to support a wide quality
+range in a single model. Previous NVC with this capability only supports about
+3.8 dB PSNR range on average. To tackle this limitation, we modulate the latent
+feature of the current frame via the learnable quantization scaler. During the
+training, we specially design the uniform quantization parameter sampling
+mechanism to improve the harmonization of encoding and quantization. This
+results in a better learning of the quantization scaler and helps our NVC
+support about 11.4 dB PSNR range. The second is how to make NVC still work
+under a long prediction chain. We expose that the previous SOTA NVC has an
+obvious quality degradation problem when using a large intra-period setting. To
+this end, we propose modulating the temporal feature with a periodically
+refreshing mechanism to boost the quality. %Besides solving the above two
+problems, we also design a single model that can support both RGB and YUV
+colorspaces. Notably, under single intra-frame setting, our codec can achieve
+29.7\% bitrate saving over previous SOTA NVC with 16\% MACs reduction. Our
+codec serves as a notable landmark in the journey of NVC evolution. The codes
+are at https://github.com/microsoft/DCVC.",cs.CV,"['cs.CV', 'eess.IV']"
+Meta-Point Learning and Refining for Category-Agnostic Pose Estimation,Junjie Chen · Jiebin Yan · Yuming Fang · Li Niu, ,https://arxiv.org/abs/2403.13647,,2403.13647.pdf,Meta-Point Learning and Refining for Category-Agnostic Pose Estimation,"Category-agnostic pose estimation (CAPE) aims to predict keypoints for
+arbitrary classes given a few support images annotated with keypoints. Existing
+methods only rely on the features extracted at support keypoints to predict or
+refine the keypoints on query image, but a few support feature vectors are
+local and inadequate for CAPE. Considering that human can quickly perceive
+potential keypoints of arbitrary objects, we propose a novel framework for CAPE
+based on such potential keypoints (named as meta-points). Specifically, we
+maintain learnable embeddings to capture inherent information of various
+keypoints, which interact with image feature maps to produce meta-points
+without any support. The produced meta-points could serve as meaningful
+potential keypoints for CAPE. Due to the inevitable gap between inherency and
+annotation, we finally utilize the identities and details offered by support
+keypoints to assign and refine meta-points to desired keypoints in query image.
+In addition, we propose a progressive deformable point decoder and a slacked
+regression loss for better prediction and supervision. Our novel framework not
+only reveals the inherency of keypoints but also outperforms existing methods
+of CAPE. Comprehensive experiments and in-depth studies on large-scale MP-100
+dataset demonstrate the effectiveness of our framework.",cs.CV,['cs.CV']
+Real-World Efficient Blind Motion Deblurring via Blur Pixel Discretization,Insoo Kim · Jae Seok Choi · Geonseok Seo · Kinam Kwon · Jinwoo Shin · Hyong-Euk Lee, ,https://arxiv.org/abs/2404.12168,,2404.12168.pdf,Real-World Efficient Blind Motion Deblurring via Blur Pixel Discretization,"As recent advances in mobile camera technology have enabled the capability to
+capture high-resolution images, such as 4K images, the demand for an efficient
+deblurring model handling large motion has increased. In this paper, we
+discover that the image residual errors, i.e., blur-sharp pixel differences,
+can be grouped into some categories according to their motion blur type and how
+complex their neighboring pixels are. Inspired by this, we decompose the
+deblurring (regression) task into blur pixel discretization (pixel-level blur
+classification) and discrete-to-continuous conversion (regression with blur
+class map) tasks. Specifically, we generate the discretized image residual
+errors by identifying the blur pixels and then transform them to a continuous
+form, which is computationally more efficient than naively solving the original
+regression problem with continuous values. Here, we found that the
+discretization result, i.e., blur segmentation map, remarkably exhibits visual
+similarity with the image residual errors. As a result, our efficient model
+shows comparable performance to state-of-the-art methods in realistic
+benchmarks, while our method is up to 10 times computationally more efficient.",cs.CV,"['cs.CV', 'cs.AI']"
+Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention,Xingyu Zhou · Leheng Zhang · Xiaorui Zhao · Keze Wang · Leida Li · Shuhang Gu, ,https://arxiv.org/abs/2401.06312,,2401.06312.pdf,Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention,"Recently, Vision Transformer has achieved great success in recovering missing
+details in low-resolution sequences, i.e., the video super-resolution (VSR)
+task. Despite its superiority in VSR accuracy, the heavy computational burden
+as well as the large memory footprint hinder the deployment of
+Transformer-based VSR models on constrained devices. In this paper, we address
+the above issue by proposing a novel feature-level masked processing framework:
+VSR with Masked Intra and inter frame Attention (MIA-VSR). The core of MIA-VSR
+is leveraging feature-level temporal continuity between adjacent frames to
+reduce redundant computations and make more rational use of previously enhanced
+SR features. Concretely, we propose an intra-frame and inter-frame attention
+block which takes the respective roles of past features and input features into
+consideration and only exploits previously enhanced features to provide
+supplementary information. In addition, an adaptive block-wise mask prediction
+module is developed to skip unimportant computations according to feature
+similarity between adjacent frames. We conduct detailed ablation studies to
+validate our contributions and compare the proposed method with recent
+state-of-the-art VSR approaches. The experimental results demonstrate that
+MIA-VSR improves the memory and computation efficiency over state-of-the-art
+methods, without trading off PSNR accuracy. The code is available at
+https://github.com/LabShuHangGU/MIA-VSR.",cs.CV,['cs.CV']
+Any-Shift Prompting for Generalization over Distributions,Zehao Xiao · Jiayi Shen · Mohammad Mahdi Derakhshani · Shengcai Liao · Cees G. M. Snoek, ,https://arxiv.org/abs/2402.10099,,2402.10099.pdf,Any-Shift Prompting for Generalization over Distributions,"Image-language models with prompt learning have shown remarkable advances in
+numerous downstream vision tasks. Nevertheless, conventional prompt learning
+methods overfit their training distribution and lose the generalization ability
+on test distributions. To improve generalization across various distribution
+shifts, we propose any-shift prompting: a general probabilistic inference
+framework that considers the relationship between training and test
+distributions during prompt learning. We explicitly connect training and test
+distributions in the latent space by constructing training and test prompts in
+a hierarchical architecture. Within this framework, the test prompt exploits
+the distribution relationships to guide the generalization of the CLIP
+image-language model from training to any test distribution. To effectively
+encode the distribution information and their relationships, we further
+introduce a transformer inference network with a pseudo-shift training
+mechanism. The network generates the tailored test prompt with both training
+and test information in a feedforward pass, avoiding extra training costs at
+test time. Extensive experiments on twenty-three datasets demonstrate the
+effectiveness of any-shift prompting on the generalization over various
+distribution shifts.",cs.CV,['cs.CV']
+Mosaic-SDF for 3D Generative Models,Lior Yariv · Omri Puny · Oran Gafni · Yaron Lipman,https://lioryariv.github.io/msdf/,https://arxiv.org/abs/2312.09222,,2312.09222.pdf,Mosaic-SDF for 3D Generative Models,"Current diffusion or flow-based generative models for 3D shapes divide to
+two: distilling pre-trained 2D image diffusion models, and training directly on
+3D shapes. When training a diffusion or flow models on 3D shapes a crucial
+design choice is the shape representation. An effective shape representation
+needs to adhere three design principles: it should allow an efficient
+conversion of large 3D datasets to the representation form; it should provide a
+good tradeoff of approximation power versus number of parameters; and it should
+have a simple tensorial form that is compatible with existing powerful neural
+architectures. While standard 3D shape representations such as volumetric grids
+and point clouds do not adhere to all these principles simultaneously, we
+advocate in this paper a new representation that does. We introduce Mosaic-SDF
+(M-SDF): a simple 3D shape representation that approximates the Signed Distance
+Function (SDF) of a given shape by using a set of local grids spread near the
+shape's boundary. The M-SDF representation is fast to compute for each shape
+individually making it readily parallelizable; it is parameter efficient as it
+only covers the space around the shape's boundary; and it has a simple matrix
+form, compatible with Transformer-based architectures. We demonstrate the
+efficacy of the M-SDF representation by using it to train a 3D generative flow
+model including class-conditioned generation with the 3D Warehouse dataset, and
+text-to-3D generation using a dataset of about 600k caption-shape pairs.",cs.CV,"['cs.CV', 'cs.GR']"
+Fourier-basis functions to bridge augmentation gap: Rethinking frequency augmentation in image classification,Mei Vaish · Shunxin Wang · Nicola Strisciuglio,https://github.com/nis-research/afa-augment,https://arxiv.org/abs/2403.01944,,2403.01944.pdf,Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification,"Computer vision models normally witness degraded performance when deployed in
+real-world scenarios, due to unexpected changes in inputs that were not
+accounted for during training. Data augmentation is commonly used to address
+this issue, as it aims to increase data variety and reduce the distribution gap
+between training and test data. However, common visual augmentations might not
+guarantee extensive robustness of computer vision models. In this paper, we
+propose Auxiliary Fourier-basis Augmentation (AFA), a complementary technique
+targeting augmentation in the frequency domain and filling the augmentation gap
+left by visual augmentations. We demonstrate the utility of augmentation via
+Fourier-basis additive noise in a straightforward and efficient adversarial
+setting. Our results show that AFA benefits the robustness of models against
+common corruptions, OOD generalization, and consistency of performance of
+models against increasing perturbations, with negligible deficit to the
+standard performance of models. It can be seamlessly integrated with other
+augmentation techniques to further boost performance. Code and models can be
+found at: https://github.com/nis-research/afa-augment",cs.CV,"['cs.CV', 'cs.LG']"
+CDMAD: Class-Distribution-Mismatch-Aware Debiasing for Class-Imbalanced Semi-Supervised Learning,Hyuck Lee · Heeyoung Kim, ,https://arxiv.org/abs/2403.10391,,2403.10391.pdf,CDMAD: Class-Distribution-Mismatch-Aware Debiasing for Class-Imbalanced Semi-Supervised Learning,"Pseudo-label-based semi-supervised learning (SSL) algorithms trained on a
+class-imbalanced set face two cascading challenges: 1) Classifiers tend to be
+biased towards majority classes, and 2) Biased pseudo-labels are used for
+training. It is difficult to appropriately re-balance the classifiers in SSL
+because the class distribution of an unlabeled set is often unknown and could
+be mismatched with that of a labeled set. We propose a novel class-imbalanced
+SSL algorithm called class-distribution-mismatch-aware debiasing (CDMAD). For
+each iteration of training, CDMAD first assesses the classifier's biased degree
+towards each class by calculating the logits on an image without any patterns
+(e.g., solid color image), which can be considered irrelevant to the training
+set. CDMAD then refines biased pseudo-labels of the base SSL algorithm by
+ensuring the classifier's neutrality. CDMAD uses these refined pseudo-labels
+during the training of the base SSL algorithm to improve the quality of the
+representations. In the test phase, CDMAD similarly refines biased class
+predictions on test samples. CDMAD can be seen as an extension of post-hoc
+logit adjustment to address a challenge of incorporating the unknown class
+distribution of the unlabeled set for re-balancing the biased classifier under
+class distribution mismatch. CDMAD ensures Fisher consistency for the balanced
+error. Extensive experiments verify the effectiveness of CDMAD.",cs.CV,['cs.CV']
+LoS: Local Structure Guided Stereo Matching,Kunhong Li · Longguang Wang · Ye Zhang · Kaiwen Xue · Shunbo Zhou · Yulan Guo, ,https://ar5iv.labs.arxiv.org/html/2309.16992,,2309.16992.pdf,Segment Anything Model is a Good Teacher for Local Feature Learning,"Local feature detection and description play an important role in many
+computer vision tasks, which are designed to detect and describe keypoints in
+""any scene"" and ""any downstream task"". Data-driven local feature learning
+methods need to rely on pixel-level correspondence for training, which is
+challenging to acquire at scale, thus hindering further improvements in
+performance. In this paper, we propose SAMFeat to introduce SAM (segment
+anything model), a fundamental model trained on 11 million images, as a teacher
+to guide local feature learning and thus inspire higher performance on limited
+datasets. To do so, first, we construct an auxiliary task of Pixel Semantic
+Relational Distillation (PSRD), which distillates feature relations with
+category-agnostic semantic information learned by the SAM encoder into a local
+feature learning network, to improve local feature description using semantic
+discrimination. Second, we develop a technique called Weakly Supervised
+Contrastive Learning Based on Semantic Grouping (WSC), which utilizes semantic
+groupings derived from SAM as weakly supervised signals, to optimize the metric
+space of local descriptors. Third, we design an Edge Attention Guidance (EAG)
+to further improve the accuracy of local feature detection and description by
+prompting the network to pay more attention to the edge region guided by SAM.
+SAMFeat's performance on various tasks such as image matching on HPatches, and
+long-term visual localization on Aachen Day-Night showcases its superiority
+over previous local features. The release code is available at
+https://github.com/vignywang/SAMFeat.",cs.CV,"['cs.CV', 'cs.LG']"
+Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data,Xinting Liao · Weiming Liu · Chaochao Chen · Pengyang Zhou · Fengyuan Yu · Huabin Zhu · Binhui Yao · Tao Wang · Xiaolin Zheng · Yanchao Tan, ,https://arxiv.org/abs/2403.16398,,2403.16398.pdf,Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data,"Federated learning achieves effective performance in modeling decentralized
+data. In practice, client data are not well-labeled, which makes it potential
+for federated unsupervised learning (FUSL) with non-IID data. However, the
+performance of existing FUSL methods suffers from insufficient representations,
+i.e., (1) representation collapse entanglement among local and global models,
+and (2) inconsistent representation spaces among local models. The former
+indicates that representation collapse in local model will subsequently impact
+the global model and other local models. The latter means that clients model
+data representation with inconsistent parameters due to the deficiency of
+supervision signals. In this work, we propose FedU2 which enhances generating
+uniform and unified representation in FUSL with non-IID data. Specifically,
+FedU2 consists of flexible uniform regularizer (FUR) and efficient unified
+aggregator (EUA). FUR in each client avoids representation collapse via
+dispersing samples uniformly, and EUA in server promotes unified representation
+by constraining consistent client model updating. To extensively validate the
+performance of FedU2, we conduct both cross-device and cross-silo evaluation
+experiments on two benchmark datasets, i.e., CIFAR10 and CIFAR100.",cs.LG,"['cs.LG', 'cs.AI']"
+Towards Detailed and Robust 3D Clothed Human Reconstruction with High-Frequency and Low-Frequency Information of Parametric Body Models,Yifan Yang · Dong Liu · Shuhai Zhang · Zeshuai Deng · Zixiong Huang · Mingkui Tan, ,https://arxiv.org/abs/2404.04876,,2404.04876.pdf,HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Models,"Reconstructing 3D clothed human involves creating a detailed geometry of
+individuals in clothing, with applications ranging from virtual try-on, movies,
+to games. To enable practical and widespread applications, recent advances
+propose to generate a clothed human from an RGB image. However, they struggle
+to reconstruct detailed and robust avatars simultaneously. We empirically find
+that the high-frequency (HF) and low-frequency (LF) information from a
+parametric model has the potential to enhance geometry details and improve
+robustness to noise, respectively. Based on this, we propose HiLo, namely
+clothed human reconstruction with high- and low-frequency information, which
+contains two components. 1) To recover detailed geometry using HF information,
+we propose a progressive HF Signed Distance Function to enhance the detailed 3D
+geometry of a clothed human. We analyze that our progressive learning manner
+alleviates large gradients that hinder model convergence. 2) To achieve robust
+reconstruction against inaccurate estimation of the parametric model by using
+LF information, we propose a spatial interaction implicit function. This
+function effectively exploits the complementary spatial information from a
+low-resolution voxel grid of the parametric model. Experimental results
+demonstrate that HiLo outperforms the state-of-the-art methods by 10.43% and
+9.54% in terms of Chamfer distance on the Thuman2.0 and CAPE datasets,
+respectively. Additionally, HiLo demonstrates robustness to noise from the
+parametric model, challenging poses, and various clothing styles.",cs.CV,['cs.CV']
+MS-DETR: Efficient DETR Training  with Mixed Supervision,Chuyang Zhao · Yifan Sun · Wenhao Wang · Qiang Chen · Errui Ding · Yi Yang · Jingdong Wang,https://github.com/Atten4Vis/MS-DETR,https://arxiv.org/abs/2401.03989,,2401.03989.pdf,MS-DETR: Efficient DETR Training with Mixed Supervision,"DETR accomplishes end-to-end object detection through iteratively generating
+multiple object candidates based on image features and promoting one candidate
+for each ground-truth object. The traditional training procedure using
+one-to-one supervision in the original DETR lacks direct supervision for the
+object detection candidates.
+  We aim at improving the DETR training efficiency by explicitly supervising
+the candidate generation procedure through mixing one-to-one supervision and
+one-to-many supervision. Our approach, namely MS-DETR, is simple, and places
+one-to-many supervision to the object queries of the primary decoder that is
+used for inference. In comparison to existing DETR variants with one-to-many
+supervision, such as Group DETR and Hybrid DETR, our approach does not need
+additional decoder branches or object queries. The object queries of the
+primary decoder in our approach directly benefit from one-to-many supervision
+and thus are superior in object candidate prediction. Experimental results show
+that our approach outperforms related DETR variants, such as DN-DETR, Hybrid
+DETR, and Group DETR, and the combination with related DETR variants further
+improves the performance.",cs.CV,['cs.CV']
+Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models,Yabin Zhang · Wenjie Zhu · Hui Tang · Zhiyuan Ma · Kaiyang Zhou · Lei Zhang,https://github.com/YBZh/DMN,https://arxiv.org/abs/2403.17589,,2403.17589.pdf,Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models,"With the emergence of pre-trained vision-language models like CLIP, how to
+adapt them to various downstream classification tasks has garnered significant
+attention in recent research. The adaptation strategies can be typically
+categorized into three paradigms: zero-shot adaptation, few-shot adaptation,
+and the recently-proposed training-free few-shot adaptation. Most existing
+approaches are tailored for a specific setting and can only cater to one or two
+of these paradigms. In this paper, we introduce a versatile adaptation approach
+that can effectively work under all three settings. Specifically, we propose
+the dual memory networks that comprise dynamic and static memory components.
+The static memory caches training data knowledge, enabling training-free
+few-shot adaptation, while the dynamic memory preserves historical test
+features online during the testing process, allowing for the exploration of
+additional data insights beyond the training set. This novel capability
+enhances model performance in the few-shot setting and enables model usability
+in the absence of training data. The two memory networks employ the same
+flexible memory interactive strategy, which can operate in a training-free mode
+and can be further enhanced by incorporating learnable projection layers. Our
+approach is tested across 11 datasets under the three task settings.
+Remarkably, in the zero-shot scenario, it outperforms existing methods by over
+3\% and even shows superior results against methods utilizing external training
+data. Additionally, our method exhibits robust performance against natural
+distribution shifts. Codes are available at \url{https://github.com/YBZh/DMN}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+Fully Convolutional Slice-to-Volume Reconstruction for Single-Stack MRI,Sean I. Young · Yaël Balbastre · Bruce Fischl · Polina Golland · Juan Iglesias, ,https://arxiv.org/abs/2312.03102,,2312.03102.pdf,Fully Convolutional Slice-to-Volume Reconstruction for Single-Stack MRI,"In magnetic resonance imaging (MRI), slice-to-volume reconstruction (SVR)
+refers to computational reconstruction of an unknown 3D magnetic resonance
+volume from stacks of 2D slices corrupted by motion. While promising, current
+SVR methods require multiple slice stacks for accurate 3D reconstruction,
+leading to long scans and limiting their use in time-sensitive applications
+such as fetal fMRI. Here, we propose a SVR method that overcomes the
+shortcomings of previous work and produces state-of-the-art reconstructions in
+the presence of extreme inter-slice motion. Inspired by the recent success of
+single-view depth estimation methods, we formulate SVR as a single-stack motion
+estimation task and train a fully convolutional network to predict a motion
+stack for a given slice stack, producing a 3D reconstruction as a byproduct of
+the predicted motion. Extensive experiments on the SVR of adult and fetal
+brains demonstrate that our fully convolutional method is twice as accurate as
+previous SVR methods. Our code is available at github.com/seannz/svr.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG']"
+Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers,Subhadeep Koley · Ayan Kumar Bhunia · Aneeshan Sain · Pinaki Nath Chowdhury · Tao Xiang · Yi-Zhe Song,https://subhadeepkoley.github.io/DiffusionZSSBIR/,https://arxiv.org/abs/2403.07214,,2403.07214.pdf,Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers,"This paper, for the first time, explores text-to-image diffusion models for
+Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal
+discovery: the capacity of text-to-image diffusion models to seamlessly bridge
+the gap between sketches and photos. This proficiency is underpinned by their
+robust cross-modal capabilities and shape bias, findings that are substantiated
+through our pilot studies. In order to harness pre-trained diffusion models
+effectively, we introduce a straightforward yet powerful strategy focused on
+two key aspects: selecting optimal feature layers and utilising visual and
+textual prompts. For the former, we identify which layers are most enriched
+with information and are best suited for the specific retrieval requirements
+(category-level or fine-grained). Then we employ visual and textual prompts to
+guide the model's feature extraction process, enabling it to generate more
+discriminative and contextually relevant cross-modal representations. Extensive
+experiments on several benchmark datasets validate significant performance
+improvements.",cs.CV,['cs.CV']
+Enhance Image Classification Via Inter-Class Image Mixup With Diffusion Model,Zhicai Wang · Longhui Wei · Tan Wang · Heyu Chen · Yanbin Hao · Xiang Wang · Xiangnan He · Qi Tian, ,https://arxiv.org/abs/2403.19600,,2403.19600.pdf,Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model,"Text-to-image (T2I) generative models have recently emerged as a powerful
+tool, enabling the creation of photo-realistic images and giving rise to a
+multitude of applications. However, the effective integration of T2I models
+into fundamental image classification tasks remains an open question. A
+prevalent strategy to bolster image classification performance is through
+augmenting the training set with synthetic images generated by T2I models. In
+this study, we scrutinize the shortcomings of both current generative and
+conventional data augmentation techniques. Our analysis reveals that these
+methods struggle to produce images that are both faithful (in terms of
+foreground objects) and diverse (in terms of background contexts) for
+domain-specific concepts. To tackle this challenge, we introduce an innovative
+inter-class data augmentation method known as Diff-Mix
+(https://github.com/Zhicaiwww/Diff-Mix), which enriches the dataset by
+performing image translations between classes. Our empirical results
+demonstrate that Diff-Mix achieves a better balance between faithfulness and
+diversity, leading to a marked improvement in performance across diverse image
+classification scenarios, including few-shot, conventional, and long-tail
+classifications for domain-specific datasets.",cs.CV,['cs.CV']
+Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata,Dongsu Zhang · Francis Williams · Žan Gojčič · Karsten Kreis · Sanja Fidler · Young Min Kim · Amlan Kar, ,,https://www.tandfonline.com/doi/full/10.1080/15481603.2023.2290352,,,,,nan
+3D Neural Edge Reconstruction,Lei Li · Songyou Peng · Zehao Yu · Shaohui Liu · Rémi Pautrat · Xiaochuan Yin · Marc Pollefeys,https://neural-edge-map.github.io/,https://arxiv.org/abs/2405.19295,,2405.19295.pdf,3D Neural Edge Reconstruction,"Real-world objects and environments are predominantly composed of edge
+features, including straight lines and curves. Such edges are crucial elements
+for various applications, such as CAD modeling, surface meshing, lane mapping,
+etc. However, existing traditional methods only prioritize lines over curves
+for simplicity in geometric modeling. To this end, we introduce EMAP, a new
+method for learning 3D edge representations with a focus on both lines and
+curves. Our method implicitly encodes 3D edge distance and direction in
+Unsigned Distance Functions (UDF) from multi-view edge maps. On top of this
+neural representation, we propose an edge extraction algorithm that robustly
+abstracts parametric 3D edges from the inferred edge points and their
+directions. Comprehensive evaluations demonstrate that our method achieves
+better 3D edge reconstruction on multiple challenging datasets. We further show
+that our learned UDF field enhances neural surface reconstruction by capturing
+more details.",cs.CV,['cs.CV']
+ProMark: Proactive Diffusion Watermarking for Causal Attribution,Vishal Asnani · John Collomosse · Tu Bui · Xiaoming Liu · Shruti Agarwal, ,https://arxiv.org/abs/2403.09914,,2403.09914.pdf,ProMark: Proactive Diffusion Watermarking for Causal Attribution,"Generative AI (GenAI) is transforming creative workflows through the
+capability to synthesize and manipulate images via high-level prompts. Yet
+creatives are not well supported to receive recognition or reward for the use
+of their content in GenAI training. To this end, we propose ProMark, a causal
+attribution technique to attribute a synthetically generated image to its
+training data concepts like objects, motifs, templates, artists, or styles. The
+concept information is proactively embedded into the input training images
+using imperceptible watermarks, and the diffusion models (unconditional or
+conditional) are trained to retain the corresponding watermarks in generated
+images. We show that we can embed as many as $2^{16}$ unique watermarks into
+the training data, and each training image can contain more than one watermark.
+ProMark can maintain image quality whilst outperforming correlation-based
+attribution. Finally, several qualitative examples are presented, providing the
+confidence that the presence of the watermark conveys a causative relationship
+between training data and synthetic images.",cs.CV,['cs.CV']
+Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior,Chen Cheng · Xiaofeng Yang · Fan Yang · Chengzeng Feng · ZHOUJIE FU · Chuan-Sheng Foo · Guosheng Lin · Fayao Liu, ,https://arxiv.org/abs/2403.09140,,2403.09140.pdf,Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior,"Recent works on text-to-3d generation show that using only 2D diffusion
+supervision for 3D generation tends to produce results with inconsistent
+appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals
+with extra legs). Existing methods mainly address this issue by retraining
+diffusion models with images rendered from 3D data to ensure multi-view
+consistency while struggling to balance 2D generation quality with 3D
+consistency. In this paper, we present a new framework Sculpt3D that equips the
+current pipeline with explicit injection of 3D priors from retrieved reference
+objects without re-training the 2D diffusion model. Specifically, we
+demonstrate that high-quality and diverse 3D geometry can be guaranteed by
+keypoints supervision through a sparse ray sampling approach. Moreover, to
+ensure accurate appearances of different views, we further modulate the output
+of the 2D diffusion model to the correct patterns of the template views without
+altering the generated object's style. These two decoupled designs effectively
+harness 3D information from reference objects to generate 3D objects while
+preserving the generation quality of the 2D diffusion model. Extensive
+experiments show our method can largely improve the multi-view consistency
+while retaining fidelity and diversity. Our project page is available at:
+https://stellarcheng.github.io/Sculpt3D/.",cs.CV,['cs.CV']
+Empowering Resampling Operation for Ultra-High-Definition Image Enhancement with Model-Aware Guidance,Yu · Jie Huang · Li · Kaiwen Zheng · Qi Zhu · Man Zhou · Feng Zhao, ,,https://github.com/YPatrickW/LMAR,,,,,nan
+You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval,Subhadeep Koley · Ayan Kumar Bhunia · Aneeshan Sain · Pinaki Nath Chowdhury · Tao Xiang · Yi-Zhe Song,https://subhadeepkoley.github.io/Sketch2Word/,https://arxiv.org/abs/2403.07222v2,,2403.07222v2.pdf,You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval,"Two primary input modalities prevail in image retrieval: sketch and text.
+While text is widely used for inter-category retrieval tasks, sketches have
+been established as the sole preferred modality for fine-grained image
+retrieval due to their ability to capture intricate visual details. In this
+paper, we question the reliance on sketches alone for fine-grained image
+retrieval by simultaneously exploring the fine-grained representation
+capabilities of both sketch and text, orchestrating a duet between the two. The
+end result enables precise retrievals previously unattainable, allowing users
+to pose ever-finer queries and incorporate attributes like colour and
+contextual cues from text. For this purpose, we introduce a novel
+compositionality framework, effectively combining sketches and text using
+pre-trained CLIP models, while eliminating the need for extensive fine-grained
+textual descriptions. Last but not least, our system extends to novel
+applications in composed image retrieval, domain attribute transfer, and
+fine-grained generation, providing solutions for various real-world scenarios.",cs.CV,['cs.CV']
+Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos,Kumaranage Ravindu Nagasinghe · Honglu Zhou · Malitha Gunawardhana · Martin Renqiang Min · Daniel Harari · Muhammad Haris Khan,https://ravindu-yasas-nagasinghe.github.io/KEPP-Project_Page/,https://arxiv.org/abs/2403.02782,,2403.02782.pdf,Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos,"In this paper, we explore the capability of an agent to construct a logical
+sequence of action steps, thereby assembling a strategic procedural plan. This
+plan is crucial for navigating from an initial visual observation to a target
+visual outcome, as depicted in real-life instructional videos. Existing works
+have attained partial success by extensively leveraging various sources of
+information available in the datasets, such as heavy intermediate visual
+observations, procedural names, or natural language step-by-step instructions,
+for features or supervision signals. However, the task remains formidable due
+to the implicit causal constraints in the sequencing of steps and the
+variability inherent in multiple feasible plans. To tackle these intricacies
+that previous efforts have overlooked, we propose to enhance the capabilities
+of the agent by infusing it with procedural knowledge. This knowledge, sourced
+from training procedure plans and structured as a directed weighted graph,
+equips the agent to better navigate the complexities of step sequencing and its
+potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced
+Procedure Planning system, which harnesses a probabilistic procedural knowledge
+graph extracted from training data, effectively acting as a comprehensive
+textbook for the training domain. Experimental evaluations across three
+widely-used datasets under settings of varying complexity reveal that KEPP
+attains superior, state-of-the-art results while requiring only minimal
+supervision.",cs.CV,['cs.CV']
+Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes,Ziqian Bai · Feitong Tan · Sean Fanello · Rohit Pandey · Mingsong Dou · Shichen Liu · Ping Tan · Yinda Zhang,https://augmentedperception.github.io/monoavatar-plus/,https://arxiv.org/abs/2404.01543,,2404.01543.pdf,Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes,"3D head avatars built with neural implicit volumetric representations have
+achieved unprecedented levels of photorealism. However, the computational cost
+of these methods remains a significant barrier to their widespread adoption,
+particularly in real-time applications such as virtual reality and
+teleconferencing. While attempts have been made to develop fast neural
+rendering approaches for static scenes, these methods cannot be simply employed
+to support realistic facial expressions, such as in the case of a dynamic
+facial performance. To address these challenges, we propose a novel fast 3D
+neural implicit head avatar model that achieves real-time rendering while
+maintaining fine-grained controllability and high rendering quality. Our key
+idea lies in the introduction of local hash table blendshapes, which are
+learned and attached to the vertices of an underlying face parametric model.
+These per-vertex hash-tables are linearly merged with weights predicted via a
+CNN, resulting in expression dependent embeddings. Our novel representation
+enables efficient density and color predictions using a lightweight MLP, which
+is further accelerated by a hierarchical nearest neighbor search method.
+Extensive experiments show that our approach runs in real-time while achieving
+comparable rendering quality to state-of-the-arts and decent results on
+challenging expressions.",cs.CV,"['cs.CV', 'cs.GR']"
+"Towards Co-Evaluation of Cameras, HDR, and Algorithms for Industrial-Grade 6DoF Pose Estimation",Agastya Kalra · Guy Stoppi · Dmitrii Marin · Vage Taamazyan · Aarrushi Shandilya · Rishav Agarwal · Anton Boykov · Aaron Chong · Michael Stark,https://github.com/intrinsic-ai/ipd,https://arxiv.org/abs/2403.03221,,2403.03221.pdf,"FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation","Estimating relative camera poses between images has been a central problem in
+computer vision. Methods that find correspondences and solve for the
+fundamental matrix offer high precision in most cases. Conversely, methods
+predicting pose directly using neural networks are more robust to limited
+overlap and can infer absolute translation scale, but at the expense of reduced
+precision. We show how to combine the best of both methods; our approach yields
+results that are both precise and robust, while also accurately inferring
+translation scales. At the heart of our model lies a Transformer that (1)
+learns to balance between solved and learned pose estimations, and (2) provides
+a prior to guide a solver. A comprehensive analysis supports our design choices
+and demonstrates that our method adapts flexibly to various feature extractors
+and correspondence estimators, showing state-of-the-art performance in 6DoF
+pose estimation on Matterport3D, InteriorNet, StreetLearn, and Map-free
+Relocalization.",cs.CV,['cs.CV']
+A Generative Approach for Wikipedia-Scale Visual Entity Recognition,Mathilde Caron · Ahmet Iscen · Alireza Fathi · Cordelia Schmid,https://github.com/google-research/scenic/tree/main/scenic/projects/gerald,https://arxiv.org/abs/2403.02041,,2403.02041.pdf,A Generative Approach for Wikipedia-Scale Visual Entity Recognition,"In this paper, we address web-scale visual entity recognition, specifically
+the task of mapping a given query image to one of the 6 million existing
+entities in Wikipedia. One way of approaching a problem of such scale is using
+dual-encoder models (eg CLIP), where all the entity names and query images are
+embedded into a unified space, paving the way for an approximate k-NN search.
+Alternatively, it is also possible to re-purpose a captioning model to directly
+generate the entity names for a given image. In contrast, we introduce a novel
+Generative Entity Recognition (GER) framework, which given an input image
+learns to auto-regressively decode a semantic and discriminative ``code''
+identifying the target entity. Our experiments demonstrate the efficacy of this
+GER paradigm, showcasing state-of-the-art performance on the challenging OVEN
+benchmark. GER surpasses strong captioning, dual-encoder, visual matching and
+hierarchical classification baselines, affirming its advantage in tackling the
+complexities of web-scale recognition.",cs.CV,['cs.CV']
+How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?,Subhadeep Koley · Ayan Kumar Bhunia · Aneeshan Sain · Pinaki Nath Chowdhury · Tao Xiang · Yi-Zhe Song,https://subhadeepkoley.github.io/AbstractAway/,https://arxiv.org/abs/2403.07203,,2403.07203.pdf,How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?,"In this paper, we propose a novel abstraction-aware sketch-based image
+retrieval framework capable of handling sketch abstraction at varied levels.
+Prior works had mainly focused on tackling sub-factors such as drawing style
+and order, we instead attempt to model abstraction as a whole, and propose
+feature-level and retrieval granularity-level designs so that the system builds
+into its DNA the necessary means to interpret abstraction. On learning
+abstraction-aware features, we for the first-time harness the rich semantic
+embedding of pre-trained StyleGAN model, together with a novel
+abstraction-level mapper that deciphers the level of abstraction and
+dynamically selects appropriate dimensions in the feature matrix
+correspondingly, to construct a feature matrix embedding that can be freely
+traversed to accommodate different levels of abstraction. For granularity-level
+abstraction understanding, we dictate that the retrieval model should not treat
+all abstraction-levels equally and introduce a differentiable surrogate Acc.@q
+loss to inject that understanding into the system. Different to the
+gold-standard triplet loss, our Acc.@q loss uniquely allows a sketch to
+narrow/broaden its focus in terms of how stringent the evaluation should be -
+the more abstract a sketch, the less stringent (higher q). Extensive
+experiments depict our method to outperform existing state-of-the-arts in
+standard SBIR tasks along with challenging scenarios like early retrieval,
+forensic sketch-photo matching, and style-invariant retrieval.",cs.CV,['cs.CV']
+A Recipe for Scaling up Text-to-Video Generation with Text-free Videos,Xiang Wang · Shiwei Zhang · Hangjie Yuan · Zhiwu Qing · Biao Gong · Yingya Zhang · Yujun Shen · Changxin Gao · Nong Sang,https://tf-t2v.github.io/,https://arxiv.org/abs/2312.15770,,2312.15770.pdf,A Recipe for Scaling up Text-to-Video Generation with Text-free Videos,"Diffusion-based text-to-video generation has witnessed impressive progress in
+the past year yet still falls behind text-to-image generation. One of the key
+reasons is the limited scale of publicly available data (e.g., 10M video-text
+pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost
+of video captioning. Instead, it could be far easier to collect unlabeled clips
+from video platforms like YouTube. Motivated by this, we come up with a novel
+text-to-video generation framework, termed TF-T2V, which can directly learn
+with text-free videos. The rationale behind is to separate the process of text
+decoding from that of temporal modeling. To this end, we employ a content
+branch and a motion branch, which are jointly optimized with weights shared.
+Following such a pipeline, we study the effect of doubling the scale of
+training set (i.e., video-only WebVid10M) with some randomly collected
+text-free videos and are encouraged to observe the performance improvement (FID
+from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of
+our approach. We also find that our model could enjoy sustainable performance
+gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some
+text labels for training. Finally, we validate the effectiveness and
+generalizability of our ideology on both native text-to-video generation and
+compositional video synthesis paradigms. Code and models will be publicly
+available at https://tf-t2v.github.io/.",cs.CV,"['cs.CV', 'cs.AI']"
+HOIAnimator: Text-Prompt Human-Object Animations Generation with Perceptive Diffusion Models,Wenfeng Song · Xinyu Zhang · Shuai Li · Yang Gao · Aimin Hao · Xia HOU · Chenglizhao Chen · Ning Li · Hong Qin, ,https://arxiv.org/abs/2312.06553,,2312.06553.pdf,HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models,"We address the problem of generating realistic 3D human-object interactions
+(HOIs) driven by textual prompts. To this end, we take a modular design and
+decompose the complex task into simpler sub-tasks. We first develop a
+dual-branch diffusion model (HOI-DM) to generate both human and object motions
+conditioned on the input text, and encourage coherent motions by a
+cross-attention communication module between the human and object motion
+generation branches. We also develop an affordance prediction diffusion model
+(APDM) to predict the contacting area between the human and object during the
+interactions driven by the textual prompt. The APDM is independent of the
+results by the HOI-DM and thus can correct potential errors by the latter.
+Moreover, it stochastically generates the contacting points to diversify the
+generated motions. Finally, we incorporate the estimated contacting points into
+the classifier-guidance to achieve accurate and close contact between humans
+and objects. To train and evaluate our approach, we annotate BEHAVE dataset
+with text descriptions. Experimental results on BEHAVE and OMOMO demonstrate
+that our approach produces realistic HOIs with various interactions and
+different types of objects.",cs.CV,['cs.CV']
+Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement,Ziyu Wang · Yue Xu · Cewu Lu · Yonglu Li, ,https://arxiv.org/abs/2312.00362,,2312.00362.pdf,Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement,"Recently, dataset distillation has paved the way towards efficient machine
+learning, especially for image datasets. However, the distillation for videos,
+characterized by an exclusive temporal dimension, remains an underexplored
+domain. In this work, we provide the first systematic study of video
+distillation and introduce a taxonomy to categorize temporal compression. Our
+investigation reveals that the temporal information is usually not well learned
+during distillation, and the temporal dimension of synthetic data contributes
+little. The observations motivate our unified framework of disentangling the
+dynamic and static information in the videos. It first distills the videos into
+still images as static memory and then compensates the dynamic and motion
+information with a learnable dynamic memory block. Our method achieves
+state-of-the-art on video datasets at different scales, with a notably smaller
+memory storage budget. Our code is available at
+https://github.com/yuz1wan/video_distillation.",cs.CV,"['cs.CV', 'cs.LG']"
+Readout Guidance: Learning Control from Diffusion Features,Grace Luo · Trevor Darrell · Oliver Wang · Dan B Goldman · Aleksander Holynski,https://readout-guidance.github.io,https://arxiv.org/abs/2312.02150,,2312.02150.pdf,Readout Guidance: Learning Control from Diffusion Features,"We present Readout Guidance, a method for controlling text-to-image diffusion
+models with learned signals. Readout Guidance uses readout heads, lightweight
+networks trained to extract signals from the features of a pre-trained, frozen
+diffusion model at every timestep. These readouts can encode single-image
+properties, such as pose, depth, and edges; or higher-order properties that
+relate multiple images, such as correspondence and appearance similarity.
+Furthermore, by comparing the readout estimates to a user-defined target, and
+back-propagating the gradient through the readout head, these estimates can be
+used to guide the sampling process. Compared to prior methods for conditional
+generation, Readout Guidance requires significantly fewer added parameters and
+training samples, and offers a convenient and simple recipe for reproducing
+different forms of conditional control under a single framework, with a single
+architecture and sampling procedure. We showcase these benefits in the
+applications of drag-based manipulation, identity-consistent generation, and
+spatially aligned control. Project page: https://readout-guidance.github.io.",cs.CV,['cs.CV']
+BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection,Zhenxin Li · Shiyi Lan · Jose M. Alvarez · Zuxuan Wu, ,https://arxiv.org/abs/2312.01696v1,,2312.01696v1.pdf,BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection,"Recently, the rise of query-based Transformer decoders is reshaping
+camera-based 3D object detection. These query-based decoders are surpassing the
+traditional dense BEV (Bird's Eye View)-based methods. However, we argue that
+dense BEV frameworks remain important due to their outstanding abilities in
+depth estimation and object localization, depicting 3D scenes accurately and
+comprehensively. This paper aims to address the drawbacks of the existing dense
+BEV-based 3D object detectors by introducing our proposed enhanced components,
+including a CRF-modulated depth estimation module enforcing object-level
+consistencies, a long-term temporal aggregation module with extended receptive
+fields, and a two-stage object decoder combining perspective techniques with
+CRF-modulated depth embedding. These enhancements lead to a ""modernized"" dense
+BEV framework dubbed BEVNeXt. On the nuScenes benchmark, BEVNeXt outperforms
+both BEV-based and query-based frameworks under various settings, achieving a
+state-of-the-art result of 64.2 NDS on the nuScenes test set.",cs.CV,['cs.CV']
+It's All About Your Sketch: Democratising Sketch Control in Diffusion Models,Subhadeep Koley · Ayan Kumar Bhunia · Deeptanshu Sekhri · Aneeshan Sain · Pinaki Nath Chowdhury · Tao Xiang · Yi-Zhe Song,https://subhadeepkoley.github.io/StableSketching/,https://arxiv.org/abs/2403.07234,,2403.07234.pdf,It's All About Your Sketch: Democratising Sketch Control in Diffusion Models,"This paper unravels the potential of sketches for diffusion models,
+addressing the deceptive promise of direct sketch control in generative AI. We
+importantly democratise the process, enabling amateur sketches to generate
+precise images, living up to the commitment of ""what you sketch is what you
+get"". A pilot study underscores the necessity, revealing that deformities in
+existing models stem from spatial-conditioning. To rectify this, we propose an
+abstraction-aware framework, utilising a sketch adapter, adaptive time-step
+sampling, and discriminative guidance from a pre-trained fine-grained
+sketch-based image retrieval model, working synergistically to reinforce
+fine-grained sketch-photo association. Our approach operates seamlessly during
+inference without the need for textual prompts; a simple, rough sketch akin to
+what you and I can create suffices! We welcome everyone to examine results
+presented in the paper and its supplementary. Contributions include
+democratising sketch control, introducing an abstraction-aware framework, and
+leveraging discriminative guidance, validated through extensive experiments.",cs.CV,['cs.CV']
+COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction,Qihang Ma · Xin Tan · Yanyun Qu · Lizhuang Ma · Zhizhong Zhang · Yuan Xie,https://github.com/NotACracker/COTR,https://arxiv.org/abs/2312.01919,,2312.01919.pdf,COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction,"The autonomous driving community has shown significant interest in 3D
+occupancy prediction, driven by its exceptional geometric perception and
+general object recognition capabilities. To achieve this, current works try to
+construct a Tri-Perspective View (TPV) or Occupancy (OCC) representation
+extending from the Bird-Eye-View perception. However, compressed views like TPV
+representation lose 3D geometry information while raw and sparse OCC
+representation requires heavy but redundant computational costs. To address the
+above limitations, we propose Compact Occupancy TRansformer (COTR), with a
+geometry-aware occupancy encoder and a semantic-aware group decoder to
+reconstruct a compact 3D OCC representation. The occupancy encoder first
+generates a compact geometrical OCC feature through efficient explicit-implicit
+view transformation. Then, the occupancy decoder further enhances the semantic
+discriminability of the compact OCC representation by a coarse-to-fine semantic
+grouping strategy. Empirical experiments show that there are evident
+performance gains across multiple baselines, e.g., COTR outperforms baselines
+with a relative improvement of 8%-15%, demonstrating the superiority of our
+method.",cs.CV,['cs.CV']
+Global and Local Prompts Cooperation via Optimal Transport for Federated Learning,Hongxia Li · Wei Huang · Jingya Wang · Ye Shi,https://github.com/HongxiaLee/FedOTP,https://arxiv.org/abs/2403.00041,,2403.00041.pdf,Global and Local Prompts Cooperation via Optimal Transport for Federated Learning,"Prompt learning in pretrained visual-language models has shown remarkable
+flexibility across various downstream tasks. Leveraging its inherent
+lightweight nature, recent research attempted to integrate the powerful
+pretrained models into federated learning frameworks to simultaneously reduce
+communication costs and promote local training on insufficient data. Despite
+these efforts, current federated prompt learning methods lack specialized
+designs to systematically address severe data heterogeneities, e.g., data
+distribution with both label and feature shifts involved. To address this
+challenge, we present Federated Prompts Cooperation via Optimal Transport
+(FedOTP), which introduces efficient collaborative prompt learning strategies
+to capture diverse category traits on a per-client basis. Specifically, for
+each client, we learn a global prompt to extract consensus knowledge among
+clients, and a local prompt to capture client-specific category
+characteristics. Unbalanced Optimal Transport is then employed to align local
+visual features with these prompts, striking a balance between global consensus
+and local personalization. By relaxing one of the equality constraints, FedOTP
+enables prompts to focus solely on the core regions of image patches. Extensive
+experiments on datasets with various types of heterogeneities have demonstrated
+that our FedOTP outperforms the state-of-the-art methods.",cs.LG,"['cs.LG', 'cs.AI', 'cs.DC']"
+Rethinking the Evaluation Protocol of Domain Generalization,Han Yu · Xingxuan Zhang · Renzhe Xu · Jiashuo Liu · Yue He · Peng Cui, ,https://arxiv.org/abs/2307.11108,,2307.11108.pdf,Flatness-Aware Minimization for Domain Generalization,"Domain generalization (DG) seeks to learn robust models that generalize well
+under unknown distribution shifts. As a critical aspect of DG, optimizer
+selection has not been explored in depth. Currently, most DG methods follow the
+widely used benchmark, DomainBed, and utilize Adam as the default optimizer for
+all datasets. However, we reveal that Adam is not necessarily the optimal
+choice for the majority of current DG methods and datasets. Based on the
+perspective of loss landscape flatness, we propose a novel approach,
+Flatness-Aware Minimization for Domain Generalization (FAD), which can
+efficiently optimize both zeroth-order and first-order flatness simultaneously
+for DG. We provide theoretical analyses of the FAD's out-of-distribution (OOD)
+generalization error and convergence. Our experimental results demonstrate the
+superiority of FAD on various DG datasets. Additionally, we confirm that FAD is
+capable of discovering flatter optima in comparison to other zeroth-order and
+first-order flatness-aware optimization methods.",cs.CV,"['cs.CV', 'cs.LG']"
+"The More You See in 2D, the More You Perceive in 3D",Xinyang Han · Zelin Gao · Angjoo Kanazawa · Shubham Goel · Yossi Gandelsman, ,https://arxiv.org/abs/2404.03652,,2404.03652.pdf,"The More You See in 2D, the More You Perceive in 3D","Humans can infer 3D structure from 2D images of an object based on past
+experience and improve their 3D understanding as they see more images. Inspired
+by this behavior, we introduce SAP3D, a system for 3D reconstruction and novel
+view synthesis from an arbitrary number of unposed images. Given a few unposed
+images of an object, we adapt a pre-trained view-conditioned diffusion model
+together with the camera poses of the images via test-time fine-tuning. The
+adapted diffusion model and the obtained camera poses are then utilized as
+instance-specific priors for 3D reconstruction and novel view synthesis. We
+show that as the number of input images increases, the performance of our
+approach improves, bridging the gap between optimization-based prior-less 3D
+reconstruction methods and single-image-to-3D diffusion-based methods. We
+demonstrate our system on real images as well as standard synthetic benchmarks.
+Our ablation studies confirm that this adaption behavior is key for more
+accurate 3D understanding.",cs.CV,['cs.CV']
+"Selective, Interpretable and Motion Consistent Privacy Attribute Obfuscation for Action Recognition",Filip Ilic · He Zhao · Thomas Pock · Richard P. Wildes,https://f-ilic.github.io/SelectivePrivacyPreservation,https://arxiv.org/abs/2403.12710,,2403.12710.pdf,"Selective, Interpretable, and Motion Consistent Privacy Attribute Obfuscation for Action Recognition","Concerns for the privacy of individuals captured in public imagery have led
+to privacy-preserving action recognition. Existing approaches often suffer from
+issues arising through obfuscation being applied globally and a lack of
+interpretability. Global obfuscation hides privacy sensitive regions, but also
+contextual regions important for action recognition. Lack of interpretability
+erodes trust in these new technologies. We highlight the limitations of current
+paradigms and propose a solution: Human selected privacy templates that yield
+interpretability by design, an obfuscation scheme that selectively hides
+attributes and also induces temporal consistency, which is important in action
+recognition. Our approach is architecture agnostic and directly modifies input
+imagery, while existing approaches generally require architecture training. Our
+approach offers more flexibility, as no retraining is required, and outperforms
+alternatives on three widely used datasets.",cs.CV,"['cs.CV', 'cs.LG']"
+OpticalDR: A Deep Optical Imaging Model for Privacy-Protective Depression Recognition,Yuchen Pan · Junjun Jiang · Kui Jiang · Zhihao Wu · Keyuan Yu · Xianming Liu, ,https://arxiv.org/abs/2402.18786,,2402.18786.pdf,OpticalDR: A Deep Optical Imaging Model for Privacy-Protective Depression Recognition,"Depression Recognition (DR) poses a considerable challenge, especially in the
+context of the growing concerns surrounding privacy. Traditional automatic
+diagnosis of DR technology necessitates the use of facial images, undoubtedly
+expose the patient identity features and poses privacy risks. In order to
+mitigate the potential risks associated with the inappropriate disclosure of
+patient facial images, we design a new imaging system to erase the identity
+information of captured facial images while retain disease-relevant features.
+It is irreversible for identity information recovery while preserving essential
+disease-related characteristics necessary for accurate DR. More specifically,
+we try to record a de-identified facial image (erasing the identifiable
+features as much as possible) by a learnable lens, which is optimized in
+conjunction with the following DR task as well as a range of face analysis
+related auxiliary tasks in an end-to-end manner. These aforementioned
+strategies form our final Optical deep Depression Recognition network
+(OpticalDR). Experiments on CelebA, AVEC 2013, and AVEC 2014 datasets
+demonstrate that our OpticalDR has achieved state-of-the-art privacy protection
+performance with an average AUC of 0.51 on popular facial recognition models,
+and competitive results for DR with MAE/RMSE of 7.53/8.48 on AVEC 2013 and
+7.89/8.82 on AVEC 2014, respectively.",cs.CV,['cs.CV']
+NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis,Nilesh Kulkarni · Davis Rempe · Kyle Genova · Abhijit Kundu · Justin Johnson · David Fouhey · Leonidas Guibas, ,https://arxiv.org/abs/2307.07511,,2307.07511.pdf,NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis,"We address the problem of generating realistic 3D motions of humans
+interacting with objects in a scene. Our key idea is to create a neural
+interaction field attached to a specific object, which outputs the distance to
+the valid interaction manifold given a human pose as input. This interaction
+field guides the sampling of an object-conditioned human motion diffusion
+model, so as to encourage plausible contacts and affordance semantics. To
+support interactions with scarcely available data, we propose an automated
+synthetic data pipeline. For this, we seed a pre-trained motion model, which
+has priors for the basics of human movement, with interaction-specific anchor
+poses extracted from limited motion capture data. Using our guided diffusion
+model trained on generated synthetic data, we synthesize realistic motions for
+sitting and lifting with several objects, outperforming alternative approaches
+in terms of motion quality and successful action completion. We call our
+framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.",cs.CV,['cs.CV']
+On The Vulnerability of Efficient Vision Transformers to Adversarial Computation Attacks,Navaneet K L · Soroush Abbasi Koohpayegani · Essam Sleiman · Hamed Pirsiavash, ,https://arxiv.org/html/2208.09602v2,,2208.09602v2.pdf,Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective,"The Vision Transformer has emerged as a powerful tool for image
+classification tasks, surpassing the performance of convolutional neural
+networks (CNNs). Recently, many researchers have attempted to understand the
+robustness of Transformers against adversarial attacks. However, previous
+researches have focused solely on perturbations in the spatial domain. This
+paper proposes an additional perspective that explores the adversarial
+robustness of Transformers against frequency-selective perturbations in the
+spectral domain. To facilitate comparison between these two domains, an attack
+framework is formulated as a flexible tool for implementing attacks on images
+in the spatial and spectral domains. The experiments reveal that Transformers
+rely more on phase and low frequency information, which can render them more
+vulnerable to frequency-selective attacks than CNNs. This work offers new
+insights into the properties and adversarial robustness of Transformers.",cs.CV,['cs.CV']
+Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation,Daichi Horita · Naoto Inoue · Kotaro Kikuchi · Kota Yamaguchi · Kiyoharu Aizawa,https://udonda.github.io/RALF/,https://arxiv.org/abs/2311.13602,,2311.13602.pdf,Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation,"Content-aware graphic layout generation aims to automatically arrange visual
+elements along with a given content, such as an e-commerce product image. In
+this paper, we argue that the current layout generation approaches suffer from
+the limited training data for the high-dimensional layout structure. We show
+that a simple retrieval augmentation can significantly improve the generation
+quality. Our model, which is named Retrieval-Augmented Layout Transformer
+(RALF), retrieves nearest neighbor layout examples based on an input image and
+feeds these results into an autoregressive generator. Our model can apply
+retrieval augmentation to various controllable generation tasks and yield
+high-quality layouts within a unified architecture. Our extensive experiments
+show that RALF successfully generates content-aware layouts in both constrained
+and unconstrained settings and significantly outperforms the baselines.",cs.CV,['cs.CV']
+Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks,Shin'ya Yamaguchi · Sekitoshi Kanai · Kazuki Adachi · Daiki Chijiwa,https://github.com/yshinya6/adarand,https://arxiv.org/abs/2403.10097,,2403.10097.pdf,Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks,"While fine-tuning is a de facto standard method for training deep neural
+networks, it still suffers from overfitting when using small target datasets.
+Previous methods improve fine-tuning performance by maintaining knowledge of
+the source datasets or introducing regularization terms such as contrastive
+loss. However, these methods require auxiliary source information (e.g., source
+labels or datasets) or heavy additional computations. In this paper, we propose
+a simple method called adaptive random feature regularization (AdaRand).
+AdaRand helps the feature extractors of training models to adaptively change
+the distribution of feature vectors for downstream classification tasks without
+auxiliary source information and with reasonable computation costs. To this
+end, AdaRand minimizes the gap between feature vectors and random reference
+vectors that are sampled from class conditional Gaussian distributions.
+Furthermore, AdaRand dynamically updates the conditional distribution to follow
+the currently updated feature extractors and balance the distance between
+classes in feature spaces. Our experiments show that AdaRand outperforms the
+other fine-tuning regularization, which requires auxiliary source information
+and heavy computation costs.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+UFC-Net: Unrolling Fixed-point Continuous Network for Deep Compressive Sensing,Xiaoyang Wang · Hongping Gan, ,,https://link.springer.com/article/10.1007/s11263-023-01814-w,,,,,nan
+Error Detection in Egocentric Procedural Task Videos,Shih-Po Lee · Zijia Lu · Zekun Zhang · Minh Hoai · Ehsan Elhamifar, ,https://arxiv.org/abs/2404.01933,,2404.01933.pdf,PREGO: online mistake detection in PRocedural EGOcentric videos,"Promptly identifying procedural errors from egocentric videos in an online
+setting is highly challenging and valuable for detecting mistakes as soon as
+they happen. This capability has a wide range of applications across various
+fields, such as manufacturing and healthcare. The nature of procedural mistakes
+is open-set since novel types of failures might occur, which calls for
+one-class classifiers trained on correctly executed procedures. However, no
+technique can currently detect open-set procedural mistakes online. We propose
+PREGO, the first online one-class classification model for mistake detection in
+PRocedural EGOcentric videos. PREGO is based on an online action recognition
+component to model the current action, and a symbolic reasoning module to
+predict the next actions. Mistake detection is performed by comparing the
+recognized current action with the expected future one. We evaluate PREGO on
+two procedural egocentric video datasets, Assembly101 and Epic-tent, which we
+adapt for online benchmarking of procedural mistake detection to establish
+suitable benchmarks, thus defining the Assembly101-O and Epic-tent-O datasets,
+respectively.",cs.CV,['cs.CV']
+Low-Rank Knowledge Decomposition for Medical Foundation Models,Yuhang Zhou · Haolin li · Siyuan Du · Jiangchao Yao · Ya Zhang · Yanfeng Wang, ,https://arxiv.org/abs/2404.17184,,2404.17184.pdf,Low-Rank Knowledge Decomposition for Medical Foundation Models,"The popularity of large-scale pre-training has promoted the development of
+medical foundation models. However, some studies have shown that although
+foundation models exhibit strong general feature extraction capabilities, their
+performance on specific tasks is still inferior to task-specific methods. In
+this paper, we explore a new perspective called ``Knowledge Decomposition'' to
+improve the performance on specific medical tasks, which deconstruct the
+foundation model into multiple lightweight expert models, each dedicated to a
+particular task, with the goal of improving specialization while concurrently
+mitigating resource expenditure. To accomplish the above objective, we design a
+novel framework named Low-Rank Knowledge Decomposition (LoRKD), which
+explicitly separates graidents by incorporating low-rank expert modules and the
+efficient knowledge separation convolution. Extensive experimental results
+demonstrate that the decomposed models perform well in terms of performance and
+transferability, even surpassing the original foundation models.",cs.CV,['cs.CV']
+GS-IR: 3D Gaussian Splatting for Inverse Rendering,Zhihao Liang · Qi Zhang · Ying Feng · Ying Shan · Kui Jia, ,https://arxiv.org/abs/2311.16473,,2311.16473.pdf,GS-IR: 3D Gaussian Splatting for Inverse Rendering,"We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian
+Splatting (GS) that leverages forward mapping volume rendering to achieve
+photorealistic novel view synthesis and relighting results. Unlike previous
+works that use implicit neural representations and volume rendering (e.g.
+NeRF), which suffer from low expressive power and high computational
+complexity, we extend GS, a top-performance representation for novel view
+synthesis, to estimate scene geometry, surface material, and environment
+illumination from multi-view images captured under unknown lighting conditions.
+There are two main problems when introducing GS to inverse rendering: 1) GS
+does not support producing plausible normal natively; 2) forward mapping (e.g.
+rasterization and splatting) cannot trace the occlusion like backward mapping
+(e.g. ray tracing). To address these challenges, our GS-IR proposes an
+efficient optimization scheme that incorporates a depth-derivation-based
+regularization for normal estimation and a baking-based occlusion to model
+indirect lighting. The flexible and expressive GS representation allows us to
+achieve fast and compact geometry reconstruction, photorealistic novel view
+synthesis, and effective physically-based rendering. We demonstrate the
+superiority of our method over baseline methods through qualitative and
+quantitative evaluations on various challenging scenes.",cs.CV,['cs.CV']
+Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation,Yuanhong Chen · Yuyuan Liu · Hu Wang · Fengbei Liu · Chong Wang · Helen Frazer · Gustavo Carneiro, ,https://arxiv.org/abs/2310.18709,,2310.18709.pdf,Audio-Visual Instance Segmentation,"In this paper, we propose a new multi-modal task, namely audio-visual
+instance segmentation (AVIS), in which the goal is to identify, segment, and
+track individual sounding object instances in audible videos, simultaneously.
+To our knowledge, it is the first time that instance segmentation has been
+extended into the audio-visual domain. To better facilitate this research, we
+construct the first audio-visual instance segmentation benchmark (AVISeg).
+Specifically, AVISeg consists of 1,258 videos with an average duration of 62.6
+seconds from YouTube and public audio-visual datasets, where 117 videos have
+been annotated by using an interactive semi-automatic labeling tool based on
+the Segment Anything Model (SAM). In addition, we present a simple baseline
+model for the AVIS task. Our new model introduces an audio branch and a
+cross-modal fusion module to Mask2Former to locate all sounding objects.
+Finally, we evaluate the proposed method using two backbones on AVISeg. We
+believe that AVIS will inspire the community towards a more comprehensive
+multi-modal understanding.",cs.CV,"['cs.CV', 'cs.LG', 'cs.MM', 'cs.SD', 'eess.AS']"
+Towards Generalizable Multi-Object Tracking,Zheng Qin · Le Wang · Sanping Zhou · Panpan Fu · Gang Hua · Wei Tang, ,http://export.arxiv.org/abs/2311.10382,,2311.10382.pdf,Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking,"Multi-Object Tracking (MOT) remains a vital component of intelligent video
+analysis, which aims to locate targets and maintain a consistent identity for
+each target throughout a video sequence. Existing works usually learn a
+discriminative feature representation, such as motion and appearance, to
+associate the detections across frames, which are easily affected by mutual
+occlusion and background clutter in practice. In this paper, we propose a
+simple yet effective two-stage feature learning paradigm to jointly learn
+single-shot and multi-shot features for different targets, so as to achieve
+robust data association in the tracking process. For the detections without
+being associated, we design a novel single-shot feature learning module to
+extract discriminative features of each detection, which can efficiently
+associate targets between adjacent frames. For the tracklets being lost several
+frames, we design a novel multi-shot feature learning module to extract
+discriminative features of each tracklet, which can accurately refind these
+lost targets after a long period. Once equipped with a simple data association
+logic, the resulting VisualTracker can perform robust MOT based on the
+single-shot and multi-shot feature representations. Extensive experimental
+results demonstrate that our method has achieved significant improvements on
+MOT17 and MOT20 datasets while reaching state-of-the-art performance on
+DanceTrack dataset.",cs.CV,['cs.CV']
+Authentic Hand Avatar from a Phone Scan via Universal Hand Model,Gyeongsik Moon · Weipeng Xu · Rohan Joshi · Chenglei Wu · Takaaki Shiratori, ,https://arxiv.org/abs/2405.07933,,2405.07933.pdf,Authentic Hand Avatar from a Phone Scan via Universal Hand Model,"The authentic 3D hand avatar with every identifiable information, such as
+hand shapes and textures, is necessary for immersive experiences in AR/VR. In
+this paper, we present a universal hand model (UHM), which 1) can universally
+represent high-fidelity 3D hand meshes of arbitrary identities (IDs) and 2) can
+be adapted to each person with a short phone scan for the authentic hand
+avatar. For effective universal hand modeling, we perform tracking and modeling
+at the same time, while previous 3D hand models perform them separately. The
+conventional separate pipeline suffers from the accumulated errors from the
+tracking stage, which cannot be recovered in the modeling stage. On the other
+hand, ours does not suffer from the accumulated errors while having a much more
+concise overall pipeline. We additionally introduce a novel image matching loss
+function to address a skin sliding during the tracking and modeling, while
+existing works have not focused on it much. Finally, using learned priors from
+our UHM, we effectively adapt our UHM to each person's short phone scan for the
+authentic hand avatar.",cs.CV,['cs.CV']
+WANDR: Intention-guided Human Motion Generation,Markos Diomataris · Nikos Athanasiou · Omid Taheri · Xi Wang · Otmar Hilliges · Michael J. Black,https://wandr.is.tue.mpg.de/,https://arxiv.org/abs/2404.15383,,2404.15383.pdf,WANDR: Intention-guided Human Motion Generation,"Synthesizing natural human motions that enable a 3D human avatar to walk and
+reach for arbitrary goals in 3D space remains an unsolved problem with many
+applications. Existing methods (data-driven or using reinforcement learning)
+are limited in terms of generalization and motion naturalness. A primary
+obstacle is the scarcity of training data that combines locomotion with goal
+reaching. To address this, we introduce WANDR, a data-driven model that takes
+an avatar's initial pose and a goal's 3D position and generates natural human
+motions that place the end effector (wrist) on the goal location. To solve
+this, we introduce novel intention features that drive rich goal-oriented
+movement. Intention guides the agent to the goal, and interactively adapts the
+generation to novel situations without needing to define sub-goals or the
+entire motion path. Crucially, intention allows training on datasets that have
+goal-oriented motions as well as those that do not. WANDR is a conditional
+Variational Auto-Encoder (c-VAE), which we train using the AMASS and CIRCLE
+datasets. We evaluate our method extensively and demonstrate its ability to
+generate natural and long-term motions that reach 3D goals and generalize to
+unseen goal locations. Our models and code are available for research purposes
+at wandr.is.tue.mpg.de.",cs.CV,"['cs.CV', 'cs.AI']"
+SynSP: Synergy of Smoothness and Precision in Pose Sequences Refinement,Tao Wang · Lei Jin · Zheng Wang · Jianshu Li · Liang Li · Fang Zhao · Yu Cheng · Li Yuan · Li ZHOU · Junliang Xing · Jian Zhao, ,https://arxiv.org/abs/2311.09543,,2311.09543.pdf,Temporal-Aware Refinement for Video-based Human Pose and Shape Recovery,"Though significant progress in human pose and shape recovery from monocular
+RGB images has been made in recent years, obtaining 3D human motion with high
+accuracy and temporal consistency from videos remains challenging. Existing
+video-based methods tend to reconstruct human motion from global image
+features, which lack detailed representation capability and limit the
+reconstruction accuracy. In this paper, we propose a Temporal-Aware Refining
+Network (TAR), to synchronously explore temporal-aware global and local image
+features for accurate pose and shape recovery. First, a global transformer
+encoder is introduced to obtain temporal global features from static feature
+sequences. Second, a bidirectional ConvGRU network takes the sequence of
+high-resolution feature maps as input, and outputs temporal local feature maps
+that maintain high resolution and capture the local motion of the human body.
+Finally, a recurrent refinement module iteratively updates estimated SMPL
+parameters by leveraging both global and local temporal information to achieve
+accurate and smooth results. Extensive experiments demonstrate that our TAR
+obtains more accurate results than previous state-of-the-art methods on popular
+benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M.",cs.CV,['cs.CV']
+vid-TLDR: Training Free Token merging for Light-weight Video Transformer,Joonmyung Choi · Sanghyeok Lee · Jaewon Chu · Minhyuk Choi · Hyunwoo J. Kim,https://github.com/mlvlab/vid-TLDR,https://arxiv.org/abs/2403.13347,,2403.13347.pdf,vid-TLDR: Training Free Token merging for Light-weight Video Transformer,"Video Transformers have become the prevalent solution for various video
+downstream tasks with superior expressive power and flexibility. However, these
+video transformers suffer from heavy computational costs induced by the massive
+number of tokens across the entire video frames, which has been the major
+barrier to training the model. Further, the patches irrelevant to the main
+contents, e.g., backgrounds, degrade the generalization performance of models.
+To tackle these issues, we propose training free token merging for lightweight
+video Transformer (vid-TLDR) that aims to enhance the efficiency of video
+Transformers by merging the background tokens without additional training. For
+vid-TLDR, we introduce a novel approach to capture the salient regions in
+videos only with the attention map. Further, we introduce the saliency-aware
+token merging strategy by dropping the background tokens and sharpening the
+object scores. Our experiments show that vid-TLDR significantly mitigates the
+computational complexity of video Transformers while achieving competitive
+performance compared to the base model without vid-TLDR. Code is available at
+https://github.com/mlvlab/vid-TLDR.",cs.CV,['cs.CV']
+Boosting Image Restoration via Priors from Pre-trained Models,Xiaogang Xu · Shu Kong · Tao Hu · Zhe Liu · Hujun Bao, ,https://arxiv.org/abs/2403.06793,,2403.06793.pdf,Boosting Image Restoration via Priors from Pre-trained Models,"Pre-trained models with large-scale training data, such as CLIP and Stable
+Diffusion, have demonstrated remarkable performance in various high-level
+computer vision tasks such as image understanding and generation from language
+descriptions. Yet, their potential for low-level tasks such as image
+restoration remains relatively unexplored. In this paper, we explore such
+models to enhance image restoration. As off-the-shelf features (OSF) from
+pre-trained models do not directly serve image restoration, we propose to learn
+an additional lightweight module called Pre-Train-Guided Refinement Module
+(PTG-RM) to refine restoration results of a target restoration network with
+OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying
+Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention
+(PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations,
+while PTG-CSA enhances spatial-channel attention for restoration-related
+learning. Extensive experiments demonstrate that PTG-RM, with its compact size
+($<$1M parameters), effectively enhances restoration performance of various
+models across different tasks, including low-light enhancement, deraining,
+deblurring, and denoising.",cs.CV,['cs.CV']
+HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting,Yuheng Jiang · Zhehao Shen · Penghao Wang · Zhuo Su · Yu Hong · Yingliang Zhang · Jingyi Yu · Lan Xu,https://nowheretrix.github.io/HiFi4G/,https://arxiv.org/abs/2312.03461,,2312.03461.pdf,HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting,"We have recently seen tremendous progress in photo-real human modeling and
+rendering. Yet, efficiently rendering realistic human performance and
+integrating it into the rasterization pipeline remains challenging. In this
+paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
+high-fidelity human performance rendering from dense footage. Our core
+intuition is to marry the 3D Gaussian representation with non-rigid tracking,
+achieving a compact and compression-friendly representation. We first propose a
+dual-graph mechanism to obtain motion priors, with a coarse deformation graph
+for effective initialization and a fine-grained Gaussian graph to enforce
+subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
+adaptive spatial-temporal regularizers to effectively balance the non-rigid
+prior and Gaussian updating. We also present a companion compression scheme
+with residual compensation for immersive experiences on various platforms. It
+achieves a substantial compression rate of approximately 25 times, with less
+than 2MB of storage per frame. Extensive experiments demonstrate the
+effectiveness of our approach, which significantly outperforms existing
+approaches in terms of optimization speed, rendering quality, and storage
+overhead.",cs.CV,['cs.CV']
+Preserving Fairness Generalization in Deepfake Detection,Li Lin · Li Lin · Xinan He · Yan Ju · Xin Wang · Feng Ding · Shu Hu, ,https://arxiv.org/abs/2402.17229v1,,2402.17229v1.pdf,Preserving Fairness Generalization in Deepfake Detection,"Although effective deepfake detection models have been developed in recent
+years, recent studies have revealed that these models can result in unfair
+performance disparities among demographic groups, such as race and gender. This
+can lead to particular groups facing unfair targeting or exclusion from
+detection, potentially allowing misclassified deepfakes to manipulate public
+opinion and undermine trust in the model. The existing method for addressing
+this problem is providing a fair loss function. It shows good fairness
+performance for intra-domain evaluation but does not maintain fairness for
+cross-domain testing. This highlights the significance of fairness
+generalization in the fight against deepfakes. In this work, we propose the
+first method to address the fairness generalization problem in deepfake
+detection by simultaneously considering features, loss, and optimization
+aspects. Our method employs disentanglement learning to extract demographic and
+domain-agnostic forgery features, fusing them to encourage fair learning across
+a flattened loss landscape. Extensive experiments on prominent deepfake
+datasets demonstrate our method's effectiveness, surpassing state-of-the-art
+approaches in preserving fairness during cross-domain deepfake detection. The
+code is available at https://github.com/Purdue-M2/Fairness-Generalization",cs.CV,"['cs.CV', 'cs.CY', 'cs.LG']"
+CoSeR: Bridging Image and Language for Cognitive Super-Resolution,Haoze Sun · Wenbo Li · Jianzhuang Liu · Haoyu Chen · Renjing Pei · Xueyi Zou · Youliang Yan · Yujiu Yang, ,https://arxiv.org/abs/2311.16512,,2311.16512.pdf,CoSeR: Bridging Image and Language for Cognitive Super-Resolution,"Existing super-resolution (SR) models primarily focus on restoring local
+texture details, often neglecting the global semantic information within the
+scene. This oversight can lead to the omission of crucial semantic details or
+the introduction of inaccurate textures during the recovery process. In our
+work, we introduce the Cognitive Super-Resolution (CoSeR) framework, empowering
+SR models with the capacity to comprehend low-resolution images. We achieve
+this by marrying image appearance and language understanding to generate a
+cognitive embedding, which not only activates prior information from large
+text-to-image diffusion models but also facilitates the generation of
+high-quality reference images to optimize the SR process. To further improve
+image fidelity, we propose a novel condition injection scheme called
+""All-in-Attention"", consolidating all conditional information into a single
+module. Consequently, our method successfully restores semantically correct and
+photorealistic details, demonstrating state-of-the-art performance across
+multiple benchmarks. Code: https://github.com/VINHYU/CoSeR",cs.CV,"['cs.CV', 'cs.AI']"
+Task-Customized Mixture of Adapters for General Image Fusion,Pengfei Zhu · Yang Sun · Bing Cao · Qinghua Hu, ,https://arxiv.org/abs/2403.12494,,2403.12494.pdf,Task-Customized Mixture of Adapters for General Image Fusion,"General image fusion aims at integrating important information from
+multi-source images. However, due to the significant cross-task gap, the
+respective fusion mechanism varies considerably in practice, resulting in
+limited performance across subtasks. To handle this problem, we propose a novel
+task-customized mixture of adapters (TC-MoA) for general image fusion,
+adaptively prompting various fusion tasks in a unified model. We borrow the
+insight from the mixture of experts (MoE), taking the experts as efficient
+tuning adapters to prompt a pre-trained foundation model. These adapters are
+shared across different tasks and constrained by mutual information
+regularization, ensuring compatibility with different tasks while
+complementarity for multi-source images. The task-specific routing networks
+customize these adapters to extract task-specific information from different
+sources with dynamic dominant intensity, performing adaptive visual feature
+prompt fusion. Notably, our TC-MoA controls the dominant intensity bias for
+different fusion tasks, successfully unifying multiple fusion tasks in a single
+model. Extensive experiments show that TC-MoA outperforms the competing
+approaches in learning commonalities while retaining compatibility for general
+image fusion (multi-modal, multi-exposure, and multi-focus), and also
+demonstrating striking controllability on more generalization experiments. The
+code is available at https://github.com/YangSun22/TC-MoA .",cs.CV,['cs.CV']
+Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation,Shuting He · Henghui Ding, ,https://arxiv.org/abs/2404.03645,,2404.03645.pdf,Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation,"Referring video segmentation relies on natural language expressions to
+identify and segment objects, often emphasizing motion clues. Previous works
+treat a sentence as a whole and directly perform identification at the
+video-level, mixing up static image-level cues with temporal motion cues.
+However, image-level features cannot well comprehend motion cues in sentences,
+and static cues are not crucial for temporal perception. In fact, static cues
+can sometimes interfere with temporal perception by overshadowing motion cues.
+In this work, we propose to decouple video-level referring expression
+understanding into static and motion perception, with a specific emphasis on
+enhancing temporal comprehension. Firstly, we introduce an
+expression-decoupling module to make static cues and motion cues perform their
+distinct role, alleviating the issue of sentence embeddings overlooking motion
+cues. Secondly, we propose a hierarchical motion perception module to capture
+temporal information effectively across varying timescales. Furthermore, we
+employ contrastive learning to distinguish the motions of visually similar
+objects. These contributions yield state-of-the-art performance across five
+datasets, including a remarkable $\textbf{9.2%}$ $\mathcal{J\&F}$ improvement
+on the challenging $\textbf{MeViS}$ dataset. Code is available at
+https://github.com/heshuting555/DsHmp.",cs.CV,['cs.CV']
+MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark,Sanghyun Woo · Kwanyong Park · Inkyu Shin · Myungchul Kim · In So Kweon,https://sites.google.com/view/mtmmc,https://arxiv.org/abs/2403.20225,,2403.20225.pdf,MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark,"Multi-target multi-camera tracking is a crucial task that involves
+identifying and tracking individuals over time using video streams from
+multiple cameras. This task has practical applications in various fields, such
+as visual surveillance, crowd behavior analysis, and anomaly detection.
+However, due to the difficulty and cost of collecting and labeling data,
+existing datasets for this task are either synthetically generated or
+artificially constructed within a controlled camera network setting, which
+limits their ability to model real-world dynamics and generalize to diverse
+camera configurations. To address this issue, we present MTMMC, a real-world,
+large-scale dataset that includes long video sequences captured by 16
+multi-modal cameras in two different environments - campus and factory - across
+various time, weather, and season conditions. This dataset provides a
+challenging test-bed for studying multi-camera tracking under diverse
+real-world complexities and includes an additional input modality of spatially
+aligned and temporally synchronized RGB and thermal cameras, which enhances the
+accuracy of multi-camera tracking. MTMMC is a super-set of existing datasets,
+benefiting independent fields such as person detection, re-identification, and
+multiple object tracking. We provide baselines and new learning setups on this
+dataset and set the reference scores for future studies. The datasets, models,
+and test server will be made publicly available.",cs.CV,['cs.CV']
+Pseudo Label Refinery for Unsupervised Domain Adaptation on Cross-dataset 3D Object Detection,Zhanwei Zhang · Minghao Chen · Shuai Xiao · Liang Peng · Hengjia Li · Binbin Lin · Ping Li · Wenxiao Wang · Boxi Wu · Deng Cai, ,https://arxiv.org/abs/2404.19384,,2404.19384.pdf,Pseudo Label Refinery for Unsupervised Domain Adaptation on Cross-dataset 3D Object Detection,"Recent self-training techniques have shown notable improvements in
+unsupervised domain adaptation for 3D object detection (3D UDA). These
+techniques typically select pseudo labels, i.e., 3D boxes, to supervise models
+for the target domain. However, this selection process inevitably introduces
+unreliable 3D boxes, in which 3D points cannot be definitively assigned as
+foreground or background. Previous techniques mitigate this by reweighting
+these boxes as pseudo labels, but these boxes can still poison the training
+process. To resolve this problem, in this paper, we propose a novel pseudo
+label refinery framework. Specifically, in the selection process, to improve
+the reliability of pseudo boxes, we propose a complementary augmentation
+strategy. This strategy involves either removing all points within an
+unreliable box or replacing it with a high-confidence box. Moreover, the point
+numbers of instances in high-beam datasets are considerably higher than those
+in low-beam datasets, also degrading the quality of pseudo labels during the
+training process. We alleviate this issue by generating additional proposals
+and aligning RoI features across different domains. Experimental results
+demonstrate that our method effectively enhances the quality of pseudo labels
+and consistently surpasses the state-of-the-art methods on six autonomous
+driving benchmarks. Code will be available at
+https://github.com/Zhanwei-Z/PERE.",cs.CV,"['cs.CV', 'cs.AI']"
+Unbiased Faster R-CNN for Single-source Domain Generalized Object Detection,Yajing Liu · Shijun Zhou · Xiyao Liu · chunhui Hao · Baojie Fan · Jiandong Tian, ,https://arxiv.org/abs/2405.15225,,2405.15225.pdf,Unbiased Faster R-CNN for Single-source Domain Generalized Object Detection,"Single-source domain generalization (SDG) for object detection is a
+challenging yet essential task as the distribution bias of the unseen domain
+degrades the algorithm performance significantly. However, existing methods
+attempt to extract domain-invariant features, neglecting that the biased data
+leads the network to learn biased features that are non-causal and poorly
+generalizable. To this end, we propose an Unbiased Faster R-CNN (UFR) for
+generalizable feature learning. Specifically, we formulate SDG in object
+detection from a causal perspective and construct a Structural Causal Model
+(SCM) to analyze the data bias and feature bias in the task, which are caused
+by scene confounders and object attribute confounders. Based on the SCM, we
+design a Global-Local Transformation module for data augmentation, which
+effectively simulates domain diversity and mitigates the data bias.
+Additionally, we introduce a Causal Attention Learning module that incorporates
+a designed attention invariance loss to learn image-level features that are
+robust to scene confounders. Moreover, we develop a Causal Prototype Learning
+module with an explicit instance constraint and an implicit prototype
+constraint, which further alleviates the negative impact of object attribute
+confounders. Experimental results on five scenes demonstrate the prominent
+generalization ability of our method, with an improvement of 3.9% mAP on the
+Night-Clear scene.",cs.CV,['cs.CV']
+DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning,Sikai Bai · Jie ZHANG · Song Guo · Shuaicheng Li · Jingcai Guo · Jun Hou · Tao Han · Xiaocheng Lu, ,https://arxiv.org/abs/2403.08506,,2403.08506.pdf,DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning,"Federated learning (FL) has emerged as a powerful paradigm for learning from
+decentralized data, and federated domain generalization further considers the
+test dataset (target domain) is absent from the decentralized training data
+(source domains). However, most existing FL methods assume that domain labels
+are provided during training, and their evaluation imposes explicit constraints
+on the number of domains, which must strictly match the number of clients.
+Because of the underutilization of numerous edge devices and additional
+cross-client domain annotations in the real world, such restrictions may be
+impractical and involve potential privacy leaks. In this paper, we propose an
+efficient and novel approach, called Disentangled Prompt Tuning (DiPrompT), a
+method that tackles the above restrictions by learning adaptive prompts for
+domain generalization in a distributed manner. Specifically, we first design
+two types of prompts, i.e., global prompt to capture general knowledge across
+all clients and domain prompts to capture domain-specific knowledge. They
+eliminate the restriction on the one-to-one mapping between source domains and
+local clients. Furthermore, a dynamic query metric is introduced to
+automatically search the suitable domain label for each sample, which includes
+two-substep text-image alignments based on prompt tuning without
+labor-intensive annotation. Extensive experiments on multiple datasets
+demonstrate that our DiPrompT achieves superior domain generalization
+performance over state-of-the-art FL methods when domain labels are not
+provided, and even outperforms many centralized learning methods using domain
+labels.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+HIVE: Harnessing Human Feedback for Instructional Visual Editing,Shu Zhang · Xinyi Yang · Yihao Feng · Can Qin · Chia-Chih Chen · Ning Yu · Zeyuan Chen · Huan Wang · Silvio Savarese · Stefano Ermon · Caiming Xiong · Ran Xu, ,,https://www.semanticscholar.org/paper/HQ-Edit:-A-High-Quality-Dataset-for-Image-Editing-Hui-Yang/09609bd28855fd9b27f043b4dbf509615229bd08,,,,,nan
+LightIt: Illumination Modeling and Control for Diffusion Models,Peter Kocsis · Kalyan Sunkavalli · Julien Philip · Matthias Nießner · Yannick Hold-Geoffroy,https://peter-kocsis.github.io/LightIt/,https://arxiv.org/abs/2403.10615,,2403.10615.pdf,LightIt: Illumination Modeling and Control for Diffusion Models,"We introduce LightIt, a method for explicit illumination control for image
+generation. Recent generative methods lack lighting control, which is crucial
+to numerous artistic aspects of image generation such as setting the overall
+mood or cinematic appearance. To overcome these limitations, we propose to
+condition the generation on shading and normal maps. We model the lighting with
+single bounce shading, which includes cast shadows. We first train a shading
+estimation module to generate a dataset of real-world images and shading pairs.
+Then, we train a control network using the estimated shading and normals as
+input. Our method demonstrates high-quality image generation and lighting
+control in numerous scenes. Additionally, we use our generated dataset to train
+an identity-preserving relighting model, conditioned on an image and a target
+shading. Our method is the first that enables the generation of images with
+controllable, consistent lighting and performs on par with specialized
+relighting state-of-the-art methods.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG', 'I.4.8; I.2.10']"
+Byzantine-robust Decentralized Federated Learning via Dual-domain Clustering and Trust Bootstrapping,Peng Sun · Xinyang Liu · Zhibo Wang · Bo Liu, ,,https://dl.acm.org/doi/abs/10.1145/3637494.3638729,,,,,nan
+Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing,Bi'an Du · Xiang Gao · Wei Hu · Renjie Liao, ,https://arxiv.org/abs/2402.17464,,2402.17464.pdf,Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing,"Generative 3D part assembly involves understanding part relationships and
+predicting their 6-DoF poses for assembling a realistic 3D shape. Prior work
+often focus on the geometry of individual parts, neglecting part-whole
+hierarchies of objects. Leveraging two key observations: 1) super-part poses
+provide strong hints about part poses, and 2) predicting super-part poses is
+easier due to fewer superparts, we propose a part-whole-hierarchy message
+passing network for efficient 3D part assembly. We first introduce super-parts
+by grouping geometrically similar parts without any semantic labels. Then we
+employ a part-whole hierarchical encoder, wherein a super-part encoder predicts
+latent super-part poses based on input parts. Subsequently, we transform the
+point cloud using the latent poses, feeding it to the part encoder for
+aggregating super-part information and reasoning about part relationships to
+predict all part poses. In training, only ground-truth part poses are required.
+During inference, the predicted latent poses of super-parts enhance
+interpretability. Experimental results on the PartNet dataset show that our
+method achieves state-of-the-art performance in part and connectivity accuracy
+and enables an interpretable hierarchical part assembly. Code is available at
+https://github.com/pkudba/3DHPA.",cs.CV,['cs.CV']
+FreeMan: Towards benchmarking 3D human pose estimation under Real-World Conditions,Jiong WANG · Fengyu Yang · Bingliang Li · Wenbo Gou · Danqi Yan · Ailing Zeng · Ailing Zeng · Yijun Gao · Junle Wang · Yanqing Jing · Ruimao Zhang,https://wangjiongw.github.io/freeman/,https://arxiv.org/abs/2309.05073,,2309.05073.pdf,FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions,"Estimating the 3D structure of the human body from natural scenes is a
+fundamental aspect of visual perception. 3D human pose estimation is a vital
+step in advancing fields like AIGC and human-robot interaction, serving as a
+crucial technique for understanding and interacting with human actions in
+real-world settings. However, the current datasets, often collected under
+single laboratory conditions using complex motion capture equipment and
+unvarying backgrounds, are insufficient. The absence of datasets on variable
+conditions is stalling the progress of this crucial task. To facilitate the
+development of 3D pose estimation, we present FreeMan, the first large-scale,
+multi-view dataset collected under the real-world conditions. FreeMan was
+captured by synchronizing 8 smartphones across diverse scenarios. It comprises
+11M frames from 8000 sequences, viewed from different perspectives. These
+sequences cover 40 subjects across 10 different scenarios, each with varying
+lighting conditions. We have also established an semi-automated pipeline
+containing error detection to reduce the workload of manual check and ensure
+precise annotation. We provide comprehensive evaluation baselines for a range
+of tasks, underlining the significant challenges posed by FreeMan. Further
+evaluations of standard indoor/outdoor human sensing datasets reveal that
+FreeMan offers robust representation transferability in real and complex
+scenes. Code and data are available at https://wangjiongw.github.io/freeman.",cs.CV,['cs.CV']
+Generative Multimodal Models are In-Context Learners,Quan Sun · Yufeng Cui · Yufeng Cui · Xiaosong Zhang · Fan Zhang · Qiying Yu · Yueze Wang · Yongming Rao · Jingjing Liu · Tiejun Huang · Xinlong Wang, ,https://arxiv.org/abs/2312.13286,,2312.13286.pdf,Generative Multimodal Models are In-Context Learners,"The human ability to easily solve multimodal tasks in context (i.e., with
+only a few demonstrations or simple instructions), is what current multimodal
+systems have largely struggled to imitate. In this work, we demonstrate that
+the task-agnostic in-context learning capabilities of large multimodal models
+can be significantly enhanced by effective scaling-up. We introduce Emu2, a
+generative multimodal model with 37 billion parameters, trained on large-scale
+multimodal sequences with a unified autoregressive objective. Emu2 exhibits
+strong multimodal in-context learning abilities, even emerging to solve tasks
+that require on-the-fly reasoning, such as visual prompting and object-grounded
+generation. The model sets a new record on multiple multimodal understanding
+tasks in few-shot settings. When instruction-tuned to follow specific
+instructions, Emu2 further achieves new state-of-the-art on challenging tasks
+such as question answering benchmarks for large multimodal models and
+open-ended subject-driven generation. These achievements demonstrate that Emu2
+can serve as a base model and general-purpose interface for a wide range of
+multimodal tasks. Code and models are publicly available to facilitate future
+research.",cs.CV,['cs.CV']
+SVDTree: Semantic Voxel Diffusion for Single Image Tree Reconstruction,Yuan Li · Zhihao Liu · Bedrich Benes · Xiaopeng Zhang · Jianwei Guo,https://github.com/RyuZhihao123/SVDTree,https://arxiv.org/abs/2402.12712,,2402.12712.pdf,MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction,"This paper presents a neural architecture MVDiffusion++ for 3D object
+reconstruction that synthesizes dense and high-resolution views of an object
+given one or a few images without camera poses. MVDiffusion++ achieves superior
+flexibility and scalability with two surprisingly simple ideas: 1) A
+``pose-free architecture'' where standard self-attention among 2D latent
+features learns 3D consistency across an arbitrary number of conditional and
+generation views without explicitly using camera pose information; and 2) A
+``view dropout strategy'' that discards a substantial number of output views
+during training, which reduces the training-time memory footprint and enables
+dense and high-resolution view synthesis at test time. We use the Objaverse for
+training and the Google Scanned Objects for evaluation with standard novel view
+synthesis and 3D reconstruction metrics, where MVDiffusion++ significantly
+outperforms the current state of the arts. We also demonstrate a text-to-3D
+application example by combining MVDiffusion++ with a text-to-image generative
+model. The project page is at https://mvdiffusion-plusplus.github.io.",cs.CV,['cs.CV']
+HomoFormer: Homogenized Transformer for Image Shadow Removal,Jie Xiao · Xueyang Fu · Yurui Zhu · Dong Li · Jie Huang · Kai Zhu · Zheng-Jun Zha, ,https://arxiv.org/abs/2404.18433,,2404.18433.pdf,ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal,"Transformer recently emerged as the de facto model for computer vision tasks
+and has also been successfully applied to shadow removal. However, these
+existing methods heavily rely on intricate modifications to the attention
+mechanisms within the transformer blocks while using a generic patch embedding.
+As a result, it often leads to complex architectural designs requiring
+additional computation resources. In this work, we aim to explore the efficacy
+of incorporating shadow information within the early processing stage.
+Accordingly, we propose a transformer-based framework with a novel patch
+embedding that is tailored for shadow removal, dubbed ShadowMaskFormer.
+Specifically, we present a simple and effective mask-augmented patch embedding
+to integrate shadow information and promote the model's emphasis on acquiring
+knowledge for shadow regions. Extensive experiments conducted on the ISTD,
+ISTD+, and SRD benchmark datasets demonstrate the efficacy of our method
+against state-of-the-art approaches while using fewer model parameters.",cs.CV,['cs.CV']
+Novel Class Discovery for Ultra-Fine-Grained Visual Categorization,Qi Jia · Yaqi Cai · Qi Jia · Binglin Qiu · Weimin Wang · Nan Pu,https://github.com/SSDUT-Caiyq/UFG-NCD,https://arxiv.org/abs/2405.06283,,2405.06283.pdf,Novel Class Discovery for Ultra-Fine-Grained Visual Categorization,"Ultra-fine-grained visual categorization (Ultra-FGVC) aims at distinguishing
+highly similar sub-categories within fine-grained objects, such as different
+soybean cultivars. Compared to traditional fine-grained visual categorization,
+Ultra-FGVC encounters more hurdles due to the small inter-class and large
+intra-class variation. Given these challenges, relying on human annotation for
+Ultra-FGVC is impractical. To this end, our work introduces a novel task termed
+Ultra-Fine-Grained Novel Class Discovery (UFG-NCD), which leverages partially
+annotated data to identify new categories of unlabeled images for Ultra-FGVC.
+To tackle this problem, we devise a Region-Aligned Proxy Learning (RAPL)
+framework, which comprises a Channel-wise Region Alignment (CRA) module and a
+Semi-Supervised Proxy Learning (SemiPL) strategy. The CRA module is designed to
+extract and utilize discriminative features from local regions, facilitating
+knowledge transfer from labeled to unlabeled classes. Furthermore, SemiPL
+strengthens representation learning and knowledge transfer with proxy-guided
+supervised learning and proxy-guided contrastive learning. Such techniques
+leverage class distribution information in the embedding space, improving the
+mining of subtle differences between labeled and unlabeled ultra-fine-grained
+classes. Extensive experiments demonstrate that RAPL significantly outperforms
+baselines across various datasets, indicating its effectiveness in handling the
+challenges of UFG-NCD. Code is available at
+https://github.com/SSDUT-Caiyq/UFG-NCD.",cs.CV,['cs.CV']
+RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method,Ming Yan · Yan Zhang · Shuqiang Cai · Shuqi Fan · Xincheng Lin · Yudi Dai · Siqi Shen · Chenglu Wen · Lan Xu · Yuexin Ma · Cheng Wang,http://www.lidarhumanmotion.net/reli11d/,https://arxiv.org/abs/2403.19501,,2403.19501.pdf,RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method,"Comprehensive capturing of human motions requires both accurate captures of
+complex poses and precise localization of the human within scenes. Most of the
+HPE datasets and methods primarily rely on RGB, LiDAR, or IMU data. However,
+solely using these modalities or a combination of them may not be adequate for
+HPE, particularly for complex and fast movements. For holistic human motion
+understanding, we present RELI11D, a high-quality multimodal human motion
+dataset involves LiDAR, IMU system, RGB camera, and Event camera. It records
+the motions of 10 actors performing 5 sports in 7 scenes, including 3.32 hours
+of synchronized LiDAR point clouds, IMU measurement data, RGB videos and Event
+steams. Through extensive experiments, we demonstrate that the RELI11D presents
+considerable challenges and opportunities as it contains many rapid and complex
+motions that require precise location. To address the challenge of integrating
+different modalities, we propose LEIR, a multimodal baseline that effectively
+utilizes LiDAR Point Cloud, Event stream, and RGB through our cross-attention
+fusion strategy. We show that LEIR exhibits promising results for rapid motions
+and daily motions and that utilizing the characteristics of multiple modalities
+can indeed improve HPE performance. Both the dataset and source code will be
+released publicly to the research community, fostering collaboration and
+enabling further exploration in this field.",cs.CV,['cs.CV']
+Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation guided by the Characteristic Dance Primitives,Ronghui Li · Yuxiang Zhang · Yachao Zhang · Hongwen Zhang · Jie Guo · Yan Zhang · Yebin Liu · Xiu Li,https://li-ronghui.github.io/lodge,https://arxiv.org/abs/2403.10518,,2403.10518.pdf,Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives,"We propose Lodge, a network capable of generating extremely long dance
+sequences conditioned on given music. We design Lodge as a two-stage coarse to
+fine diffusion architecture, and propose the characteristic dance primitives
+that possess significant expressiveness as intermediate representations between
+two diffusion models. The first stage is global diffusion, which focuses on
+comprehending the coarse-level music-dance correlation and production
+characteristic dance primitives. In contrast, the second-stage is the local
+diffusion, which parallelly generates detailed motion sequences under the
+guidance of the dance primitives and choreographic rules. In addition, we
+propose a Foot Refine Block to optimize the contact between the feet and the
+ground, enhancing the physical realism of the motion. Our approach can
+parallelly generate dance sequences of extremely long length, striking a
+balance between global choreographic patterns and local motion quality and
+expressiveness. Extensive experiments validate the efficacy of our method.",cs.CV,"['cs.CV', 'cs.GR', 'cs.SD', 'eess.AS']"
+ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models,Meng-Li Shih · Wei-Chiu Ma · Lorenzo Boyice · Aleksander Holynski · Forrester Cole · Brian Curless · Janne Kontkanen, ,https://arxiv.org/abs/2401.00979,,2401.00979.pdf,3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands,"Neural radiance fields (NeRFs) are promising 3D representations for scenes,
+objects, and humans. However, most existing methods require multi-view inputs
+and per-scene training, which limits their real-life applications. Moreover,
+current methods focus on single-subject cases, leaving scenes of interacting
+hands that involve severe inter-hand occlusions and challenging view variations
+remain unsolved. To tackle these issues, this paper proposes a generalizable
+visibility-aware NeRF (VA-NeRF) framework for interacting hands. Specifically,
+given an image of interacting hands as input, our VA-NeRF first obtains a
+mesh-based representation of hands and extracts their corresponding geometric
+and textural features. Subsequently, a feature fusion module that exploits the
+visibility of query points and mesh vertices is introduced to adaptively merge
+features of both hands, enabling the recovery of features in unseen areas.
+Additionally, our VA-NeRF is optimized together with a novel discriminator
+within an adversarial learning paradigm. In contrast to conventional
+discriminators that predict a single real/fake label for the synthesized image,
+the proposed discriminator generates a pixel-wise visibility map, providing
+fine-grained supervision for unseen areas and encouraging the VA-NeRF to
+improve the visual quality of synthesized images. Experiments on the
+Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms
+conventional NeRFs significantly. Project Page:
+\url{https://github.com/XuanHuang0/VANeRF}.",cs.CV,['cs.CV']
+"Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision",Mohammad Reza Hosseinzadeh Taher · Michael Gotway · Jianming Liang, ,https://arxiv.org/abs/2404.15672,,2404.15672.pdf,"Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision","Humans effortlessly interpret images by parsing them into part-whole
+hierarchies; deep learning excels in learning multi-level feature spaces, but
+they often lack explicit coding of part-whole relations, a prominent property
+of medical imaging. To overcome this limitation, we introduce Adam-v2, a new
+self-supervised learning framework extending Adam [79] by explicitly
+incorporating part-whole hierarchies into its learning objectives through three
+key branches: (1) Localizability, acquiring discriminative representations to
+distinguish different anatomical patterns; (2) Composability, learning each
+anatomical structure in a parts-to-whole manner; and (3) Decomposability,
+comprehending each anatomical structure in a whole-to-parts manner.
+Experimental results across 10 tasks, compared to 11 baselines in zero-shot,
+few-shot transfer, and full fine-tuning settings, showcase Adam-v2's superior
+performance over large-scale medical models and existing SSL methods across
+diverse downstream tasks. The higher generality and robustness of Adam-v2's
+representations originate from its explicit construction of hierarchies for
+distinct anatomical structures from unlabeled medical images. Adam-v2 preserves
+a semantic balance of anatomical diversity and harmony in its embedding,
+yielding representations that are both generic and semantically meaningful, yet
+overlooked in existing SSL methods. All code and pretrained models are
+available at https://github.com/JLiangLab/Eden.",cs.CV,['cs.CV']
+Robust Synthetic-to-Real Transfer for Stereo Matching,Jiawei Zhang · Jiahe Li · Lei Huang · Xiaohan Yu · Lin Gu · Jin Zheng · Xiao Bai, ,https://arxiv.org/abs/2403.07705,,2403.07705.pdf,Robust Synthetic-to-Real Transfer for Stereo Matching,"With advancements in domain generalized stereo matching networks, models
+pre-trained on synthetic data demonstrate strong robustness to unseen domains.
+However, few studies have investigated the robustness after fine-tuning them in
+real-world scenarios, during which the domain generalization ability can be
+seriously degraded. In this paper, we explore fine-tuning stereo matching
+networks without compromising their robustness to unseen domains. Our
+motivation stems from comparing Ground Truth (GT) versus Pseudo Label (PL) for
+fine-tuning: GT degrades, but PL preserves the domain generalization ability.
+Empirically, we find the difference between GT and PL implies valuable
+information that can regularize networks during fine-tuning. We also propose a
+framework to utilize this difference for fine-tuning, consisting of a frozen
+Teacher, an exponential moving average (EMA) Teacher, and a Student network.
+The core idea is to utilize the EMA Teacher to measure what the Student has
+learned and dynamically improve GT and PL for fine-tuning. We integrate our
+framework with state-of-the-art networks and evaluate its effectiveness on
+several real-world datasets. Extensive experiments show that our method
+effectively preserves the domain generalization ability during fine-tuning.",cs.CV,['cs.CV']
+Towards Robust Learning to Optimize with Theoretical Guarantees,Qingyu Song · Wei Lin · Juncheng Wang · Hong Xu,https://github.com/NetX-lab/GoMathL2O-Official,,https://henryhxu.github.io/papers.html,,,,,nan
+UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs,Yanwu Xu · Yang Zhao · Zhisheng Xiao · Tingbo Hou, ,https://arxiv.org/abs/2311.09257,,2311.09257.pdf,UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs,"Text-to-image diffusion models have demonstrated remarkable capabilities in
+transforming textual prompts into coherent images, yet the computational cost
+of their inference remains a persistent challenge. To address this issue, we
+present UFOGen, a novel generative model designed for ultra-fast, one-step
+text-to-image synthesis. In contrast to conventional approaches that focus on
+improving samplers or employing distillation techniques for diffusion models,
+UFOGen adopts a hybrid methodology, integrating diffusion models with a GAN
+objective. Leveraging a newly introduced diffusion-GAN objective and
+initialization with pre-trained diffusion models, UFOGen excels in efficiently
+generating high-quality images conditioned on textual descriptions in a single
+step. Beyond traditional text-to-image generation, UFOGen showcases versatility
+in applications. Notably, UFOGen stands among the pioneering models enabling
+one-step text-to-image generation and diverse downstream tasks, presenting a
+significant advancement in the landscape of efficient generative models.",cs.CV,['cs.CV']
+Instance-aware Contrastive Learning for Occluded Human Mesh Reconstruction,Mi-Gyeong Gwon · Gi-Mun Um · Won-Sik Cheong · Wonjun Kim,https://github.com/DCVL-3D/InstanceHMR_release,https://arxiv.org/abs/2307.16377,,2307.16377.pdf,JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery,"In this study, we focus on the problem of 3D human mesh recovery from a
+single image under obscured conditions. Most state-of-the-art methods aim to
+improve 2D alignment technologies, such as spatial averaging and 2D joint
+sampling. However, they tend to neglect the crucial aspect of 3D alignment by
+improving 3D representations. Furthermore, recent methods struggle to separate
+the target human from occlusion or background in crowded scenes as they
+optimize the 3D space of target human with 3D joint coordinates as local
+supervision. To address these issues, a desirable method would involve a
+framework for fusing 2D and 3D features and a strategy for optimizing the 3D
+space globally. Therefore, this paper presents 3D JOint contrastive learning
+with TRansformers (JOTR) framework for handling occluded 3D human mesh
+recovery. Our method includes an encoder-decoder transformer architecture to
+fuse 2D and 3D representations for achieving 2D$\&$3D aligned results in a
+coarse-to-fine manner and a novel 3D joint contrastive learning approach for
+adding explicitly global supervision for the 3D feature space. The contrastive
+learning approach includes two contrastive losses: joint-to-joint contrast for
+enhancing the similarity of semantically similar voxels (i.e., human joints),
+and joint-to-non-joint contrast for ensuring discrimination from others (e.g.,
+occlusions and background). Qualitative and quantitative analyses demonstrate
+that our method outperforms state-of-the-art competitors on both
+occlusion-specific and standard benchmarks, significantly improving the
+reconstruction of occluded humans.",cs.CV,['cs.CV']
+CCEdit: Creative and Controllable Video Editing via Diffusion Models,Ruoyu Feng · Wenming Weng · Yanhui Wang · Yuhui Yuan · Jianmin Bao · Chong Luo · Zhibo Chen · Baining Guo, ,https://arxiv.org/abs/2309.16496,,2309.16496.pdf,CCEdit: Creative and Controllable Video Editing via Diffusion Models,"In this paper, we present CCEdit, a versatile generative video editing
+framework based on diffusion models. Our approach employs a novel trident
+network structure that separates structure and appearance control, ensuring
+precise and creative editing capabilities. Utilizing the foundational
+ControlNet architecture, we maintain the structural integrity of the video
+during editing. The incorporation of an additional appearance branch enables
+users to exert fine-grained control over the edited key frame. These two side
+branches seamlessly integrate into the main branch, which is constructed upon
+existing text-to-image (T2I) generation models, through learnable temporal
+layers. The versatility of our framework is demonstrated through a diverse
+range of choices in both structure representations and personalized T2I models,
+as well as the option to provide the edited key frame. To facilitate
+comprehensive evaluation, we introduce the BalanceCC benchmark dataset,
+comprising 100 videos and 4 target prompts for each video. Our extensive user
+studies compare CCEdit with eight state-of-the-art video editing methods. The
+outcomes demonstrate CCEdit's substantial superiority over all other methods.",cs.CV,['cs.CV']
+CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoor Object Detection from Multi-view Images,Guanlin Shen · Jingwei Huang · Zhihua Hu · Bin Wang,https://github.com/SerCharles/CN-RMA,https://arxiv.org/abs/2403.04198,,2403.04198.pdf,CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoors Object Detection from Multi-view Images,"This paper introduces CN-RMA, a novel approach for 3D indoor object detection
+from multi-view images. We observe the key challenge as the ambiguity of image
+and 3D correspondence without explicit geometry to provide occlusion
+information. To address this issue, CN-RMA leverages the synergy of 3D
+reconstruction networks and 3D object detection networks, where the
+reconstruction network provides a rough Truncated Signed Distance Function
+(TSDF) and guides image features to vote to 3D space correctly in an end-to-end
+manner. Specifically, we associate weights to sampled points of each ray
+through ray marching, representing the contribution of a pixel in an image to
+corresponding 3D locations. Such weights are determined by the predicted signed
+distances so that image features vote only to regions near the reconstructed
+surface. Our method achieves state-of-the-art performance in 3D object
+detection from multi-view images, as measured by mAP@0.25 and mAP@0.5 on the
+ScanNet and ARKitScenes datasets. The code and models are released at
+https://github.com/SerCharles/CN-RMA.",cs.CV,['cs.CV']
+One-Class Face Anti-spoofing via Spoof Cue Map-Guided Feature Learning,Pei-Kai Huang · Cheng-Hsuan Chiang · Tzu-Hsien Chen · Jun-Xiong Chong · Tyng-Luh Liu · Chiou-Ting Hsu, ,,https://link.springer.com/article/10.1007/s11042-023-17739-y,,,,,nan
+PanoRecon: Real-Time Panoptic 3D Reconstruction from Monocular Video,Dong Wu · Zike Yan · Hongbin Zha, ,,,,,,,nan
+Grounding Everything: Emerging Localization Properties in Vision-Language Transformers,Walid Bousselham · Felix Petersen · Vittorio Ferrari · Hilde Kuehne, ,https://arxiv.org/abs/2312.00878,,2312.00878.pdf,Grounding Everything: Emerging Localization Properties in Vision-Language Transformers,"Vision-language foundation models have shown remarkable performance in
+various zero-shot settings such as image retrieval, classification, or
+captioning. But so far, those models seem to fall behind when it comes to
+zero-shot localization of referential expressions and objects in images. As a
+result, they need to be fine-tuned for this task. In this paper, we show that
+pretrained vision-language (VL) models allow for zero-shot open-vocabulary
+object localization without any fine-tuning. To leverage those capabilities, we
+propose a Grounding Everything Module (GEM) that generalizes the idea of
+value-value attention introduced by CLIPSurgery to a self-self attention path.
+We show that the concept of self-self attention corresponds to clustering, thus
+enforcing groups of tokens arising from the same object to be similar while
+preserving the alignment with the language space. To further guide the group
+formation, we propose a set of regularizations that allows the model to finally
+generalize across datasets and backbones. We evaluate the proposed GEM
+framework on various benchmark tasks and datasets for semantic segmentation. It
+shows that GEM not only outperforms other training-free open-vocabulary
+localization methods, but also achieves state-of-the-art results on the
+recently proposed OpenImagesV7 large-scale segmentation benchmark.",cs.CV,"['cs.CV', 'cs.AI']"
+Brain Decodes Deep Nets,Huzheng Yang · James Gee · Jianbo Shi,https://huzeyann.github.io/brain-decodes-deep-nets,https://arxiv.org/abs/2312.01280,,2312.01280.pdf,Brain Decodes Deep Nets,"We developed a tool for visualizing and analyzing large pre-trained vision
+models by mapping them onto the brain, thus exposing their hidden inside. Our
+innovation arises from a surprising usage of brain encoding: predicting brain
+fMRI measurements in response to images. We report two findings. First,
+explicit mapping between the brain and deep-network features across dimensions
+of space, layers, scales, and channels is crucial. This mapping method,
+FactorTopy, is plug-and-play for any deep-network; with it, one can paint a
+picture of the network onto the brain (literally!). Second, our visualization
+shows how different training methods matter: they lead to remarkable
+differences in hierarchical organization and scaling behavior, growing with
+more data or network capacity. It also provides insight into fine-tuning: how
+pre-trained models change when adapting to small datasets. We found brain-like
+hierarchically organized network suffer less from catastrophic forgetting after
+fine-tuned.",cs.CV,['cs.CV']
+Revisiting Spatial-Frequency Information Integration from a Hierarchical Perspective for Panchromatic and Multi-Spectral Image Fusion,Jiangtong Tan · Jie Huang · Naishan Zheng · Man Zhou · Keyu Yan · Danfeng Hong · Feng Zhao, ,,https://ieeexplore.ieee.org/document/10443302,,,,,nan
+Prompting Vision Foundation Models for Pathology Image Analysis,CHONG YIN · Siqi Liu · Kaiyang Zhou · Vincent Wong · Pong C. Yuen, ,https://arxiv.org/abs/2403.16497,,2403.16497.pdf,PathoTune: Adapting Visual Foundation Model to Pathological Specialists,"As natural image understanding moves towards the pretrain-finetune era,
+research in pathology imaging is concurrently evolving. Despite the predominant
+focus on pretraining pathological foundation models, how to adapt foundation
+models to downstream tasks is little explored. For downstream adaptation, we
+propose the existence of two domain gaps, i.e., the Foundation-Task Gap and the
+Task-Instance Gap. To mitigate these gaps, we introduce PathoTune, a framework
+designed to efficiently adapt pathological or even visual foundation models to
+pathology-specific tasks via multi-modal prompt tuning. The proposed framework
+leverages Task-specific Visual Prompts and Task-specific Textual Prompts to
+identify task-relevant features, along with Instance-specific Visual Prompts
+for encoding single pathological image features. Results across multiple
+datasets at both patch-level and WSI-level demonstrate its superior performance
+over single-modality prompt tuning approaches. Significantly, PathoTune
+facilitates the direct adaptation of natural visual foundation models to
+pathological tasks, drastically outperforming pathological foundation models
+with simple linear probing. The code will be available upon acceptance.",cs.CV,"['cs.CV', 'cs.LG']"
+DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization,Jiahe Li · Jiawei Zhang · Xiao Bai · Jin Zheng · Xin Ning · Jun Zhou · Lin Gu,https://fictionarry.github.io/DNGaussian/,https://arxiv.org/abs/2403.06912,,2403.06912.pdf,DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization,"Radiance fields have demonstrated impressive performance in synthesizing
+novel views from sparse input views, yet prevailing methods suffer from high
+training costs and slow inference speed. This paper introduces DNGaussian, a
+depth-regularized framework based on 3D Gaussian radiance fields, offering
+real-time and high-quality few-shot novel view synthesis at low costs. Our
+motivation stems from the highly efficient representation and surprising
+quality of the recent 3D Gaussian Splatting, despite it will encounter a
+geometry degradation when input views decrease. In the Gaussian radiance
+fields, we find this degradation in scene geometry primarily lined to the
+positioning of Gaussian primitives and can be mitigated by depth constraint.
+Consequently, we propose a Hard and Soft Depth Regularization to restore
+accurate scene geometry under coarse monocular depth supervision while
+maintaining a fine-grained color appearance. To further refine detailed
+geometry reshaping, we introduce Global-Local Depth Normalization, enhancing
+the focus on small local depth changes. Extensive experiments on LLFF, DTU, and
+Blender datasets demonstrate that DNGaussian outperforms state-of-the-art
+methods, achieving comparable or better results with significantly reduced
+memory cost, a $25 \times$ reduction in training time, and over $3000 \times$
+faster rendering speed.",cs.CV,['cs.CV']
+Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation,Xiao Ma · Sumit Patidar · Iain Haughton · Stephen James,https://yusufma03.github.io/projects/hdp/,https://arxiv.org/abs/2403.03890v1,,2403.03890v1.pdf,Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation,"This paper introduces Hierarchical Diffusion Policy (HDP), a hierarchical
+agent for multi-task robotic manipulation. HDP factorises a manipulation policy
+into a hierarchical structure: a high-level task-planning agent which predicts
+a distant next-best end-effector pose (NBP), and a low-level goal-conditioned
+diffusion policy which generates optimal motion trajectories. The factorised
+policy representation allows HDP to tackle both long-horizon task planning
+while generating fine-grained low-level actions. To generate context-aware
+motion trajectories while satisfying robot kinematics constraints, we present a
+novel kinematics-aware goal-conditioned control agent, Robot Kinematics
+Diffuser (RK-Diffuser). Specifically, RK-Diffuser learns to generate both the
+end-effector pose and joint position trajectories, and distill the accurate but
+kinematics-unaware end-effector pose diffuser to the kinematics-aware but less
+accurate joint position diffuser via differentiable kinematics. Empirically, we
+show that HDP achieves a significantly higher success rate than the
+state-of-the-art methods in both simulation and real-world.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV', 'cs.LG']"
+Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition,Yifei Chen · Dapeng Chen · Ruijin Liu · Sai Zhou · Wenyuan Xue · Wei Peng, ,https://arxiv.org/abs/2311.15619,,2311.15619.pdf,Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition,"Large-scale visual-language pre-trained models have achieved significant
+success in various video tasks. However, most existing methods follow an ""adapt
+then align"" paradigm, which adapts pre-trained image encoders to model
+video-level representations and utilizes one-hot or text embedding of the
+action labels for supervision. This paradigm overlooks the challenge of mapping
+from static images to complicated activity concepts. In this paper, we propose
+a novel ""Align before Adapt"" (ALT) paradigm. Prior to adapting to video
+representation learning, we exploit the entity-to-region alignments for each
+frame. The alignments are fulfilled by matching the region-aware image
+embeddings to an offline-constructed text corpus. With the aligned entities, we
+feed their text embeddings to a transformer-based video adapter as the queries,
+which can help extract the semantics of the most important entities from a
+video to a vector. This paradigm reuses the visual-language alignment of VLP
+during adaptation and tries to explain an action by the underlying entities.
+This helps understand actions by bridging the gap with complex activity
+semantics, particularly when facing unfamiliar or unseen categories. ALT
+demonstrates competitive performance while maintaining remarkably low
+computational costs. In fully supervised experiments, it achieves 88.1% top-1
+accuracy on Kinetics-400 with only 4947 GFLOPs. Moreover, ALT outperforms the
+previous state-of-the-art methods in both zero-shot and few-shot experiments,
+emphasizing its superior generalizability across various learning scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+Discriminative Probing and Tuning for Text-to-Image Generation,Leigang Qu · Wenjie Wang · Yongqi Li · Hanwang Zhang · Liqiang Nie · Tat-seng Chua,https://dpt-t2i.github.io/,https://arxiv.org/abs/2403.04321,,2403.04321.pdf,Discriminative Probing and Tuning for Text-to-Image Generation,"Despite advancements in text-to-image generation (T2I), prior methods often
+face text-image misalignment problems such as relation confusion in generated
+images. Existing solutions involve cross-attention manipulation for better
+compositional understanding or integrating large language models for improved
+layout planning. However, the inherent alignment capabilities of T2I models are
+still inadequate. By reviewing the link between generative and discriminative
+modeling, we posit that T2I models' discriminative abilities may reflect their
+text-image alignment proficiency during generation. In this light, we advocate
+bolstering the discriminative abilities of T2I models to achieve more precise
+text-to-image alignment for generation. We present a discriminative adapter
+built on T2I models to probe their discriminative abilities on two
+representative tasks and leverage discriminative fine-tuning to improve their
+text-image alignment. As a bonus of the discriminative adapter, a
+self-correction mechanism can leverage discriminative gradients to better align
+generated images to text prompts during inference. Comprehensive evaluations
+across three benchmark datasets, including both in-distribution and
+out-of-distribution scenarios, demonstrate our method's superior generation
+performance. Meanwhile, it achieves state-of-the-art discriminative performance
+on the two discriminative tasks compared to other generative models.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.MM']"
+CLIP-KD: An Empirical Study of CLIP Model Distillation,Chuanguang Yang · Zhulin An · Libo Huang · Junyu Bi · XinQiang Yu · Han Yang · boyu diao · Yongjun Xu, ,https://arxiv.org/abs/2307.12732,,2307.12732.pdf,CLIP-KD: An Empirical Study of CLIP Model Distillation,"Contrastive Language-Image Pre-training (CLIP) has become a promising
+language-supervised visual pre-training framework. This paper aims to distill
+small CLIP models supervised by a large teacher CLIP model. We propose several
+distillation strategies, including relation, feature, gradient and contrastive
+paradigms, to examine the effectiveness of CLIP-Knowledge Distillation (KD). We
+show that a simple feature mimicry with Mean Squared Error loss works
+surprisingly well. Moreover, interactive contrastive learning across teacher
+and student encoders is also effective in performance improvement. We explain
+that the success of CLIP-KD can be attributed to maximizing the feature
+similarity between teacher and student. The unified method is applied to
+distill several student models trained on CC3M+12M. CLIP-KD improves student
+CLIP models consistently over zero-shot ImageNet classification and cross-modal
+retrieval benchmarks. When using ViT-L/14 pretrained on Laion-400M as the
+teacher, CLIP-KD achieves 57.5\% and 55.4\% zero-shot top-1 ImageNet accuracy
+over ViT-B/16 and ResNet-50, surpassing the original CLIP without KD by 20.5\%
+and 20.1\% margins, respectively. Our code is released on
+https://github.com/winycg/CLIP-KD.",cs.CV,['cs.CV']
+FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation,Shuai Yang · Yifan Zhou · Ziwei Liu · Chen Change Loy,https://www.mmlab-ntu.com/project/fresco/,https://arxiv.org/abs/2403.12962,,2403.12962.pdf,FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation,"The remarkable efficacy of text-to-image diffusion models has motivated
+extensive exploration of their potential application in video domains.
+Zero-shot methods seek to extend image diffusion models to videos without
+necessitating model training. Recent methods mainly focus on incorporating
+inter-frame correspondence into attention mechanisms. However, the soft
+constraint imposed on determining where to attend to valid features can
+sometimes be insufficient, resulting in temporal inconsistency. In this paper,
+we introduce FRESCO, intra-frame correspondence alongside inter-frame
+correspondence to establish a more robust spatial-temporal constraint. This
+enhancement ensures a more consistent transformation of semantically similar
+content across frames. Beyond mere attention guidance, our approach involves an
+explicit update of features to achieve high spatial-temporal consistency with
+the input video, significantly improving the visual coherence of the resulting
+translated videos. Extensive experiments demonstrate the effectiveness of our
+proposed framework in producing high-quality, coherent videos, marking a
+notable improvement over existing zero-shot methods.",cs.CV,['cs.CV']
+XFibrosis: Explicit Vessel-Fiber Modeling for Fibrosis Staging from Liver Pathology Images,CHONG YIN · Siqi Liu · Fei Lyu · Jiahao Lu · Sune Darkner · Vincent Wong · Pong C. Yuen, ,,https://www.youtube.com/watch?v=_Yiu5g71ZHo,,,,,nan
+LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model,Dongkai Wang · shiyu xuan · Shiliang Zhang, ,https://arxiv.org/abs/2310.00582,,2310.00582.pdf,Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs,"Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities
+in various multi-modal tasks. Nevertheless, their performance in fine-grained
+image understanding tasks is still limited. To address this issue, this paper
+proposes a new framework to enhance the fine-grained image understanding
+abilities of MLLMs. Specifically, we present a new method for constructing the
+instruction tuning dataset at a low cost by leveraging annotations in existing
+datasets. A self-consistent bootstrapping method is also introduced to extend
+existing dense object annotations into high-quality
+referring-expression-bounding-box pairs. These methods enable the generation of
+high-quality instruction data which includes a wide range of fundamental
+abilities essential for fine-grained image perception. Moreover, we argue that
+the visual encoder should be tuned during instruction tuning to mitigate the
+gap between full image perception and fine-grained image perception.
+Experimental results demonstrate the superior performance of our method. For
+instance, our model exhibits a 5.2% accuracy improvement over Qwen-VL on GQA
+and surpasses the accuracy of Kosmos-2 by 24.7% on RefCOCO_val. We have also
+attained the top rank on the leaderboard of MMBench. This promising performance
+is achieved by training on only publicly available data, making it easily
+reproducible. The models, datasets, and codes are publicly available at
+https://github.com/SY-Xuan/Pink.",cs.CV,"['cs.CV', 'cs.AI']"
+Producing and Leveraging Online Map Uncertainty in Trajectory Prediction,Xunjiang Gu · Guanyu Song · Igor Gilitschenski · Marco Pavone · Boris Ivanovic,https://github.com/alfredgu001324/MapUncertaintyPrediction,https://arxiv.org/abs/2403.16439v1,,2403.16439v1.pdf,Producing and Leveraging Online Map Uncertainty in Trajectory Prediction,"High-definition (HD) maps have played an integral role in the development of
+modern autonomous vehicle (AV) stacks, albeit with high associated labeling and
+maintenance costs. As a result, many recent works have proposed methods for
+estimating HD maps online from sensor data, enabling AVs to operate outside of
+previously-mapped regions. However, current online map estimation approaches
+are developed in isolation of their downstream tasks, complicating their
+integration in AV stacks. In particular, they do not produce uncertainty or
+confidence estimates. In this work, we extend multiple state-of-the-art online
+map estimation methods to additionally estimate uncertainty and show how this
+enables more tightly integrating online mapping with trajectory forecasting. In
+doing so, we find that incorporating uncertainty yields up to 50% faster
+training convergence and up to 15% better prediction performance on the
+real-world nuScenes driving dataset.",cs.RO,"['cs.RO', 'cs.CV', 'cs.LG']"
+GenZI: Zero-Shot 3D Human-Scene Interaction Generation,Lei Li · Angela Dai,https://craigleili.github.io/projects/genzi/,https://arxiv.org/abs/2311.17737,,2311.17737.pdf,GenZI: Zero-Shot 3D Human-Scene Interaction Generation,"Can we synthesize 3D humans interacting with scenes without learning from any
+3D human-scene interaction data? We propose GenZI, the first zero-shot approach
+to generating 3D human-scene interactions. Key to GenZI is our distillation of
+interaction priors from large vision-language models (VLMs), which have learned
+a rich semantic space of 2D human-scene compositions. Given a natural language
+description and a coarse point location of the desired interaction in a 3D
+scene, we first leverage VLMs to imagine plausible 2D human interactions
+inpainted into multiple rendered views of the scene. We then formulate a robust
+iterative optimization to synthesize the pose and shape of a 3D human model in
+the scene, guided by consistency with the 2D interaction hypotheses. In
+contrast to existing learning-based approaches, GenZI circumvents the
+conventional need for captured 3D interaction data, and allows for flexible
+control of the 3D interaction synthesis with easy-to-use text prompts.
+Extensive experiments show that our zero-shot approach has high flexibility and
+generality, making it applicable to diverse scene types, including both indoor
+and outdoor environments.",cs.CV,"['cs.CV', 'cs.GR']"
+LPSNet: End-to-End Human Pose and Shape Estimation with Lensless Imaging,Haoyang Ge · Qiao Feng · Hailong Jia · Xiongzheng Li · Xiangjun Yin · You Zhou · Jingyu Yang · Kun Li,https://cic.tju.edu.cn/faculty/likun/projects/LPSNet/index.html,https://arxiv.org/abs/2404.01941,,2404.01941.pdf,LPSNet: End-to-End Human Pose and Shape Estimation with Lensless Imaging,"Human pose and shape (HPS) estimation with lensless imaging is not only
+beneficial to privacy protection but also can be used in covert surveillance
+scenarios due to the small size and simple structure of this device. However,
+this task presents significant challenges due to the inherent ambiguity of the
+captured measurements and lacks effective methods for directly estimating human
+pose and shape from lensless data. In this paper, we propose the first
+end-to-end framework to recover 3D human poses and shapes from lensless
+measurements to our knowledge. We specifically design a multi-scale lensless
+feature decoder to decode the lensless measurements through the optically
+encoded mask for efficient feature extraction. We also propose a double-head
+auxiliary supervision mechanism to improve the estimation accuracy of human
+limb ends. Besides, we establish a lensless imaging system and verify the
+effectiveness of our method on various datasets acquired by our lensless
+imaging system.",cs.CV,['cs.CV']
+Can I Trust Your Answer? Visually Grounded Video Question Answering,Junbin Xiao · Angela Yao · Yicong Li · Tat-seng Chua, ,https://arxiv.org/abs/2309.01327,,2309.01327.pdf,Can I Trust Your Answer? Visually Grounded Video Question Answering,"We study visually grounded VideoQA in response to the emerging trends of
+utilizing pretraining techniques for video-language understanding.
+Specifically, by forcing vision-language models (VLMs) to answer questions and
+simultaneously provide visual evidence, we seek to ascertain the extent to
+which the predictions of such techniques are genuinely anchored in relevant
+video content, versus spurious correlations from language or irrelevant visual
+context. Towards this, we construct NExT-GQA -- an extension of NExT-QA with
+10.5$K$ temporal grounding (or location) labels tied to the original QA pairs.
+With NExT-GQA, we scrutinize a series of state-of-the-art VLMs. Through
+post-hoc attention analysis, we find that these models are extremely weak in
+substantiating the answers despite their strong QA performance. This exposes
+the limitation of current VLMs in making reliable predictions. As a remedy, we
+further explore and propose a grounded-QA method via Gaussian mask optimization
+and cross-modal learning. Experiments with different backbones demonstrate that
+this grounding mechanism improves both grounding and QA. With these efforts, we
+aim to push towards trustworthy VLMs in VQA systems. Our dataset and code are
+available at https://github.com/doc-doc/NExT-GQA.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM']"
+RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction,Baptiste Brument · Robin Bruneau · Yvain Queau · Jean Mélou · Francois Lauze · Jean-Denis Durou · Lilian Calvet,https://robinbruneau.github.io/publications/rnb_neus.html,https://arxiv.org/abs/2312.01215,,2312.01215.pdf,RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction,"This paper introduces a versatile paradigm for integrating multi-view
+reflectance (optional) and normal maps acquired through photometric stereo. Our
+approach employs a pixel-wise joint re-parameterization of reflectance and
+normal, considering them as a vector of radiances rendered under simulated,
+varying illumination. This re-parameterization enables the seamless integration
+of reflectance and normal maps as input data in neural volume rendering-based
+3D reconstruction while preserving a single optimization objective. In
+contrast, recent multi-view photometric stereo (MVPS) methods depend on
+multiple, potentially conflicting objectives. Despite its apparent simplicity,
+our proposed approach outperforms state-of-the-art approaches in MVPS
+benchmarks across F-score, Chamfer distance, and mean angular error metrics.
+Notably, it significantly improves the detailed 3D reconstruction of areas with
+high curvature or low visibility.",cs.CV,['cs.CV']
+Multimodal Sense-Informed Prediction of 3D Human Motions,Zhenyu Lou · Qiongjie Cui · Haofan Wang · Xu Tang · Hong Zhou, ,https://arxiv.org/abs/2405.02911,,2405.02911.pdf,Multimodal Sense-Informed Prediction of 3D Human Motions,"Predicting future human pose is a fundamental application for machine
+intelligence, which drives robots to plan their behavior and paths ahead of
+time to seamlessly accomplish human-robot collaboration in real-world 3D
+scenarios. Despite encouraging results, existing approaches rarely consider the
+effects of the external scene on the motion sequence, leading to pronounced
+artifacts and physical implausibilities in the predictions. To address this
+limitation, this work introduces a novel multi-modal sense-informed motion
+prediction approach, which conditions high-fidelity generation on two modal
+information: external 3D scene, and internal human gaze, and is able to
+recognize their salience for future human activity. Furthermore, the gaze
+information is regarded as the human intention, and combined with both motion
+and scene features, we construct a ternary intention-aware attention to
+supervise the generation to match where the human wants to reach. Meanwhile, we
+introduce semantic coherence-aware attention to explicitly distinguish the
+salient point clouds and the underlying ones, to ensure a reasonable
+interaction of the generated sequence with the 3D scene. On two real-world
+benchmarks, the proposed method achieves state-of-the-art performance both in
+3D human pose and trajectory prediction.",cs.CV,['cs.CV']
+SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge,Andong Wang · Bo Wu · Sunli Chen · Zhenfang Chen · Haotian Guan · Wei-Ning Lee · Li Erran Li · Chuang Gan, ,https://arxiv.org/abs/2405.09713,,2405.09713.pdf,SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge,"Learning commonsense reasoning from visual contexts and scenes in real-world
+is a crucial step toward advanced artificial intelligence. However, existing
+video reasoning benchmarks are still inadequate since they were mainly designed
+for factual or situated reasoning and rarely involve broader knowledge in the
+real world. Our work aims to delve deeper into reasoning evaluations,
+specifically within dynamic, open-world, and structured context knowledge. We
+propose a new benchmark (SOK-Bench), consisting of 44K questions and 10K
+situations with instance-level annotations depicted in the videos. The
+reasoning process is required to understand and apply situated knowledge and
+general knowledge for problem-solving. To create such a dataset, we propose an
+automatic and scalable generation method to generate question-answer pairs,
+knowledge graphs, and rationales by instructing the combinations of LLMs and
+MLLMs. Concretely, we first extract observable situated entities, relations,
+and processes from videos for situated knowledge and then extend to open-world
+knowledge beyond the visible content. The task generation is facilitated
+through multiple dialogues as iterations and subsequently corrected and refined
+by our designed self-promptings and demonstrations. With a corpus of both
+explicit situated facts and implicit commonsense, we generate associated
+question-answer pairs and reasoning processes, finally followed by manual
+reviews for quality assurance. We evaluated recent mainstream large
+vision-language models on the benchmark and found several insightful
+conclusions. For more information, please refer to our benchmark at
+www.bobbywu.com/SOKBench.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+SURE: SUrvey REcipes for building reliable and robust deep networks,Yuting Li · Yingyi Chen · Xuanlong Yu · Dexiong Chen · Xi Shen,https://yutingli0606.github.io/SURE/,https://arxiv.org/abs/2403.00543,,2403.00543.pdf,SURE: SUrvey REcipes for building reliable and robust deep networks,"In this paper, we revisit techniques for uncertainty estimation within deep
+neural networks and consolidate a suite of techniques to enhance their
+reliability. Our investigation reveals that an integrated application of
+diverse techniques--spanning model regularization, classifier and
+optimization--substantially improves the accuracy of uncertainty predictions in
+image classification tasks. The synergistic effect of these techniques
+culminates in our novel SURE approach. We rigorously evaluate SURE against the
+benchmark of failure prediction, a critical testbed for uncertainty estimation
+efficacy. Our results showcase a consistently better performance than models
+that individually deploy each technique, across various datasets and model
+architectures. When applied to real-world challenges, such as data corruption,
+label noise, and long-tailed class distribution, SURE exhibits remarkable
+robustness, delivering results that are superior or on par with current
+state-of-the-art specialized methods. Particularly on Animal-10N and Food-101N
+for learning with noisy labels, SURE achieves state-of-the-art performance
+without any task-specific adjustments. This work not only sets a new benchmark
+for robust uncertainty estimation but also paves the way for its application in
+diverse, real-world scenarios where reliability is paramount. Our code is
+available at \url{https://yutingli0606.github.io/SURE/}.",cs.CV,['cs.CV']
+"ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation",Yan Di · Chenyangguang Zhang · Chaowei Wang · Ruida Zhang · Guangyao Zhai · Yanyan Li · Bowen Fu · Xiangyang Ji · Shan Gao, ,https://arxiv.org/abs/2311.11106,,2311.11106.pdf,"ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation","In this paper, we present ShapeMatcher, a unified self-supervised learning
+framework for joint shape canonicalization, segmentation, retrieval and
+deformation. Given a partially-observed object in an arbitrary pose, we first
+canonicalize the object by extracting point-wise affine-invariant features,
+disentangling inherent structure of the object with its pose and size. These
+learned features are then leveraged to predict semantically consistent part
+segmentation and corresponding part centers. Next, our lightweight retrieval
+module aggregates the features within each part as its retrieval token and
+compare all the tokens with source shapes from a pre-established database to
+identify the most geometrically similar shape. Finally, we deform the retrieved
+shape in the deformation module to tightly fit the input object by harnessing
+part center guided neural cage deformation. The key insight of ShapeMaker is
+the simultaneous training of the four highly-associated processes:
+canonicalization, segmentation, retrieval, and deformation, leveraging
+cross-task consistency losses for mutual supervision. Extensive experiments on
+synthetic datasets PartNet, ComplementMe, and real-world dataset Scan2CAD
+demonstrate that ShapeMaker surpasses competitors by a large margin.",cs.CV,['cs.CV']
+DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars,Tobias Kirschstein · Simon Giebenhain · Matthias Nießner,https://tobias-kirschstein.github.io/diffusion-avatars/,https://arxiv.org/abs/2311.18635,,2311.18635.pdf,DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars,"DiffusionAvatars synthesizes a high-fidelity 3D head avatar of a person,
+offering intuitive control over both pose and expression. We propose a
+diffusion-based neural renderer that leverages generic 2D priors to produce
+compelling images of faces. For coarse guidance of the expression and head
+pose, we render a neural parametric head model (NPHM) from the target
+viewpoint, which acts as a proxy geometry of the person. Additionally, to
+enhance the modeling of intricate facial expressions, we condition
+DiffusionAvatars directly on the expression codes obtained from NPHM via
+cross-attention. Finally, to synthesize consistent surface details across
+different viewpoints and expressions, we rig learnable spatial features to the
+head's surface via TriPlane lookup in NPHM's canonical space. We train
+DiffusionAvatars on RGB videos and corresponding fitted NPHM meshes of a person
+and test the obtained avatars in both self-reenactment and animation scenarios.
+Our experiments demonstrate that DiffusionAvatars generates temporally
+consistent and visually appealing videos for novel poses and expressions of a
+person, outperforming existing approaches.",cs.CV,['cs.CV']
+PREGO: online mistake detection in PRocedural EGOcentric videos,Alessandro Flaborea · Guido M. D&amp;#x27;Amely di Melendugno · Leonardo Plini · Luca Scofano · Edoardo De Matteis · Antonino Furnari · Giovanni Maria Farinella · Fabio Galasso,https://github.com/aleflabo/PREGO,https://arxiv.org/abs/2404.01933,,,PREGO: online mistake detection in PRocedural EGOcentric videos,"Promptly identifying procedural errors from egocentric videos in an online
+setting is highly challenging and valuable for detecting mistakes as soon as
+they happen. This capability has a wide range of applications across various
+fields, such as manufacturing and healthcare. The nature of procedural mistakes
+is open-set since novel types of failures might occur, which calls for
+one-class classifiers trained on correctly executed procedures. However, no
+technique can currently detect open-set procedural mistakes online. We propose
+PREGO, the first online one-class classification model for mistake detection in
+PRocedural EGOcentric videos. PREGO is based on an online action recognition
+component to model the current action, and a symbolic reasoning module to
+predict the next actions. Mistake detection is performed by comparing the
+recognized current action with the expected future one. We evaluate PREGO on
+two procedural egocentric video datasets, Assembly101 and Epic-tent, which we
+adapt for online benchmarking of procedural mistake detection to establish
+suitable benchmarks, thus defining the Assembly101-O and Epic-tent-O datasets,
+respectively.",cs.CV,['cs.CV']
+TEA: Test-time Energy Adaptation,Yige Yuan · Bingbing Xu · Liang Hou · Fei Sun · Huawei Shen · Xueqi Cheng, ,https://arxiv.org/abs/2311.14402,,2311.14402.pdf,TEA: Test-time Energy Adaptation,"Test-time adaptation (TTA) aims to improve model generalizability when test
+data diverges from training distribution, offering the distinct advantage of
+not requiring access to training data and processes, especially valuable in the
+context of large pre-trained models. However, current TTA methods fail to
+address the fundamental issue: covariate shift, i.e., the decreased
+generalizability can be attributed to the model's reliance on the marginal
+distribution of the training data, which may impair model calibration and
+introduce confirmation bias. To address this, we propose a novel energy-based
+perspective, enhancing the model's perception of target data distributions
+without requiring access to training data or processes. Building on this
+perspective, we introduce $\textbf{T}$est-time $\textbf{E}$nergy
+$\textbf{A}$daptation ($\textbf{TEA}$), which transforms the trained classifier
+into an energy-based model and aligns the model's distribution with the test
+data's, enhancing its ability to perceive test distributions and thus improving
+overall generalizability. Extensive experiments across multiple tasks,
+benchmarks and architectures demonstrate TEA's superior generalization
+performance against state-of-the-art methods. Further in-depth analyses reveal
+that TEA can equip the model with a comprehensive perception of test
+distribution, ultimately paving the way toward improved generalization and
+calibration.",cs.LG,['cs.LG']
+A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network,Ruichen Ma · Guanchao Qiao · Yian Liu · Liwei Meng · Ning Ning · Yang Liu · Shaogang Hu,https://github.com/Ruichen0424/AB-BNN,https://arxiv.org/abs/2403.03739,,2403.03739.pdf,A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network,"Binary neural networks utilize 1-bit quantized weights and activations to
+reduce both the model's storage demands and computational burden. However,
+advanced binary architectures still incorporate millions of inefficient and
+nonhardware-friendly full-precision multiplication operations. A&B BNN is
+proposed to directly remove part of the multiplication operations in a
+traditional BNN and replace the rest with an equal number of bit operations,
+introducing the mask layer and the quantized RPReLU structure based on the
+normalizer-free network architecture. The mask layer can be removed during
+inference by leveraging the intrinsic characteristics of BNN with
+straightforward mathematical transformations to avoid the associated
+multiplication operations. The quantized RPReLU structure enables more
+efficient bit operations by constraining its slope to be integer powers of 2.
+Experimental results achieved 92.30%, 69.35%, and 66.89% on the CIFAR-10,
+CIFAR-100, and ImageNet datasets, respectively, which are competitive with the
+state-of-the-art. Ablation studies have verified the efficacy of the quantized
+RPReLU structure, leading to a 1.14% enhancement on the ImageNet compared to
+using a fixed slope RLeakyReLU. The proposed add&bit-operation-only BNN offers
+an innovative approach for hardware-friendly network architecture.",cs.LG,"['cs.LG', 'cs.AI']"
+"Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges",Tongtong Yuan · Xuange Zhang · Kun Liu · Bo Liu · Chen Chen · Jian Jin · Zhenzhen Jiao, ,https://arxiv.org/abs/2309.13925,,2309.13925.pdf,"Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges","Surveillance videos are an essential component of daily life with various
+critical applications, particularly in public security. However, current
+surveillance video tasks mainly focus on classifying and localizing anomalous
+events. Existing methods are limited to detecting and classifying the
+predefined events with unsatisfactory semantic understanding, although they
+have obtained considerable performance. To address this issue, we propose a new
+research direction of surveillance video-and-language understanding, and
+construct the first multimodal surveillance video dataset. We manually annotate
+the real-world surveillance dataset UCF-Crime with fine-grained event content
+and timing. Our newly annotated dataset, UCA (UCF-Crime Annotation), contains
+23,542 sentences, with an average length of 20 words, and its annotated videos
+are as long as 110.7 hours. Furthermore, we benchmark SOTA models for four
+multimodal tasks on this newly created dataset, which serve as new baselines
+for surveillance video-and-language understanding. Through our experiments, we
+find that mainstream models used in previously publicly available datasets
+perform poorly on surveillance video, which demonstrates the new challenges in
+surveillance video-and-language understanding. To validate the effectiveness of
+our UCA, we conducted experiments on multimodal anomaly detection. The results
+demonstrate that our multimodal surveillance learning can improve the
+performance of conventional anomaly detection tasks. All the experiments
+highlight the necessity of constructing this dataset to advance surveillance
+AI. The link to our dataset is provided at:
+https://xuange923.github.io/Surveillance-Video-Understanding.",cs.CV,"['cs.CV', 'cs.AI']"
+Validating Privacy-Preserving Face Recognition under a Minimum Assumption,Hui Zhang · Xingbo Dong · YenLungLai · Ying Zhou · Xiaoyan ZHANG · Xingguo Lv · Zhe Jin · Xuejun Li, ,https://arxiv.org/abs/2403.12457,,2403.12457.pdf,Privacy-Preserving Face Recognition Using Trainable Feature Subtraction,"The widespread adoption of face recognition has led to increasing privacy
+concerns, as unauthorized access to face images can expose sensitive personal
+information. This paper explores face image protection against viewing and
+recovery attacks. Inspired by image compression, we propose creating a visually
+uninformative face image through feature subtraction between an original face
+and its model-produced regeneration. Recognizable identity features within the
+image are encouraged by co-training a recognition model on its high-dimensional
+feature representation. To enhance privacy, the high-dimensional representation
+is crafted through random channel shuffling, resulting in randomized
+recognizable images devoid of attacker-leverageable texture details. We distill
+our methodologies into a novel privacy-preserving face recognition method,
+MinusFace. Experiments demonstrate its high recognition accuracy and effective
+privacy protection. Its code is available at https://github.com/Tencent/TFace.",cs.CV,['cs.CV']
+One-Shot Open Affordance Learning with Foundation Models,Gen Li · Deqing Sun · Laura Sevilla-Lara · Varun Jampani, ,https://arxiv.org/abs/2311.17776v1,,2311.17776v1.pdf,One-Shot Open Affordance Learning with Foundation Models,"We introduce One-shot Open Affordance Learning (OOAL), where a model is
+trained with just one example per base object category, but is expected to
+identify novel objects and affordances. While vision-language models excel at
+recognizing novel objects and scenes, they often struggle to understand finer
+levels of granularity such as affordances. To handle this issue, we conduct a
+comprehensive analysis of existing foundation models, to explore their inherent
+understanding of affordances and assess the potential for data-limited
+affordance learning. We then propose a vision-language framework with simple
+and effective designs that boost the alignment between visual features and
+affordance text embeddings. Experiments on two affordance segmentation
+benchmarks show that the proposed method outperforms state-of-the-art models
+with less than 1% of the full training data, and exhibits reasonable
+generalization capability on unseen objects and affordances.",cs.CV,['cs.CV']
+Automatic Controllable Colorization via Imagination,Xiaoyan Cong · Yue Wu · Qifeng Chen · Chenyang Lei, ,https://arxiv.org/abs/2404.05661,,2404.05661.pdf,Automatic Controllable Colorization via Imagination,"We propose a framework for automatic colorization that allows for iterative
+editing and modifications. The core of our framework lies in an imagination
+module: by understanding the content within a grayscale image, we utilize a
+pre-trained image generation model to generate multiple images that contain the
+same content. These images serve as references for coloring, mimicking the
+process of human experts. As the synthesized images can be imperfect or
+different from the original grayscale image, we propose a Reference Refinement
+Module to select the optimal reference composition. Unlike most previous
+end-to-end automatic colorization algorithms, our framework allows for
+iterative and localized modifications of the colorization results because we
+explicitly model the coloring samples. Extensive experiments demonstrate the
+superiority of our framework over existing automatic colorization algorithms in
+editability and flexibility. Project page:
+https://xy-cong.github.io/imagine-colorization.",cs.CV,['cs.CV']
+GAFusion: Adaptive Fusing LiDAR and Camera with Multiple Guidance for 3D Object Detection,Xiaotian Li · Baojie Fan · Jiandong Tian · Huijie Fan, ,https://arxiv.org/abs/2309.11804,,2309.11804.pdf,FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detection,"Lidars and cameras are critical sensors that provide complementary
+information for 3D detection in autonomous driving. While most prevalent
+methods progressively downscale the 3D point clouds and camera images and then
+fuse the high-level features, the downscaled features inevitably lose low-level
+detailed information. In this paper, we propose Fine-Grained Lidar-Camera
+Fusion (FGFusion) that make full use of multi-scale features of image and point
+cloud and fuse them in a fine-grained way. First, we design a dual pathway
+hierarchy structure to extract both high-level semantic and low-level detailed
+features of the image. Second, an auxiliary network is introduced to guide
+point cloud features to better learn the fine-grained spatial information.
+Finally, we propose multi-scale fusion (MSF) to fuse the last N feature maps of
+image and point cloud. Extensive experiments on two popular autonomous driving
+benchmarks, i.e. KITTI and Waymo, demonstrate the effectiveness of our method.",cs.CV,['cs.CV']
+Open Vocabulary Semantic Scene Sketch Understanding,Ahmed Bourouis · Judith Fan · Yulia Gryaditskaya,https://ahmedbourouis.github.io/Scene_Sketch_Segmentation/,https://arxiv.org/abs/2312.12463,,2312.12463.pdf,Open Vocabulary Semantic Scene Sketch Understanding,"We study the underexplored but fundamental vision problem of machine
+understanding of abstract freehand scene sketches. We introduce a sketch
+encoder that results in semantically-aware feature space, which we evaluate by
+testing its performance on a semantic sketch segmentation task. To train our
+model we rely only on the availability of bitmap sketches with their brief
+captions and do not require any pixel-level annotations. To obtain
+generalization to a large set of sketches and categories, we build on a vision
+transformer encoder pretrained with the CLIP model. We freeze the text encoder
+and perform visual-prompt tuning of the visual encoder branch while introducing
+a set of critical modifications. Firstly, we augment the classical key-query
+(k-q) self-attention blocks with value-value (v-v) self-attention blocks.
+Central to our model is a two-level hierarchical network design that enables
+efficient semantic disentanglement: The first level ensures holistic scene
+sketch encoding, and the second level focuses on individual categories. We,
+then, in the second level of the hierarchy, introduce a cross-attention between
+textual and visual branches. Our method outperforms zero-shot CLIP pixel
+accuracy of segmentation results by 37 points, reaching an accuracy of $85.5\%$
+on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us
+to identify further improvements needed over our method to reconcile machine
+and human understanding of scene sketches.",cs.CV,['cs.CV']
+View From Above: Orthogonal viewpoint aware Cross-view Localization,Shan Wang · Chuong Nguyen · Jiawei Liu · Yanhao Zhang · Sundaram Muthu · Fahira Afzal Maken · Kaihao Zhang · Hongdong Li, ,https://arxiv.org/abs/2308.08110,,2308.08110.pdf,View Consistent Purification for Accurate Cross-View Localization,"This paper proposes a fine-grained self-localization method for outdoor
+robotics that utilizes a flexible number of onboard cameras and readily
+accessible satellite images. The proposed method addresses limitations in
+existing cross-view localization methods that struggle to handle noise sources
+such as moving objects and seasonal variations. It is the first sparse
+visual-only method that enhances perception in dynamic environments by
+detecting view-consistent key points and their corresponding deep features from
+ground and satellite views, while removing off-the-ground objects and
+establishing homography transformation between the two views. Moreover, the
+proposed method incorporates a spatial embedding approach that leverages camera
+intrinsic and extrinsic information to reduce the ambiguity of purely visual
+matching, leading to improved feature matching and overall pose estimation
+accuracy. The method exhibits strong generalization and is robust to
+environmental changes, requiring only geo-poses as ground truth. Extensive
+experiments on the KITTI and Ford Multi-AV Seasonal datasets demonstrate that
+our proposed method outperforms existing state-of-the-art methods, achieving
+median spatial accuracy errors below $0.5$ meters along the lateral and
+longitudinal directions, and a median orientation accuracy error below 2
+degrees.",cs.CV,['cs.CV']
+OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation,Jisoo Jeong · Hong Cai · Risheek Garrepalli · Jamie Lin · Munawar Hayat · Fatih Porikli, ,https://arxiv.org/abs/2403.18092,,2403.18092.pdf,OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation,"The scarcity of ground-truth labels poses one major challenge in developing
+optical flow estimation models that are both generalizable and robust. While
+current methods rely on data augmentation, they have yet to fully exploit the
+rich information available in labeled video sequences. We propose OCAI, a
+method that supports robust frame interpolation by generating intermediate
+video frames alongside optical flows in between. Utilizing a forward warping
+approach, OCAI employs occlusion awareness to resolve ambiguities in pixel
+values and fills in missing values by leveraging the forward-backward
+consistency of optical flows. Additionally, we introduce a teacher-student
+style semi-supervised learning method on top of the interpolated frames. Using
+a pair of unlabeled frames and the teacher model's predicted optical flow, we
+generate interpolated frames and flows to train a student model. The teacher's
+weights are maintained using Exponential Moving Averaging of the student. Our
+evaluations demonstrate perceptually superior interpolation quality and
+enhanced optical flow accuracy on established benchmarks such as Sintel and
+KITTI.",cs.CV,['cs.CV']
+GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs,Gege Gao · Weiyang Liu · Anpei Chen · Andreas Geiger · Bernhard Schölkopf, ,https://arxiv.org/abs/2312.00093,,2312.00093.pdf,GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs,"As pretrained text-to-image diffusion models become increasingly powerful,
+recent efforts have been made to distill knowledge from these text-to-image
+pretrained models for optimizing a text-guided 3D model. Most of the existing
+methods generate a holistic 3D model from a plain text input. This can be
+problematic when the text describes a complex scene with multiple objects,
+because the vectorized text embeddings are inherently unable to capture a
+complex description with multiple entities and relationships. Holistic 3D
+modeling of the entire scene further prevents accurate grounding of text
+entities and concepts. To address this limitation, we propose GraphDreamer, a
+novel framework to generate compositional 3D scenes from scene graphs, where
+objects are represented as nodes and their interactions as edges. By exploiting
+node and edge information in scene graphs, our method makes better use of the
+pretrained text-to-image diffusion model and is able to fully disentangle
+different objects without image-level supervision. To facilitate modeling of
+object-wise relationships, we use signed distance fields as representation and
+impose a constraint to avoid inter-penetration of objects. To avoid manual
+scene graph creation, we design a text prompt for ChatGPT to generate scene
+graphs based on text inputs. We conduct both qualitative and quantitative
+experiments to validate the effectiveness of GraphDreamer in generating
+high-fidelity compositional 3D scenes with disentangled object entities.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning,Wenjin Hou · Shiming Chen · Shuhuang Chen · Ziming Hong · Yan Wang · Xuetao Feng · Salman Khan · Fahad Shahbaz Khan · Xinge You, ,https://arxiv.org/abs/2404.14808v1,,2404.14808v1.pdf,Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning,"Generative Zero-shot learning (ZSL) learns a generator to synthesize visual
+samples for unseen classes, which is an effective way to advance ZSL. However,
+existing generative methods rely on the conditions of Gaussian noise and the
+predefined semantic prototype, which limit the generator only optimized on
+specific seen classes rather than characterizing each visual instance,
+resulting in poor generalizations (\textit{e.g.}, overfitting to seen classes).
+To address this issue, we propose a novel Visual-Augmented Dynamic Semantic
+prototype method (termed VADS) to boost the generator to learn accurate
+semantic-visual mapping by fully exploiting the visual-augmented knowledge into
+semantic conditions. In detail, VADS consists of two modules: (1) Visual-aware
+Domain Knowledge Learning module (VDKL) learns the local bias and global prior
+of the visual features (referred to as domain visual knowledge), which replace
+pure Gaussian noise to provide richer prior noise information; (2)
+Vision-Oriented Semantic Updation module (VOSU) updates the semantic prototype
+according to the visual representations of the samples. Ultimately, we
+concatenate their output as a dynamic semantic prototype, which serves as the
+condition of the generator. Extensive experiments demonstrate that our VADS
+achieves superior CZSL and GZSL performances on three prominent datasets and
+outperforms other state-of-the-art methods with averaging increases by 6.4\%,
+5.9\% and 4.2\% on SUN, CUB and AWA2, respectively.",cs.CV,['cs.CV']
+EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors,Zhipeng Hu · Minda Zhao · Chaoyi Zhao · Xinyue Liang · Lincheng Li · Zeng Zhao · Changjie Fan · Xiaowei Zhou · Xin Yu, ,https://arxiv.org/abs/2308.13223,,2308.13223.pdf,EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior,"While image diffusion models have made significant progress in text-driven 3D
+content creation, they often fail to accurately capture the intended meaning of
+text prompts, especially for view information. This limitation leads to the
+Janus problem, where multi-faced 3D models are generated under the guidance of
+such diffusion models. In this paper, we propose a robust high-quality 3D
+content generation pipeline by exploiting orthogonal-view image guidance.
+First, we introduce a novel 2D diffusion model that generates an image
+consisting of four orthogonal-view sub-images based on the given text prompt.
+Then, the 3D content is created using this diffusion model. Notably, the
+generated orthogonal-view image provides strong geometric structure priors and
+thus improves 3D consistency. As a result, it effectively resolves the Janus
+problem and significantly enhances the quality of 3D content creation.
+Additionally, we present a 3D synthesis fusion network that can further improve
+the details of the generated 3D contents. Both quantitative and qualitative
+evaluations demonstrate that our method surpasses previous text-to-3D
+techniques. Project page: https://efficientdreamer.github.io.",cs.CV,['cs.CV']
+Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On,Xu Yang · Changxing Ding · Zhibin Hong · Junhao Huang · Jin Tao · Xiangmin Xu, ,https://arxiv.org/abs/2404.01089,,2404.01089.pdf,Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On,"Image-based virtual try-on is an increasingly important task for online
+shopping. It aims to synthesize images of a specific person wearing a specified
+garment. Diffusion model-based approaches have recently become popular, as they
+are excellent at image synthesis tasks. However, these approaches usually
+employ additional image encoders and rely on the cross-attention mechanism for
+texture transfer from the garment to the person image, which affects the
+try-on's efficiency and fidelity. To address these issues, we propose an
+Texture-Preserving Diffusion (TPD) model for virtual try-on, which enhances the
+fidelity of the results and introduces no additional image encoders.
+Accordingly, we make contributions from two aspects. First, we propose to
+concatenate the masked person and reference garment images along the spatial
+dimension and utilize the resulting image as the input for the diffusion
+model's denoising UNet. This enables the original self-attention layers
+contained in the diffusion model to achieve efficient and accurate texture
+transfer. Second, we propose a novel diffusion-based method that predicts a
+precise inpainting mask based on the person and reference garment images,
+further enhancing the reliability of the try-on results. In addition, we
+integrate mask prediction and image synthesis into a single compact model. The
+experimental results show that our approach can be applied to various try-on
+tasks, e.g., garment-to-person and person-to-person try-ons, and significantly
+outperforms state-of-the-art methods on popular VITON, VITON-HD databases.",cs.CV,"['cs.CV', 'cs.AI']"
+ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting,Yankai Jiang · Zhongzhen Huang · Rongzhao Zhang · Xiaofan Zhang · Shaoting Zhang,https://github.com/Yankai96/ZePT,https://arxiv.org/abs/2312.04964,,2312.04964.pdf,ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting,"The long-tailed distribution problem in medical image analysis reflects a
+high prevalence of common conditions and a low prevalence of rare ones, which
+poses a significant challenge in developing a unified model capable of
+identifying rare or novel tumor categories not encountered during training. In
+this paper, we propose a new zero-shot pan-tumor segmentation framework (ZePT)
+based on query-disentangling and self-prompting to segment unseen tumor
+categories beyond the training set. ZePT disentangles the object queries into
+two subsets and trains them in two stages. Initially, it learns a set of
+fundamental queries for organ segmentation through an object-aware feature
+grouping strategy, which gathers organ-level visual features. Subsequently, it
+refines the other set of advanced queries that focus on the auto-generated
+visual prompts for unseen tumor segmentation. Moreover, we introduce
+query-knowledge alignment at the feature level to enhance each query's
+discriminative representation and generalizability. Extensive experiments on
+various tumor segmentation tasks demonstrate the performance superiority of
+ZePT, which surpasses the previous counterparts and evidence the promising
+ability for zero-shot tumor segmentation in real-world settings.",cs.CV,['cs.CV']
+Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld,Yijun Yang · Tianyi Zhou · kanxue Li · Dapeng Tao · Lusong Li · Li Shen · Xiaodong He · Jing Jiang · Yuhui Shi,https://github.com/stevenyangyj/Emma-Alfworld,https://arxiv.org/abs/2311.16714v1,,2311.16714v1.pdf,Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld,"While large language models (LLMs) excel in a simulated world of texts, they
+struggle to interact with the more realistic world without perceptions of other
+modalities such as visual or audio signals. Although vision-language models
+(VLMs) integrate LLM modules (1) aligned with static image features, and (2)
+may possess prior knowledge of world dynamics (as demonstrated in the text
+world), they have not been trained in an embodied visual world and thus cannot
+align with its dynamics. On the other hand, training an embodied agent in a
+noisy visual world without expert guidance is often challenging and
+inefficient. In this paper, we train a VLM agent living in a visual world using
+an LLM agent excelling in a parallel text world (but inapplicable to the visual
+world). Specifically, we distill LLM's reflection outcomes (improved actions by
+analyzing mistakes) in a text world's tasks to finetune the VLM on the same
+tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA)
+quickly adapting to the visual world dynamics. Such cross-modality imitation
+learning between the two parallel worlds enables EMMA to generalize to a broad
+scope of new tasks without any further guidance from the LLM expert. Extensive
+evaluations on the ALFWorld benchmark highlight EMMA's superior performance to
+SOTA VLM-based agents across diverse tasks, e.g., 20%-70% improvement in the
+success rate.",cs.CV,['cs.CV']
+Mip-Splatting: Alias-free 3D Gaussian Splatting,Zehao Yu · Anpei Chen · Binbin Huang · Torsten Sattler · Andreas Geiger, ,https://arxiv.org/abs/2311.16493,,2311.16493.pdf,Mip-Splatting: Alias-free 3D Gaussian Splatting,"Recently, 3D Gaussian Splatting has demonstrated impressive novel view
+synthesis results, reaching high fidelity and efficiency. However, strong
+artifacts can be observed when changing the sampling rate, \eg, by changing
+focal length or camera distance. We find that the source for this phenomenon
+can be attributed to the lack of 3D frequency constraints and the usage of a 2D
+dilation filter. To address this problem, we introduce a 3D smoothing filter
+which constrains the size of the 3D Gaussian primitives based on the maximal
+sampling frequency induced by the input views, eliminating high-frequency
+artifacts when zooming in. Moreover, replacing 2D dilation with a 2D Mip
+filter, which simulates a 2D box filter, effectively mitigates aliasing and
+dilation issues. Our evaluation, including scenarios such a training on
+single-scale images and testing on multiple scales, validates the effectiveness
+of our approach.",cs.CV,['cs.CV']
+Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling,Baoquan Zhang · Huaibin Wang · Luo Chuyao · Xutao Li · Guotao liang · Yunming Ye · joeq · Yao He,https://youtu.be/N6M0jcMP9lo,https://arxiv.org/abs/2403.10071,,2403.10071.pdf,Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling,"Vector-Quantized Image Modeling (VQIM) is a fundamental research problem in
+image synthesis, which aims to represent an image with a discrete token
+sequence. Existing studies effectively address this problem by learning a
+discrete codebook from scratch and in a code-independent manner to quantize
+continuous representations into discrete tokens. However, learning a codebook
+from scratch and in a code-independent manner is highly challenging, which may
+be a key reason causing codebook collapse, i.e., some code vectors can rarely
+be optimized without regard to the relationship between codes and good codebook
+priors such that die off finally. In this paper, inspired by pretrained
+language models, we find that these language models have actually pretrained a
+superior codebook via a large number of text corpus, but such information is
+rarely exploited in VQIM. To this end, we propose a novel codebook transfer
+framework with part-of-speech, called VQCT, which aims to transfer a
+well-trained codebook from pretrained language models to VQIM for robust
+codebook learning. Specifically, we first introduce a pretrained codebook from
+language models and part-of-speech knowledge as priors. Then, we construct a
+vision-related codebook with these priors for achieving codebook transfer.
+Finally, a novel codebook transfer network is designed to exploit abundant
+semantic relationships between codes contained in pretrained codebooks for
+robust VQIM codebook learning. Experimental results on four datasets show that
+our VQCT method achieves superior VQIM performance over previous
+state-of-the-art methods.",cs.CV,['cs.CV']
+Multi-Level Neural Scene Graphs for Dynamic Urban Environments,Tobias Fischer · Lorenzo Porzi · Samuel Rota Bulò · Marc Pollefeys · Peter Kontschieder, ,https://arxiv.org/abs/2404.00168,,2404.00168.pdf,Multi-Level Neural Scene Graphs for Dynamic Urban Environments,"We estimate the radiance field of large-scale dynamic areas from multiple
+vehicle captures under varying environmental conditions. Previous works in this
+domain are either restricted to static environments, do not scale to more than
+a single short video, or struggle to separately represent dynamic object
+instances. To this end, we present a novel, decomposable radiance field
+approach for dynamic urban environments. We propose a multi-level neural scene
+graph representation that scales to thousands of images from dozens of
+sequences with hundreds of fast-moving objects. To enable efficient training
+and rendering of our representation, we develop a fast composite ray sampling
+and rendering scheme. To test our approach in urban driving scenarios, we
+introduce a new, novel view synthesis benchmark. We show that our approach
+outperforms prior art by a significant margin on both established and our
+proposed benchmark while being faster in training and rendering.",cs.CV,['cs.CV']
+Rethinking Multi-view Representation Learning via Distilled Disentangling,Guanzhou Ke · Bo Wang · Xiao-Li Wang · Shengfeng He, ,https://arxiv.org/abs/2403.10897,,2403.10897.pdf,Rethinking Multi-view Representation Learning via Distilled Disentangling,"Multi-view representation learning aims to derive robust representations that
+are both view-consistent and view-specific from diverse data sources. This
+paper presents an in-depth analysis of existing approaches in this domain,
+highlighting a commonly overlooked aspect: the redundancy between
+view-consistent and view-specific representations. To this end, we propose an
+innovative framework for multi-view representation learning, which incorporates
+a technique we term 'distilled disentangling'. Our method introduces the
+concept of masked cross-view prediction, enabling the extraction of compact,
+high-quality view-consistent representations from various sources without
+incurring extra computational overhead. Additionally, we develop a distilled
+disentangling module that efficiently filters out consistency-related
+information from multi-view representations, resulting in purer view-specific
+representations. This approach significantly reduces redundancy between
+view-consistent and view-specific representations, enhancing the overall
+efficiency of the learning process. Our empirical evaluations reveal that
+higher mask ratios substantially improve the quality of view-consistent
+representations. Moreover, we find that reducing the dimensionality of
+view-consistent representations relative to that of view-specific
+representations further refines the quality of the combined representations.
+Our code is accessible at: https://github.com/Guanzhou-Ke/MRDD.",cs.CV,"['cs.CV', 'cs.MM']"
+Neural Refinement for Absolute Pose Regression with Feature Synthesis,Shuai Chen · Yash Bhalgat · Xinghui Li · Jia-Wang Bian · Kejie Li · Zirui Wang · Victor Adrian Prisacariu, ,https://arxiv.org/html/2402.14371v2,,2402.14371v2.pdf,HR-APR: APR-agnostic Framework with Uncertainty Estimation and Hierarchical Refinement for Camera Relocalisation,"Absolute Pose Regressors (APRs) directly estimate camera poses from monocular
+images, but their accuracy is unstable for different queries. Uncertainty-aware
+APRs provide uncertainty information on the estimated pose, alleviating the
+impact of these unreliable predictions. However, existing uncertainty modelling
+techniques are often coupled with a specific APR architecture, resulting in
+suboptimal performance compared to state-of-the-art (SOTA) APR methods. This
+work introduces a novel APR-agnostic framework, HR-APR, that formulates
+uncertainty estimation as cosine similarity estimation between the query and
+database features. It does not rely on or affect APR network architecture,
+which is flexible and computationally efficient. In addition, we take advantage
+of the uncertainty for pose refinement to enhance the performance of APR. The
+extensive experiments demonstrate the effectiveness of our framework, reducing
+27.4\% and 15.2\% of computational overhead on the 7Scenes and Cambridge
+Landmarks datasets while maintaining the SOTA accuracy in single-image APRs.",cs.CV,"['cs.CV', 'cs.RO']"
+Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer,Zhen Zhao · Jingqun Tang · Chunhui Lin · Binghong Wu · Can Huang · Hao Liu · Xin Tan · Zhizhong Zhang · Yuan Xie,https://github.com/bytedance/E2STR,https://arxiv.org/abs/2311.13120,,2311.13120.pdf,Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer,"Scene text recognition (STR) in the wild frequently encounters challenges
+when coping with domain variations, font diversity, shape deformations, etc. A
+straightforward solution is performing model fine-tuning tailored to a specific
+scenario, but it is computationally intensive and requires multiple model
+copies for various scenarios. Recent studies indicate that large language
+models (LLMs) can learn from a few demonstration examples in a training-free
+manner, termed ""In-Context Learning"" (ICL). Nevertheless, applying LLMs as a
+text recognizer is unacceptably resource-consuming. Moreover, our pilot
+experiments on LLMs show that ICL fails in STR, mainly attributed to the
+insufficient incorporation of contextual information from diverse samples in
+the training stage. To this end, we introduce E$^2$STR, a STR model trained
+with context-rich scene text sequences, where the sequences are generated via
+our proposed in-context training strategy. E$^2$STR demonstrates that a
+regular-sized model is sufficient to achieve effective ICL capabilities in STR.
+Extensive experiments show that E$^2$STR exhibits remarkable training-free
+adaptation in various scenarios and outperforms even the fine-tuned
+state-of-the-art approaches on public benchmarks. The code is released at
+https://github.com/bytedance/E2STR .",cs.CV,['cs.CV']
+Cross Initialization for Face Personalization of Text-to-Image Models,Lianyu Pang · Jian Yin · Haoran Xie · Qiping Wang · Qing Li · Xudong Mao, ,https://arxiv.org/abs/2312.15905,,2312.15905.pdf,Cross Initialization for Personalized Text-to-Image Generation,"Recently, there has been a surge in face personalization techniques,
+benefiting from the advanced capabilities of pretrained text-to-image diffusion
+models. Among these, a notable method is Textual Inversion, which generates
+personalized images by inverting given images into textual embeddings. However,
+methods based on Textual Inversion still struggle with balancing the trade-off
+between reconstruction quality and editability. In this study, we examine this
+issue through the lens of initialization. Upon closely examining traditional
+initialization methods, we identified a significant disparity between the
+initial and learned embeddings in terms of both scale and orientation. The
+scale of the learned embedding can be up to 100 times greater than that of the
+initial embedding. Such a significant change in the embedding could increase
+the risk of overfitting, thereby compromising the editability. Driven by this
+observation, we introduce a novel initialization method, termed Cross
+Initialization, that significantly narrows the gap between the initial and
+learned embeddings. This method not only improves both reconstruction and
+editability but also reduces the optimization steps from 5000 to 320.
+Furthermore, we apply a regularization term to keep the learned embedding close
+to the initial embedding. We show that when combined with Cross Initialization,
+this regularization term can effectively improve editability. We provide
+comprehensive empirical evidence to demonstrate the superior performance of our
+method compared to the baseline methods. Notably, in our experiments, Cross
+Initialization is the only method that successfully edits an individual's
+facial expression. Additionally, a fast version of our method allows for
+capturing an input image in roughly 26 seconds, while surpassing the baseline
+methods in terms of both reconstruction and editability. Code will be made
+publicly available.",cs.CV,['cs.CV']
+Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis,Marianna Ohanyan · Hayk Manukyan · Zhangyang Wang · Shant Navasardyan · Humphrey Shi, ,https://arxiv.org/abs/2311.12342,,2311.12342.pdf,LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis,"Recent text-to-image diffusion models have reached an unprecedented level in
+generating high-quality images. However, their exclusive reliance on textual
+prompts often falls short in precise control of image compositions. In this
+paper, we propose LoCo, a training-free approach for layout-to-image Synthesis
+that excels in producing high-quality images aligned with both textual prompts
+and layout instructions. Specifically, we introduce a Localized Attention
+Constraint (LAC), leveraging semantic affinity between pixels in self-attention
+maps to create precise representations of desired objects and effectively
+ensure the accurate placement of objects in designated regions. We further
+propose a Padding Token Constraint (PTC) to leverage the semantic information
+embedded in previously neglected padding tokens, improving the consistency
+between object appearance and layout instructions. LoCo seamlessly integrates
+into existing text-to-image and layout-to-image models, enhancing their
+performance in spatial control and addressing semantic failures observed in
+prior methods. Extensive experiments showcase the superiority of our approach,
+surpassing existing state-of-the-art training-free layout-to-image methods both
+qualitatively and quantitatively across multiple benchmarks.",cs.CV,['cs.CV']
+NB-GTR: Narrow-Band Guided Turbulence Removal,Yifei Xia · Chu Zhou · Chengxuan Zhu · Minggui Teng · Chao Xu · Boxin Shi, ,,https://freebutuselesssoul.github.io/publications/cvpr2024b,,,,,nan
+SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models,Yuzhou Huang · Liangbin Xie · Xintao Wang · Ziyang Yuan · Xiaodong Cun · Yixiao Ge · Jiantao Zhou · Chao Dong · Rui Huang · Ruimao Zhang · Ying Shan, ,https://arxiv.org/abs/2312.06739,,2312.06739.pdf,SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models,"Current instruction-based editing methods, such as InstructPix2Pix, often
+fail to produce satisfactory results in complex scenarios due to their
+dependence on the simple CLIP text encoder in diffusion models. To rectify
+this, this paper introduces SmartEdit, a novel approach to instruction-based
+image editing that leverages Multimodal Large Language Models (MLLMs) to
+enhance their understanding and reasoning capabilities. However, direct
+integration of these elements still faces challenges in situations requiring
+complex reasoning. To mitigate this, we propose a Bidirectional Interaction
+Module that enables comprehensive bidirectional information interactions
+between the input image and the MLLM output. During training, we initially
+incorporate perception data to boost the perception and understanding
+capabilities of diffusion models. Subsequently, we demonstrate that a small
+amount of complex instruction editing data can effectively stimulate
+SmartEdit's editing capabilities for more complex instructions. We further
+construct a new evaluation dataset, Reason-Edit, specifically tailored for
+complex instruction-based image editing. Both quantitative and qualitative
+results on this evaluation dataset indicate that our SmartEdit surpasses
+previous methods, paving the way for the practical application of complex
+instruction-based image editing.",cs.CV,['cs.CV']
+FADES: Fair Disentanglement with Sensitive Relevance,Taeuk Jang · Xiaoqian Wang, ,https://arxiv.org/abs/2405.07011,,2405.07011.pdf,Fair Graph Representation Learning via Sensitive Attribute Disentanglement,"Group fairness for Graph Neural Networks (GNNs), which emphasizes algorithmic
+decisions neither favoring nor harming certain groups defined by sensitive
+attributes (e.g., race and gender), has gained considerable attention. In
+particular, the objective of group fairness is to ensure that the decisions
+made by GNNs are independent of the sensitive attribute. To achieve this
+objective, most existing approaches involve eliminating sensitive attribute
+information in node representations or algorithmic decisions. However, such
+ways may also eliminate task-related information due to its inherent
+correlation with the sensitive attribute, leading to a sacrifice in utility. In
+this work, we focus on improving the fairness of GNNs while preserving
+task-related information and propose a fair GNN framework named FairSAD.
+Instead of eliminating sensitive attribute information, FairSAD enhances the
+fairness of GNNs via Sensitive Attribute Disentanglement (SAD), which separates
+the sensitive attribute-related information into an independent component to
+mitigate its impact. Additionally, FairSAD utilizes a channel masking mechanism
+to adaptively identify the sensitive attribute-related component and
+subsequently decorrelates it. Overall, FairSAD minimizes the impact of the
+sensitive attribute on GNN outcomes rather than eliminating sensitive
+attributes, thereby preserving task-related information associated with the
+sensitive attribute. Furthermore, experiments conducted on several real-world
+datasets demonstrate that FairSAD outperforms other state-of-the-art methods by
+a significant margin in terms of both fairness and utility performance. Our
+source code is available at https://github.com/ZzoomD/FairSAD.",cs.LG,"['cs.LG', 'cs.CY']"
+VRP-SAM: SAM with Visual Reference Prompt,Yanpeng Sun · Jiahui Chen · Shan Zhang · Xinyu Zhang · Qiang Chen · gang zhang · Errui Ding · Jingdong Wang · Zechao Li, ,https://arxiv.org/abs/2402.17726,,2402.17726.pdf,VRP-SAM: SAM with Visual Reference Prompt,"In this paper, we propose a novel Visual Reference Prompt (VRP) encoder that
+empowers the Segment Anything Model (SAM) to utilize annotated reference images
+as prompts for segmentation, creating the VRP-SAM model. In essence, VRP-SAM
+can utilize annotated reference images to comprehend specific objects and
+perform segmentation of specific objects in target image. It is note that the
+VRP encoder can support a variety of annotation formats for reference images,
+including \textbf{point}, \textbf{box}, \textbf{scribble}, and \textbf{mask}.
+VRP-SAM achieves a breakthrough within the SAM framework by extending its
+versatility and applicability while preserving SAM's inherent strengths, thus
+enhancing user-friendliness. To enhance the generalization ability of VRP-SAM,
+the VRP encoder adopts a meta-learning strategy. To validate the effectiveness
+of VRP-SAM, we conducted extensive empirical studies on the Pascal and COCO
+datasets. Remarkably, VRP-SAM achieved state-of-the-art performance in visual
+reference segmentation with minimal learnable parameters. Furthermore, VRP-SAM
+demonstrates strong generalization capabilities, allowing it to perform
+segmentation of unseen objects and enabling cross-domain segmentation. The
+source code and models will be available at
+\url{https://github.com/syp2ysy/VRP-SAM}",cs.CV,['cs.CV']
+Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It,Adam Lilja · Junsheng Fu · Erik Stenborg · Lars Hammarstrand,https://github.com/LiljaAdam/geographical-splits,https://arxiv.org/abs/2312.06420,,2312.06420.pdf,Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It,"The task of online mapping is to predict a local map using current sensor
+observations, e.g. from lidar and camera, without relying on a pre-built map.
+State-of-the-art methods are based on supervised learning and are trained
+predominantly using two datasets: nuScenes and Argoverse 2. However, these
+datasets revisit the same geographic locations across training, validation, and
+test sets. Specifically, over $80$% of nuScenes and $40$% of Argoverse 2
+validation and test samples are less than $5$ m from a training sample. At test
+time, the methods are thus evaluated more on how well they localize within a
+memorized implicit map built from the training data than on extrapolating to
+unseen locations. Naturally, this data leakage causes inflated performance
+numbers and we propose geographically disjoint data splits to reveal the true
+performance in unseen environments. Experimental results show that methods
+perform considerably worse, some dropping more than $45$ mAP, when trained and
+evaluated on proper data splits. Additionally, a reassessment of prior design
+choices reveals diverging conclusions from those based on the original split.
+Notably, the impact of lifting methods and the support from auxiliary tasks
+(e.g., depth supervision) on performance appears less substantial or follows a
+different trajectory than previously perceived. Splits can be found at
+https://github.com/LiljaAdam/geographical-splits",cs.CV,['cs.CV']
+Gated Fields: Learning Scene Reconstruction from Gated Videos,Andrea Ramazzina · Stefanie Walz · Pragyan Dahal · Mario Bijelic · Felix Heide, ,https://arxiv.org/abs/2405.19819,,2405.19819.pdf,Gated Fields: Learning Scene Reconstruction from Gated Videos,"Reconstructing outdoor 3D scenes from temporal observations is a challenge
+that recent work on neural fields has offered a new avenue for. However,
+existing methods that recover scene properties, such as geometry, appearance,
+or radiance, solely from RGB captures often fail when handling poorly-lit or
+texture-deficient regions. Similarly, recovering scenes with scanning LiDAR
+sensors is also difficult due to their low angular sampling rate which makes
+recovering expansive real-world scenes difficult. Tackling these gaps, we
+introduce Gated Fields - a neural scene reconstruction method that utilizes
+active gated video sequences. To this end, we propose a neural rendering
+approach that seamlessly incorporates time-gated capture and illumination. Our
+method exploits the intrinsic depth cues in the gated videos, achieving precise
+and dense geometry reconstruction irrespective of ambient illumination
+conditions. We validate the method across day and night scenarios and find that
+Gated Fields compares favorably to RGB and LiDAR reconstruction methods. Our
+code and datasets are available at https://light.princeton.edu/gatedfields/.",cs.CV,['cs.CV']
+VINECS: Video-based Neural Character Skinning,Zhouyingcheng Liao · Vladislav Golyanik · Marc Habermann · Christian Theobalt, ,https://arxiv.org/abs/2307.00842,,2307.00842.pdf,VINECS: Video-based Neural Character Skinning,"Rigging and skinning clothed human avatars is a challenging task and
+traditionally requires a lot of manual work and expertise. Recent methods
+addressing it either generalize across different characters or focus on
+capturing the dynamics of a single character observed under different pose
+configurations. However, the former methods typically predict solely static
+skinning weights, which perform poorly for highly articulated poses, and the
+latter ones either require dense 3D character scans in different poses or
+cannot generate an explicit mesh with vertex correspondence over time. To
+address these challenges, we propose a fully automated approach for creating a
+fully rigged character with pose-dependent skinning weights, which can be
+solely learned from multi-view video. Therefore, we first acquire a rigged
+template, which is then statically skinned. Next, a coordinate-based MLP learns
+a skinning weights field parameterized over the position in a canonical pose
+space and the respective pose. Moreover, we introduce our pose- and
+view-dependent appearance field allowing us to differentiably render and
+supervise the posed mesh using multi-view imagery. We show that our approach
+outperforms state-of-the-art while not relying on dense 4D scans.",cs.CV,['cs.CV']
+LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding,Chuwei Luo · Yufan Shen · Zhaoqing Zhu · Qi Zheng · Zhi Yu · Cong Yao,https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM,https://arxiv.org/abs/2404.05225,,2404.05225.pdf,LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding,"Recently, leveraging large language models (LLMs) or multimodal large
+language models (MLLMs) for document understanding has been proven very
+promising. However, previous works that employ LLMs/MLLMs for document
+understanding have not fully explored and utilized the document layout
+information, which is vital for precise document understanding. In this paper,
+we propose LayoutLLM, an LLM/MLLM based method for document understanding. The
+core of LayoutLLM is a layout instruction tuning strategy, which is specially
+designed to enhance the comprehension and utilization of document layouts. The
+proposed layout instruction tuning strategy consists of two components:
+Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture
+the characteristics of document layout in Layout-aware Pre-training, three
+groups of pre-training tasks, corresponding to document-level, region-level and
+segment-level information, are introduced. Furthermore, a novel module called
+layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on
+regions relevant to the question and generate accurate answers. LayoutCoT is
+effective for boosting the performance of document understanding. Meanwhile, it
+brings a certain degree of interpretability, which could facilitate manual
+inspection and correction. Experiments on standard benchmarks show that the
+proposed LayoutLLM significantly outperforms existing methods that adopt
+open-source 7B LLMs/MLLMs for document understanding. The training data of the
+LayoutLLM is publicly available at
+https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM",cs.CV,"['cs.CV', 'cs.CL']"
+DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance,Zixuan Wang · Jia Jia · Shikun Sun · Haozhe Wu · Rong Han · Zhenyu Li · Di Tang · Jiaqing Zhou · Jiebo Luo, ,https://arxiv.org/abs/2403.13667,,2403.13667.pdf,DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance,"Choreographers determine what the dances look like, while cameramen determine
+the final presentation of dances. Recently, various methods and datasets have
+showcased the feasibility of dance synthesis. However, camera movement
+synthesis with music and dance remains an unsolved challenging problem due to
+the scarcity of paired data. Thus, we present DCM, a new multi-modal 3D
+dataset, which for the first time combines camera movement with dance motion
+and music audio. This dataset encompasses 108 dance sequences (3.2 hours) of
+paired dance-camera-music data from the anime community, covering 4 music
+genres. With this dataset, we uncover that dance camera movement is
+multifaceted and human-centric, and possesses multiple influencing factors,
+making dance camera synthesis a more challenging task compared to camera or
+dance synthesis alone. To overcome these difficulties, we propose
+DanceCamera3D, a transformer-based diffusion model that incorporates a novel
+body attention loss and a condition separation strategy. For evaluation, we
+devise new metrics measuring camera movement quality, diversity, and dancer
+fidelity. Utilizing these metrics, we conduct extensive experiments on our DCM
+dataset, providing both quantitative and qualitative evidence showcasing the
+effectiveness of our DanceCamera3D model. Code and video demos are available at
+https://github.com/Carmenw1203/DanceCamera3D-Official.",cs.CV,"['cs.CV', 'cs.MM']"
+Active Prompt Learning in Vision Language Models,Jihwan Bang · Sumyeong Ahn · Jae-Gil Lee, ,https://arxiv.org/abs/2311.11178,,2311.11178.pdf,Active Prompt Learning in Vision Language Models,"Pre-trained Vision Language Models (VLMs) have demonstrated notable progress
+in various zero-shot tasks, such as classification and retrieval. Despite their
+performance, because improving performance on new tasks requires task-specific
+knowledge, their adaptation is essential. While labels are needed for the
+adaptation, acquiring them is typically expensive. To overcome this challenge,
+active learning, a method of achieving a high performance by obtaining labels
+for a small number of samples from experts, has been studied. Active learning
+primarily focuses on selecting unlabeled samples for labeling and leveraging
+them to train models. In this study, we pose the question, ""how can the
+pre-trained VLMs be adapted under the active learning framework?"" In response
+to this inquiry, we observe that (1) simply applying a conventional active
+learning framework to pre-trained VLMs even may degrade performance compared to
+random selection because of the class imbalance in labeling candidates, and (2)
+the knowledge of VLMs can provide hints for achieving the balance before
+labeling. Based on these observations, we devise a novel active learning
+framework for VLMs, denoted as PCB. To assess the effectiveness of our
+approach, we conduct experiments on seven different real-world datasets, and
+the results demonstrate that PCB surpasses conventional active learning and
+random sampling methods. Code will be available in
+https://github.com/kaist-dmlab/pcb .",cs.CV,['cs.CV']
+One-Prompt to Segment All Medical Images,Wu · Min Xu, ,https://arxiv.org/html/2305.10300v3,,2305.10300v3.pdf,One-Prompt to Segment All Medical Images,"Large foundation models, known for their strong zero-shot generalization,
+have excelled in visual and language applications. However, applying them to
+medical image segmentation, a domain with diverse imaging types and target
+labels, remains an open challenge. Current approaches, such as adapting
+interactive segmentation models like Segment Anything Model (SAM), require user
+prompts for each sample during inference. Alternatively, transfer learning
+methods like few/one-shot models demand labeled samples, leading to high costs.
+This paper introduces a new paradigm toward the universal medical image
+segmentation, termed 'One-Prompt Segmentation.' One-Prompt Segmentation
+combines the strengths of one-shot and interactive methods. In the inference
+stage, with just \textbf{one prompted sample}, it can adeptly handle the unseen
+task in a single forward pass. We train One-Prompt Model on 64 open-source
+medical datasets, accompanied by the collection of over 3,000 clinician-labeled
+prompts. Tested on 14 previously unseen tasks, the One-Prompt Model showcases
+superior zero-shot segmentation capabilities, outperforming a wide range of
+related methods. The code and annotated data will be publicly released.",eess.IV,"['eess.IV', 'cs.CV']"
+Reconstructing Hands in 3D with Transformers,Georgios Pavlakos · Dandan Shan · Ilija Radosavovic · Angjoo Kanazawa · David Fouhey · Jitendra Malik, ,https://arxiv.org/abs/2312.05251,,2312.05251.pdf,Reconstructing Hands in 3D with Transformers,"We present an approach that can reconstruct hands in 3D from monocular input.
+Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based
+architecture and can analyze hands with significantly increased accuracy and
+robustness compared to previous work. The key to HaMeR's success lies in
+scaling up both the data used for training and the capacity of the deep network
+for hand reconstruction. For training data, we combine multiple datasets that
+contain 2D or 3D hand annotations. For the deep model, we use a large scale
+Vision Transformer architecture. Our final model consistently outperforms the
+previous baselines on popular 3D hand pose benchmarks. To further evaluate the
+effect of our design in non-controlled settings, we annotate existing
+in-the-wild datasets with 2D hand keypoint annotations. On this newly collected
+dataset of annotations, HInt, we demonstrate significant improvements over
+existing baselines. We make our code, data and models available on the project
+website: https://geopavlakos.github.io/hamer/.",cs.CV,['cs.CV']
+Can Biases in ImageNet Models Explain Generalization?,Paul Gavrikov · Janis Keuper,https://github.com/paulgavrikov/biases_vs_generalization,https://arxiv.org/abs/2404.01509,,2404.01509.pdf,Can Biases in ImageNet Models Explain Generalization?,"The robust generalization of models to rare, in-distribution (ID) samples
+drawn from the long tail of the training distribution and to
+out-of-training-distribution (OOD) samples is one of the major challenges of
+current deep learning methods. For image classification, this manifests in the
+existence of adversarial attacks, the performance drops on distorted images,
+and a lack of generalization to concepts such as sketches. The current
+understanding of generalization in neural networks is very limited, but some
+biases that differentiate models from human vision have been identified and
+might be causing these limitations. Consequently, several attempts with varying
+success have been made to reduce these biases during training to improve
+generalization. We take a step back and sanity-check these attempts. Fixing the
+architecture to the well-established ResNet-50, we perform a large-scale study
+on 48 ImageNet models obtained via different training methods to understand how
+and if these biases - including shape bias, spectral biases, and critical bands
+- interact with generalization. Our extensive study results reveal that
+contrary to previous findings, these biases are insufficient to accurately
+predict the generalization of a model holistically. We provide access to all
+checkpoints and evaluation code at
+https://github.com/paulgavrikov/biases_vs_generalization",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'stat.ML']"
+"Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA",Zhuowan Li · Bhavan Jasani · Peng Tang · Shabnam Ghadar, ,https://arxiv.org/abs/2403.16385,,2403.16385.pdf,"Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA","Understanding data visualizations like charts and plots requires reasoning
+about both visual elements and numerics. Although strong in extractive
+questions, current chart visual question answering (chart VQA) models suffer on
+complex reasoning questions. In this work, we address the lack of reasoning
+ability by data augmentation. We leverage Large Language Models (LLMs), which
+have shown to have strong reasoning ability, as an automatic data annotator
+that generates question-answer annotations for chart images. The key innovation
+in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data
+generator learns to decompose the complex question into step-by-step
+sub-questions (rationales), which are then used to derive the final answer
+using external tools, i.e. Python. This step-wise generation procedure is
+trained on synthetic data generated using a template-based QA generation
+pipeline. Experimental results highlight the significance of the proposed
+step-by-step generation. By training with the LLM-augmented data (LAMENDA), we
+significantly enhance the chart VQA models, achieving the state-of-the-art
+accuracy on the ChartQA and PlotQA datasets. In particular, our approach
+improves the accuracy of the previous state-of-the-art approach from 38% to 54%
+on the human-written questions in the ChartQA dataset, which needs strong
+reasoning. We hope our work underscores the potential of synthetic data and
+encourages further exploration of data augmentation using LLMs for
+reasoning-heavy tasks.",cs.CV,"['cs.CV', 'cs.CL']"
+TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video,Minye Wu · Zehao Wang · Georgios Kouros · Tinne Tuytelaars, ,https://arxiv.org/abs/2312.06713,,2312.06713.pdf,TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video,"Neural Radiance Fields (NeRF) revolutionize the realm of visual media by
+providing photorealistic Free-Viewpoint Video (FVV) experiences, offering
+viewers unparalleled immersion and interactivity. However, the technology's
+significant storage requirements and the computational complexity involved in
+generation and rendering currently limit its broader application. To close this
+gap, this paper presents Temporal Tri-Plane Radiance Fields (TeTriRF), a novel
+technology that significantly reduces the storage size for Free-Viewpoint Video
+(FVV) while maintaining low-cost generation and rendering. TeTriRF introduces a
+hybrid representation with tri-planes and voxel grids to support scaling up to
+long-duration sequences and scenes with complex motions or rapid changes. We
+propose a group training scheme tailored to achieving high training efficiency
+and yielding temporally consistent, low-entropy scene representations.
+Leveraging these properties of the representations, we introduce a compression
+pipeline with off-the-shelf video codecs, achieving an order of magnitude less
+storage size compared to the state-of-the-art. Our experiments demonstrate that
+TeTriRF can achieve competitive quality with a higher compression rate.",cs.CV,['cs.CV']
+Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation,Alexander Raistrick · Lingjie Mei · Karhan Kayan · David Yan · Yiming Zuo · Beining Han · Hongyu Wen · Meenal Parakh · Stamatis Alexandropoulos · Lahav Lipson · Zeyu Ma · Jia Deng, ,https://arxiv.org/abs/2306.09310,,2306.09310.pdf,Infinite Photorealistic Worlds using Procedural Generation,"We introduce Infinigen, a procedural generator of photorealistic 3D scenes of
+the natural world. Infinigen is entirely procedural: every asset, from shape to
+texture, is generated from scratch via randomized mathematical rules, using no
+external source and allowing infinite variation and composition. Infinigen
+offers broad coverage of objects and scenes in the natural world including
+plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and
+snow. Infinigen can be used to generate unlimited, diverse training data for a
+wide range of computer vision tasks including object detection, semantic
+segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a
+useful resource for computer vision research and beyond. Please visit
+https://infinigen.org for videos, code and pre-generated data.",cs.CV,['cs.CV']
+TetraSphere: A Neural Descriptor for O(3)-Invariant Point Cloud Analysis,Pavlo Melnyk · Andreas Robinson · Michael Felsberg · Mårten Wadenbäck,https://github.com/pavlo-melnyk/tetrasphere,,https://www.youtube.com/watch?v=MRJr0V7eMj8,,,,,nan
+Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft,Hao Li · Xue Yang · Zhaokai Wang · Xizhou Zhu · Jie Zhou · Yu Qiao · Xiaogang Wang · Hongsheng Li · Lewei Lu · Jifeng Dai,https://yangxue0827.github.io/auto_mc-reward.html,https://arxiv.org/abs/2312.09238,,2312.09238.pdf,Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft,"Many reinforcement learning environments (e.g., Minecraft) provide only
+sparse rewards that indicate task completion or failure with binary values. The
+challenge in exploration efficiency in such environments makes it difficult for
+reinforcement-learning-based agents to learn complex tasks. To address this,
+this paper introduces an advanced learning system, named Auto MC-Reward, that
+leverages Large Language Models (LLMs) to automatically design dense reward
+functions, thereby enhancing the learning efficiency. Auto MC-Reward consists
+of three important components: Reward Designer, Reward Critic, and Trajectory
+Analyzer. Given the environment information and task descriptions, the Reward
+Designer first design the reward function by coding an executable Python
+function with predefined observation inputs. Then, our Reward Critic will be
+responsible for verifying the code, checking whether the code is
+self-consistent and free of syntax and semantic errors. Further, the Trajectory
+Analyzer summarizes possible failure causes and provides refinement suggestions
+according to collected trajectories. In the next round, Reward Designer will
+further refine and iterate the dense reward function based on feedback.
+Experiments demonstrate a significant improvement in the success rate and
+learning efficiency of our agents in complex tasks in Minecraft, such as
+obtaining diamond with the efficient ability to avoid lava, and efficiently
+explore trees and animals that are sparse in the plains biome.",cs.AI,"['cs.AI', 'cs.CL', 'cs.CV', 'cs.LG']"
+Practical Measurements of Translucent Materials with Inter-Pixel Translucency Prior,Zhenyu Chen · Jie Guo · Shuichang Lai · Ruoyu Fu · mengxun kong · Chen Wang · Hongyu Sun · Zhebin Zhang · Chen Li · Yanwen Guo, ,,https://github.com/ZhenyuChen1999/IPTNet,,,,,nan
+Dual-View Visual Contextualization for Web Navigation,Jihyung Kil · Chan Hee Song · Boyuan Zheng · Xiang Deng · Yu Su · Wei-Lun Chao, ,https://arxiv.org/abs/2402.04476,,2402.04476.pdf,Dual-View Visual Contextualization for Web Navigation,"Automatic web navigation aims to build a web agent that can follow language
+instructions to execute complex and diverse tasks on real-world websites.
+Existing work primarily takes HTML documents as input, which define the
+contents and action spaces (i.e., actionable elements and operations) of
+webpages. Nevertheless, HTML documents may not provide a clear task-related
+context for each element, making it hard to select the right (sequence of)
+actions. In this paper, we propose to contextualize HTML elements through their
+""dual views"" in webpage screenshots: each HTML element has its corresponding
+bounding box and visual content in the screenshot. We build upon the insight --
+web developers tend to arrange task-related elements nearby on webpages to
+enhance user experiences -- and propose to contextualize each element with its
+neighbor elements, using both textual and visual features. The resulting
+representations of HTML elements are more informative for the agent to take
+action. We validate our method on the recently released Mind2Web dataset, which
+features diverse navigation domains and tasks on real-world websites. Our
+method consistently outperforms the baseline in all the scenarios, including
+cross-task, cross-website, and cross-domain ones.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+CAT: Exploiting Inter-Class Dynamics for Domain Adaptive Object Detection,Mikhail Kennerley · Jian-Gang Wang · Bharadwaj Veeravalli · Robby T. Tan,https://www.mikhailkennerley.com/cat,https://arxiv.org/abs/2403.19278v1,,2403.19278v1.pdf,CAT: Exploiting Inter-Class Dynamics for Domain Adaptive Object Detection,"Domain adaptive object detection aims to adapt detection models to domains
+where annotated data is unavailable. Existing methods have been proposed to
+address the domain gap using the semi-supervised student-teacher framework.
+However, a fundamental issue arises from the class imbalance in the labelled
+training set, which can result in inaccurate pseudo-labels. The relationship
+between classes, especially where one class is a majority and the other
+minority, has a large impact on class bias. We propose Class-Aware Teacher
+(CAT) to address the class bias issue in the domain adaptation setting. In our
+work, we approximate the class relationships with our Inter-Class Relation
+module (ICRm) and exploit it to reduce the bias within the model. In this way,
+we are able to apply augmentations to highly related classes, both inter- and
+intra-domain, to boost the performance of minority classes while having minimal
+impact on majority classes. We further reduce the bias by implementing a
+class-relation weight to our classification loss. Experiments conducted on
+various datasets and ablation studies show that our method is able to address
+the class bias in the domain adaptation setting. On the Cityscapes to Foggy
+Cityscapes dataset, we attained a 52.5 mAP, a substantial improvement over the
+51.2 mAP achieved by the state-of-the-art method.",cs.CV,['cs.CV']
+HOISDF: Constraining 3D Hand Object Pose Estimation with Global Signed Distance Fields,Haozhe Qi · Chen Zhao · Mathieu Salzmann · Alexander Mathis, ,https://arxiv.org/abs/2402.17062,,2402.17062.pdf,HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields,"Human hands are highly articulated and versatile at handling objects. Jointly
+estimating the 3D poses of a hand and the object it manipulates from a
+monocular camera is challenging due to frequent occlusions. Thus, existing
+methods often rely on intermediate 3D shape representations to increase
+performance. These representations are typically explicit, such as 3D point
+clouds or meshes, and thus provide information in the direct surroundings of
+the intermediate hand pose estimate. To address this, we introduce HOISDF, a
+Signed Distance Field (SDF) guided hand-object pose estimation network, which
+jointly exploits hand and object SDFs to provide a global, implicit
+representation over the complete reconstruction volume. Specifically, the role
+of the SDFs is threefold: equip the visual encoder with implicit shape
+information, help to encode hand-object interactions, and guide the hand and
+object pose regression via SDF-based sampling and by augmenting the feature
+representations. We show that HOISDF achieves state-of-the-art results on
+hand-object pose estimation benchmarks (DexYCB and HO3Dv2). Code is available
+at https://github.com/amathislab/HOISDF",cs.CV,['cs.CV']
+Learning Object State Changes in Videos: An Open-World Perspective,Zihui Xue · Kumar Ashutosh · Kristen Grauman,https://vision.cs.utexas.edu/projects/VidOSC/,https://arxiv.org/abs/2312.11782,,2312.11782.pdf,Learning Object State Changes in Videos: An Open-World Perspective,"Object State Changes (OSCs) are pivotal for video understanding. While humans
+can effortlessly generalize OSC understanding from familiar to unknown objects,
+current approaches are confined to a closed vocabulary. Addressing this gap, we
+introduce a novel open-world formulation for the video OSC problem. The goal is
+to temporally localize the three stages of an OSC -- the object's initial
+state, its transitioning state, and its end state -- whether or not the object
+has been observed during training. Towards this end, we develop VidOSC, a
+holistic learning approach that: (1) leverages text and vision-language models
+for supervisory signals to obviate manually labeling OSC training data, and (2)
+abstracts fine-grained shared state representations from objects to enhance
+generalization. Furthermore, we present HowToChange, the first open-world
+benchmark for video OSC localization, which offers an order of magnitude
+increase in the label space and annotation volume compared to the best existing
+benchmark. Experimental results demonstrate the efficacy of our approach, in
+both traditional closed-world and open-world scenarios.",cs.CV,['cs.CV']
+Depth Prompting for Sensor-Agnostic Depth Estimation,Jin-Hwi Park · Chanhwi Jeong · Junoh Lee · Hae-Gon Jeon, ,https://arxiv.org/abs/2405.11867,,2405.11867.pdf,Depth Prompting for Sensor-Agnostic Depth Estimation,"Dense depth maps have been used as a key element of visual perception tasks.
+There have been tremendous efforts to enhance the depth quality, ranging from
+optimization-based to learning-based methods. Despite the remarkable progress
+for a long time, their applicability in the real world is limited due to
+systematic measurement biases such as density, sensing pattern, and scan range.
+It is well-known that the biases make it difficult for these methods to achieve
+their generalization. We observe that learning a joint representation for input
+modalities (e.g., images and depth), which most recent methods adopt, is
+sensitive to the biases. In this work, we disentangle those modalities to
+mitigate the biases with prompt engineering. For this, we design a novel depth
+prompt module to allow the desirable feature representation according to new
+depth distributions from either sensor types or scene configurations. Our depth
+prompt can be embedded into foundation models for monocular depth estimation.
+Through this embedding process, our method helps the pretrained model to be
+free from restraint of depth scan range and to provide absolute scale depth
+maps. We demonstrate the effectiveness of our method through extensive
+evaluations. Source code is publicly available at
+https://github.com/JinhwiPark/DepthPrompting .",cs.CV,"['cs.CV', 'cs.LG', 'cs.RO']"
+PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought,Junyi Yao · Yijiang Liu · Zhen Dong · Mingfei Guo · Helan Hu · Kurt Keutzer · Li Du · Daquan Zhou · Shanghang Zhang, ,https://arxiv.org/abs/2307.13339,,2307.13339.pdf,Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions,"Chain-of-thought (CoT) prompting has been shown to empirically improve the
+accuracy of large language models (LLMs) on various question answering tasks.
+While understanding why CoT prompting is effective is crucial to ensuring that
+this phenomenon is a consequence of desired model behavior, little work has
+addressed this; nonetheless, such an understanding is a critical prerequisite
+for responsible model deployment. We address this question by leveraging
+gradient-based feature attribution methods which produce saliency scores that
+capture the influence of input tokens on model output. Specifically, we probe
+several open-source LLMs to investigate whether CoT prompting affects the
+relative importances they assign to particular input tokens. Our results
+indicate that while CoT prompting does not increase the magnitude of saliency
+scores attributed to semantically relevant tokens in the prompt compared to
+standard few-shot prompting, it increases the robustness of saliency scores to
+question perturbations and variations in model output.",cs.CL,"['cs.CL', 'cs.AI']"
+Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation,Wenxiao Deng · Wenbin Li · Tianyu Ding · Lei Wang · Hongguang Zhang · Kuihua Huang · Jing Huo · Yang Gao, ,https://arxiv.org/abs/2404.00563,,2404.00563.pdf,Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation,"Dataset distillation has emerged as a promising approach in deep learning,
+enabling efficient training with small synthetic datasets derived from larger
+real ones. Particularly, distribution matching-based distillation methods
+attract attention thanks to its effectiveness and low computational cost.
+However, these methods face two primary limitations: the dispersed feature
+distribution within the same class in synthetic datasets, reducing class
+discrimination, and an exclusive focus on mean feature consistency, lacking
+precision and comprehensiveness. To address these challenges, we introduce two
+novel constraints: a class centralization constraint and a covariance matching
+constraint. The class centralization constraint aims to enhance class
+discrimination by more closely clustering samples within classes. The
+covariance matching constraint seeks to achieve more accurate feature
+distribution matching between real and synthetic datasets through local feature
+covariance matrices, particularly beneficial when sample sizes are much smaller
+than the number of features. Experiments demonstrate notable improvements with
+these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9%
+on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the
+state-of-the-art relevant methods. In addition, our method maintains robust
+performance in cross-architecture settings, with a maximum performance drop of
+1.7% on four architectures. Code is available at
+https://github.com/VincenDen/IID.",cs.CV,['cs.CV']
+MeshPose: Unifying DensePose and 3D Body Mesh reconstruction,Eric-Tuan Le · Antonios Kakolyris · Petros Koutras · Himmy Tam · Efstratios Skordos · George Papandreou · Riza Alp Guler · Iasonas Kokkinos, ,https://arxiv.org/abs/2308.10305,,2308.10305.pdf,Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video,"Despite significant progress in single image-based 3D human mesh recovery,
+accurately and smoothly recovering 3D human motion from a video remains
+challenging. Existing video-based methods generally recover human mesh by
+estimating the complex pose and shape parameters from coupled image features,
+whose high complexity and low representation ability often result in
+inconsistent pose motion and limited shape patterns. To alleviate this issue,
+we introduce 3D pose as the intermediary and propose a Pose and Mesh
+Co-Evolution network (PMCE) that decouples this task into two parts: 1)
+video-based 3D human pose estimation and 2) mesh vertices regression from the
+estimated 3D pose and temporal image feature. Specifically, we propose a
+two-stream encoder that estimates mid-frame 3D pose and extracts a temporal
+image feature from the input image sequence. In addition, we design a
+co-evolution decoder that performs pose and mesh interactions with the
+image-guided Adaptive Layer Normalization (AdaLN) to make pose and mesh fit the
+human body shape. Extensive experiments demonstrate that the proposed PMCE
+outperforms previous state-of-the-art methods in terms of both per-frame
+accuracy and temporal consistency on three benchmark datasets: 3DPW, Human3.6M,
+and MPI-INF-3DHP. Our code is available at https://github.com/kasvii/PMCE.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces,Haithem Turki · Vasu Agrawal · Samuel Rota Bulò · Lorenzo Porzi · Peter Kontschieder · Deva Ramanan · Michael Zollhoefer · Christian Richardt,https://haithemturki.com/hybrid-nerf/,https://arxiv.org/abs/2312.03160,,2312.03160.pdf,HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces,"Neural radiance fields provide state-of-the-art view synthesis quality but
+tend to be slow to render. One reason is that they make use of volume
+rendering, thus requiring many samples (and model queries) per ray at render
+time. Although this representation is flexible and easy to optimize, most
+real-world objects can be modeled more efficiently with surfaces instead of
+volumes, requiring far fewer samples per ray. This observation has spurred
+considerable progress in surface representations such as signed distance
+functions, but these may struggle to model semi-opaque and thin structures. We
+propose a method, HybridNeRF, that leverages the strengths of both
+representations by rendering most objects as surfaces while modeling the
+(typically) small fraction of challenging regions volumetrically. We evaluate
+HybridNeRF against the challenging Eyeful Tower dataset along with other
+commonly used view synthesis datasets. When comparing to state-of-the-art
+baselines, including recent rasterization-based approaches, we improve error
+rates by 15-30% while achieving real-time framerates (at least 36 FPS) for
+virtual-reality resolutions (2Kx2K).",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+LTA-PCS: Learnable Task-Agnostic Point Cloud Sampling,Jiaheng Liu · Jianhao Li · Kaisiyuan Wang · Hongcheng Guo · Jian Yang · Junran Peng · Ke Xu · Xianglong Liu · Jinyang Guo, ,https://arxiv.org/abs/2404.00857,,2404.00857.pdf,Meta Episodic learning with Dynamic Task Sampling for CLIP-based Point Cloud Classification,"Point cloud classification refers to the process of assigning semantic labels
+or categories to individual points within a point cloud data structure. Recent
+works have explored the extension of pre-trained CLIP to 3D recognition. In
+this direction, CLIP-based point cloud models like PointCLIP, CLIP2Point have
+become state-of-the-art methods in the few-shot setup. Although these methods
+show promising performance for some classes like airplanes, desks, guitars,
+etc, the performance for some classes like the cup, flower pot, sink,
+nightstand, etc is still far from satisfactory. This is due to the fact that
+the adapter of CLIP-based models is trained using randomly sampled N-way K-shot
+data in the standard supervised learning setup. In this paper, we propose a
+novel meta-episodic learning framework for CLIP-based point cloud
+classification, addressing the challenges of limited training examples and
+sampling unknown classes. Additionally, we introduce dynamic task sampling
+within the episode based on performance memory. This sampling strategy
+effectively addresses the challenge of sampling unknown classes, ensuring that
+the model learns from a diverse range of classes and promotes the exploration
+of underrepresented categories. By dynamically updating the performance memory,
+we adaptively prioritize the sampling of classes based on their performance,
+enhancing the model's ability to handle challenging and real-world scenarios.
+Experiments show an average performance gain of 3-6\% on ModelNet40 and
+ScanobjectNN datasets in a few-shot setup.",cs.CV,['cs.CV']
+OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers,Han Liang · Jiacheng Bao · Ruichi Zhang · Sihan Ren · Yuecheng Xu · Sibei Yang · Xin Chen · Jingyi Yu · Lan Xu, ,https://arxiv.org/abs/2312.08985v3,,2312.08985v3.pdf,OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers,"We have recently seen tremendous progress in realistic text-to-motion
+generation. Yet, the existing methods often fail or produce implausible motions
+with unseen text inputs, which limits the applications. In this paper, we
+present OMG, a novel framework, which enables compelling motion generation from
+zero-shot open-vocabulary text prompts. Our key idea is to carefully tailor the
+pretrain-then-finetune paradigm into the text-to-motion generation. At the
+pre-training stage, our model improves the generation ability by learning the
+rich out-of-domain inherent motion traits. To this end, we scale up a large
+unconditional diffusion model up to 1B parameters, so as to utilize the massive
+unlabeled motion data up to over 20M motion instances. At the subsequent
+fine-tuning stage, we introduce motion ControlNet, which incorporates text
+prompts as conditioning information, through a trainable copy of the
+pre-trained model and the proposed novel Mixture-of-Controllers (MoC) block.
+MoC block adaptively recognizes various ranges of the sub-motions with a
+cross-attention mechanism and processes them separately with the
+text-token-specific experts. Such a design effectively aligns the CLIP token
+embeddings of text prompts to various ranges of compact and expressive motion
+features. Extensive experiments demonstrate that our OMG achieves significant
+improvements over the state-of-the-art methods on zero-shot text-to-motion
+generation. Project page: https://tr3e.github.io/omg-page.",cs.CV,['cs.CV']
+FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning,Gihun Lee · Minchan Jeong · SangMook Kim · Jaehoon Oh · Se-Young Yun, ,https://arxiv.org/abs/2308.12532v6,,2308.12532v6.pdf,FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning,"Federated Learning (FL) aggregates locally trained models from individual
+clients to construct a global model. While FL enables learning a model with
+data privacy, it often suffers from significant performance degradation when
+clients have heterogeneous data distributions. This data heterogeneity causes
+the model to forget the global knowledge acquired from previously sampled
+clients after being trained on local datasets. Although the introduction of
+proximal objectives in local updates helps to preserve global knowledge, it can
+also hinder local learning by interfering with local objectives. To address
+this problem, we propose a novel method, Federated Stabilized Orthogonal
+Learning (FedSOL), which adopts an orthogonal learning strategy to balance the
+two conflicting objectives. FedSOL is designed to identify gradients of local
+objectives that are inherently orthogonal to directions affecting the proximal
+objective. Specifically, FedSOL targets parameter regions where learning on the
+local objective is minimally influenced by proximal weight perturbations. Our
+experiments demonstrate that FedSOL consistently achieves state-of-the-art
+performance across various scenarios.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+NC-SDF: Enhancing Indoor Scene Reconstruction Using Neural SDFs with View-Dependent Normal Compensation,Ziyi Chen · Xiaolong Wu · Yu Zhang, ,https://arxiv.org/abs/2405.00340,,2405.00340.pdf,NC-SDF: Enhancing Indoor Scene Reconstruction Using Neural SDFs with View-Dependent Normal Compensation,"State-of-the-art neural implicit surface representations have achieved
+impressive results in indoor scene reconstruction by incorporating monocular
+geometric priors as additional supervision. However, we have observed that
+multi-view inconsistency between such priors poses a challenge for high-quality
+reconstructions. In response, we present NC-SDF, a neural signed distance field
+(SDF) 3D reconstruction framework with view-dependent normal compensation (NC).
+Specifically, we integrate view-dependent biases in monocular normal priors
+into the neural implicit representation of the scene. By adaptively learning
+and correcting the biases, our NC-SDF effectively mitigates the adverse impact
+of inconsistent supervision, enhancing both the global consistency and local
+details in the reconstructions. To further refine the details, we introduce an
+informative pixel sampling strategy to pay more attention to intricate geometry
+with higher information content. Additionally, we design a hybrid geometry
+modeling approach to improve the neural implicit representation. Experiments on
+synthetic and real-world datasets demonstrate that NC-SDF outperforms existing
+approaches in terms of reconstruction quality.",cs.CV,['cs.CV']
+GLID: Pre-training a Generalist Encoder-Decoder Vision Model,Jihao Liu · Jinliang Zheng · Yu Liu · Hongsheng Li,https://arxiv.org/abs/2404.07603,https://arxiv.org/abs/2404.07603,,2404.07603.pdf,GLID: Pre-training a Generalist Encoder-Decoder Vision Model,"This paper proposes a GeneraLIst encoder-Decoder (GLID) pre-training method
+for better handling various downstream computer vision tasks. While
+self-supervised pre-training approaches, e.g., Masked Autoencoder, have shown
+success in transfer learning, task-specific sub-architectures are still
+required to be appended for different downstream tasks, which cannot enjoy the
+benefits of large-scale pre-training. GLID overcomes this challenge by allowing
+the pre-trained generalist encoder-decoder to be fine-tuned on various vision
+tasks with minimal task-specific architecture modifications. In the GLID
+training scheme, pre-training pretext task and other downstream tasks are
+modeled as ""query-to-answer"" problems, including the pre-training pretext task
+and other downstream tasks. We pre-train a task-agnostic encoder-decoder with
+query-mask pairs. During fine-tuning, GLID maintains the pre-trained
+encoder-decoder and queries, only replacing the topmost linear transformation
+layer with task-specific linear heads. This minimizes the pretrain-finetune
+architecture inconsistency and enables the pre-trained model to better adapt to
+downstream tasks. GLID achieves competitive performance on various vision
+tasks, including object detection, image segmentation, pose estimation, and
+depth estimation, outperforming or matching specialist models such as
+Mask2Former, DETR, ViTPose, and BinsFormer.",cs.CV,['cs.CV']
+Your Transferability Barrier is Fragile: Free-Lunch for Transferring the Non-Transferable Learning,Ziming Hong · Li Shen · Tongliang Liu, ,,https://openreview.net/forum?id=FYKVPOHCpE,,,,,nan
+Steganographic Passport: An Owner and User Verifiable Credential for Deep Model IP Protection Without Retraining,Qi Cui · Ruohan Meng · Chaohui Xu · Chip Hong Chang,https://github.com/TracyCuiq/Steganographic-Passport,https://arxiv.org/abs/2404.02889,,2404.02889.pdf,Steganographic Passport: An Owner and User Verifiable Credential for Deep Model IP Protection Without Retraining,"Ensuring the legal usage of deep models is crucial to promoting trustable,
+accountable, and responsible artificial intelligence innovation. Current
+passport-based methods that obfuscate model functionality for license-to-use
+and ownership verifications suffer from capacity and quality constraints, as
+they require retraining the owner model for new users. They are also vulnerable
+to advanced Expanded Residual Block ambiguity attacks. We propose
+Steganographic Passport, which uses an invertible steganographic network to
+decouple license-to-use from ownership verification by hiding the user's
+identity images into the owner-side passport and recovering them from their
+respective user-side passports. An irreversible and collision-resistant hash
+function is used to avoid exposing the owner-side passport from the derived
+user-side passports and increase the uniqueness of the model signature. To
+safeguard both the passport and model's weights against advanced ambiguity
+attacks, an activation-level obfuscation is proposed for the verification
+branch of the owner's model. By jointly training the verification and
+deployment branches, their weights become tightly coupled. The proposed method
+supports agile licensing of deep models by providing a strong ownership proof
+and license accountability without requiring a separate model retraining for
+the admission of every new user. Experiment results show that our
+Steganographic Passport outperforms other passport-based deep model protection
+methods in robustness against various known attacks.",cs.CR,"['cs.CR', 'cs.CV']"
+NIVeL: Neural Implicit Vector Layers for Text-to-Vector Generation,Vikas Thamizharasan · Difan Liu · Matthew Fisher · Nanxuan Zhao · Evangelos Kalogerakis · Michal Lukáč, ,https://arxiv.org/abs/2405.15217,,2405.15217.pdf,NIVeL: Neural Implicit Vector Layers for Text-to-Vector Generation,"The success of denoising diffusion models in representing rich data
+distributions over 2D raster images has prompted research on extending them to
+other data representations, such as vector graphics. Unfortunately due to their
+variable structure and scarcity of vector training data, directly applying
+diffusion models on this domain remains a challenging problem. Using
+workarounds like optimization via Score Distillation Sampling (SDS) is also
+fraught with difficulty, as vector representations are non trivial to directly
+optimize and tend to result in implausible geometries such as redundant or
+self-intersecting shapes. NIVeL addresses these challenges by reinterpreting
+the problem on an alternative, intermediate domain which preserves the
+desirable properties of vector graphics -- mainly sparsity of representation
+and resolution-independence. This alternative domain is based on neural
+implicit fields expressed in a set of decomposable, editable layers. Based on
+our experiments, NIVeL produces text-to-vector graphics results of
+significantly better quality than the state-of-the-art.",cs.CV,"['cs.CV', 'cs.GR']"
+GlitchBench: Can large multimodal models detect video game glitches?,Mohammad Reza Taesiri · Tianjun Feng · Cor-Paul Bezemer · Anh Nguyen, ,https://arxiv.org/abs/2312.05291,,2312.05291.pdf,GlitchBench: Can large multimodal models detect video game glitches?,"Large multimodal models (LMMs) have evolved from large language models (LLMs)
+to integrate multiple input modalities, such as visual inputs. This integration
+augments the capacity of LLMs for tasks requiring visual comprehension and
+reasoning. However, the extent and limitations of their enhanced abilities are
+not fully understood, especially when it comes to real-world tasks. To address
+this gap, we introduce GlitchBench, a novel benchmark derived from video game
+quality assurance tasks, to test and evaluate the reasoning capabilities of
+LMMs. Our benchmark is curated from a variety of unusual and glitched scenarios
+from video games and aims to challenge both the visual and linguistic reasoning
+powers of LMMs in detecting and interpreting out-of-the-ordinary events. We
+evaluate multiple state-of-the-art LMMs, and we show that GlitchBench presents
+a new challenge for these models. Code and data are available at:
+https://glitchbench.github.io/",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions,Chunlong Xia · Xinliang Wang · Feng Lv · Xin Hao · Yifeng Shi, ,https://arxiv.org/abs/2403.07392,,2403.07392.pdf,ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions,"Although Vision Transformer (ViT) has achieved significant success in
+computer vision, it does not perform well in dense prediction tasks due to the
+lack of inner-patch information interaction and the limited diversity of
+feature scale. Most existing studies are devoted to designing vision-specific
+transformers to solve the above problems, which introduce additional
+pre-training costs. Therefore, we present a plain, pre-training-free, and
+feature-enhanced ViT backbone with Convolutional Multi-scale feature
+interaction, named ViT-CoMer, which facilitates bidirectional interaction
+between CNN and transformer. Compared to the state-of-the-art, ViT-CoMer has
+the following advantages: (1) We inject spatial pyramid multi-receptive field
+convolutional features into the ViT architecture, which effectively alleviates
+the problems of limited local information interaction and single-feature
+representation in ViT. (2) We propose a simple and efficient CNN-Transformer
+bidirectional fusion interaction module that performs multi-scale fusion across
+hierarchical features, which is beneficial for handling dense prediction tasks.
+(3) We evaluate the performance of ViT-CoMer across various dense prediction
+tasks, different frameworks, and multiple advanced pre-training. Notably, our
+ViT-CoMer-L achieves 64.3% AP on COCO val2017 without extra training data, and
+62.1% mIoU on ADE20K val, both of which are comparable to state-of-the-art
+methods. We hope ViT-CoMer can serve as a new backbone for dense prediction
+tasks to facilitate future research. The code will be released at
+https://github.com/Traffic-X/ViT-CoMer.",cs.CV,['cs.CV']
+LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis,Zehan Zheng · Fan Lu · Weiyi Xue · Guang Chen · Changjun Jiang,https://dyfcalid.github.io/LiDAR4D,https://arxiv.org/abs/2404.02742,,2404.02742.pdf,LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis,"Although neural radiance fields (NeRFs) have achieved triumphs in image novel
+view synthesis (NVS), LiDAR NVS remains largely unexplored. Previous LiDAR NVS
+methods employ a simple shift from image NVS methods while ignoring the dynamic
+nature and the large-scale reconstruction problem of LiDAR point clouds. In
+light of this, we propose LiDAR4D, a differentiable LiDAR-only framework for
+novel space-time LiDAR view synthesis. In consideration of the sparsity and
+large-scale characteristics, we design a 4D hybrid representation combined with
+multi-planar and grid features to achieve effective reconstruction in a
+coarse-to-fine manner. Furthermore, we introduce geometric constraints derived
+from point clouds to improve temporal consistency. For the realistic synthesis
+of LiDAR point clouds, we incorporate the global optimization of ray-drop
+probability to preserve cross-region patterns. Extensive experiments on
+KITTI-360 and NuScenes datasets demonstrate the superiority of our method in
+accomplishing geometry-aware and time-consistent dynamic reconstruction. Codes
+are available at https://github.com/ispc-lab/LiDAR4D.",cs.CV,['cs.CV']
+AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation,Taeckyung Lee · Sorn Chottananurak · Taesik Gong · Sung-Ju Lee,https://nmsl.kaist.ac.kr/projects/aetta/,https://arxiv.org/abs/2404.01351,,2404.01351.pdf,AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation,"Test-time adaptation (TTA) has emerged as a viable solution to adapt
+pre-trained models to domain shifts using unlabeled test data. However, TTA
+faces challenges of adaptation failures due to its reliance on blind adaptation
+to unknown test samples in dynamic scenarios. Traditional methods for
+out-of-distribution performance estimation are limited by unrealistic
+assumptions in the TTA context, such as requiring labeled data or re-training
+models. To address this issue, we propose AETTA, a label-free accuracy
+estimation algorithm for TTA. We propose the prediction disagreement as the
+accuracy estimate, calculated by comparing the target model prediction with
+dropout inferences. We then improve the prediction disagreement to extend the
+applicability of AETTA under adaptation failures. Our extensive evaluation with
+four baselines and six TTA methods demonstrates that AETTA shows an average of
+19.8%p more accurate estimation compared with the baselines. We further
+demonstrate the effectiveness of accuracy estimation with a model recovery case
+study, showcasing the practicality of our model recovery based on accuracy
+estimation. The source code is available at https://github.com/taeckyung/AETTA.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Adversarial Distillation Based on Slack Matching and Attribution Region Alignment,Shenglin Yin · Zhen Xiao · Mingxuan Song · Jieyi Long, ,https://arxiv.org/abs/2312.08912,,2312.08912.pdf,Dataset Distillation via Adversarial Prediction Matching,"Dataset distillation is the technique of synthesizing smaller condensed
+datasets from large original datasets while retaining necessary information to
+persist the effect. In this paper, we approach the dataset distillation problem
+from a novel perspective: we regard minimizing the prediction discrepancy on
+the real data distribution between models, which are respectively trained on
+the large original dataset and on the small distilled dataset, as a conduit for
+condensing information from the raw data into the distilled version. An
+adversarial framework is proposed to solve the problem efficiently. In contrast
+to existing distillation methods involving nested optimization or long-range
+gradient unrolling, our approach hinges on single-level optimization. This
+ensures the memory efficiency of our method and provides a flexible tradeoff
+between time and memory budgets, allowing us to distil ImageNet-1K using a
+minimum of only 6.5GB of GPU memory. Under the optimal tradeoff strategy, it
+requires only 2.5$\times$ less memory and 5$\times$ less runtime compared to
+the state-of-the-art. Empirically, our method can produce synthetic datasets
+just 10% the size of the original, yet achieve, on average, 94% of the test
+accuracy of models trained on the full original datasets including ImageNet-1K,
+significantly surpassing state-of-the-art. Additionally, extensive tests reveal
+that our distilled datasets excel in cross-architecture generalization
+capabilities.",cs.CV,['cs.CV']
+ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering,Haokai Pang · Heming Zhu · Adam Kortylewski · Christian Theobalt · Marc Habermann,https://vcai.mpi-inf.mpg.de/projects/ash/,https://arxiv.org/abs/2312.05941,,2312.05941.pdf,ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering,"Real-time rendering of photorealistic and controllable human avatars stands
+as a cornerstone in Computer Vision and Graphics. While recent advances in
+neural implicit rendering have unlocked unprecedented photorealism for digital
+avatars, real-time performance has mostly been demonstrated for static scenes
+only. To address this, we propose ASH, an animatable Gaussian splatting
+approach for photorealistic rendering of dynamic humans in real-time. We
+parameterize the clothed human as animatable 3D Gaussians, which can be
+efficiently splatted into image space to generate the final rendering. However,
+naively learning the Gaussian parameters in 3D space poses a severe challenge
+in terms of compute. Instead, we attach the Gaussians onto a deformable
+character model, and learn their parameters in 2D texture space, which allows
+leveraging efficient 2D convolutional architectures that easily scale with the
+required number of Gaussians. We benchmark ASH with competing methods on
+pose-controllable avatars, demonstrating that our method outperforms existing
+real-time methods by a large margin and shows comparable or even better results
+than offline methods.",cs.CV,['cs.CV']
+Design2Cloth: 3D Cloth Generation from 2D Masks,Jiali Zheng · Rolandos Alexandros Potamias · Stefanos Zafeiriou, ,https://arxiv.org/abs/2404.02686,,2404.02686.pdf,Design2Cloth: 3D Cloth Generation from 2D Masks,"In recent years, there has been a significant shift in the field of digital
+avatar research, towards modeling, animating and reconstructing clothed human
+representations, as a key step towards creating realistic avatars. However,
+current 3D cloth generation methods are garment specific or trained completely
+on synthetic data, hence lacking fine details and realism. In this work, we
+make a step towards automatic realistic garment design and propose
+Design2Cloth, a high fidelity 3D generative model trained on a real world
+dataset from more than 2000 subject scans. To provide vital contribution to the
+fashion industry, we developed a user-friendly adversarial model capable of
+generating diverse and detailed clothes simply by drawing a 2D cloth mask.
+Under a series of both qualitative and quantitative experiments, we showcase
+that Design2Cloth outperforms current state-of-the-art cloth generative models
+by a large margin. In addition to the generative properties of our network, we
+showcase that the proposed method can be used to achieve high quality
+reconstructions from single in-the-wild images and 3D scans. Dataset, code and
+pre-trained model will become publicly available.",cs.CV,['cs.CV']
+Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfer,Wenqiao Zhang · Zheqi Lv, ,https://arxiv.org/abs/2311.12905,,2311.12905.pdf,Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfer,"Active Domain Adaptation (ADA) aims to maximally boost model adaptation in a
+new target domain by actively selecting a limited number of target data to
+annotate.This setting neglects the more practical scenario where training data
+are collected from multiple sources. This motivates us to target a new and
+challenging setting of knowledge transfer that extends ADA from a single source
+domain to multiple source domains, termed Multi-source Active Domain Adaptation
+(MADA). Not surprisingly, we find that most traditional ADA methods cannot work
+directly in such a setting, mainly due to the excessive domain gap introduced
+by all the source domains and thus their uncertainty-aware sample selection can
+easily become miscalibrated under the multi-domain shifts. Considering this, we
+propose a Dynamic integrated uncertainty valuation framework(Detective) that
+comprehensively consider the domain shift between multi-source domains and
+target domain to detect the informative target samples. Specifically, the
+leverages a dynamic Domain Adaptation(DA) model that learns how to adapt the
+model's parameters to fit the union of multi-source domains. This enables an
+approximate single-source domain modeling by the dynamic model. We then
+comprehensively measure both domain uncertainty and predictive uncertainty in
+the target domain to detect informative target samples using evidential deep
+learning, thereby mitigating uncertainty miscalibration. Furthermore, we
+introduce a contextual diversity-aware calculator to enhance the diversity of
+the selected samples. Experiments demonstrate that our solution outperforms
+existing methods by a considerable margin on three domain adaptation
+benchmarks.",cs.AI,"['cs.AI', 'cs.LG']"
+Attentive Illumination Decomposition Model for Multi-Illuminant White Balancing,Dongyoung Kim · Jinwoo Kim · Junsang Yu · Seon Joo Kim,https://www.dykim.me/projects/aid,https://arxiv.org/abs/2402.18277,,2402.18277.pdf,Attentive Illumination Decomposition Model for Multi-Illuminant White Balancing,"White balance (WB) algorithms in many commercial cameras assume single and
+uniform illumination, leading to undesirable results when multiple lighting
+sources with different chromaticities exist in the scene. Prior research on
+multi-illuminant WB typically predicts illumination at the pixel level without
+fully grasping the scene's actual lighting conditions, including the number and
+color of light sources. This often results in unnatural outcomes lacking in
+overall consistency. To handle this problem, we present a deep white balancing
+model that leverages the slot attention, where each slot is in charge of
+representing individual illuminants. This design enables the model to generate
+chromaticities and weight maps for individual illuminants, which are then fused
+to compose the final illumination map. Furthermore, we propose the
+centroid-matching loss, which regulates the activation of each slot based on
+the color range, thereby enhancing the model to separate illumination more
+effectively. Our method achieves the state-of-the-art performance on both
+single- and multi-illuminant WB benchmarks, and also offers additional
+information such as the number of illuminants in the scene and their
+chromaticity. This capability allows for illumination editing, an application
+not feasible with prior methods.",cs.CV,['cs.CV']
+Vista-LLaMA: Reliable Video Teller via Equal Distance to Visual Tokens,Fan Ma · Xiaojie Jin · Heng Wang · Yuchen Xian · Jiashi Feng · Yi Yang, ,https://arxiv.org/abs/2312.08870,,2312.08870.pdf,Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens,"Recent advances in large video-language models have displayed promising
+outcomes in video comprehension. Current approaches straightforwardly convert
+video into language tokens and employ large language models for multi-modal
+tasks. However, this method often leads to the generation of irrelevant
+content, commonly known as ""hallucination"", as the length of the text increases
+and the impact of the video diminishes. To address this problem, we propose
+Vista-LLaMA, a novel framework that maintains the consistent distance between
+all visual tokens and any language tokens, irrespective of the generated text
+length. Vista-LLaMA omits relative position encoding when determining attention
+weights between visual and text tokens, retaining the position encoding for
+text and text tokens. This amplifies the effect of visual tokens on text
+generation, especially when the relative distance is longer between visual and
+text tokens. The proposed attention mechanism significantly reduces the chance
+of producing irrelevant text related to the video content. Furthermore, we
+present a sequential visual projector that projects the current video frame
+into tokens of language space with the assistance of the previous frame. This
+approach not only captures the temporal relationship within the video, but also
+allows less visual tokens to encompass the entire video. Our approach
+significantly outperforms various previous methods (e.g., Video-ChatGPT,
+MovieChat) on four challenging open-ended video question answering benchmarks.
+We reach an accuracy of 60.7 on the zero-shot NExT-QA and 60.5 on the zero-shot
+MSRVTT-QA, setting a new state-of-the-art performance. This project is
+available at https://jinxxian.github.io/Vista-LLaMA.",cs.CV,['cs.CV']
+Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images,JungEun Kim · Hangyul Yoon · Geondo Park · Kyungsu Kim · Eunho Yang, ,https://arxiv.org/abs/2404.01464,,2404.01464.pdf,Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images,"4D medical images, which represent 3D images with temporal information, are
+crucial in clinical practice for capturing dynamic changes and monitoring
+long-term disease progression. However, acquiring 4D medical images poses
+challenges due to factors such as radiation exposure and imaging duration,
+necessitating a balance between achieving high temporal resolution and
+minimizing adverse effects. Given these circumstances, not only is data
+acquisition challenging, but increasing the frame rate for each dataset also
+proves difficult. To address this challenge, this paper proposes a simple yet
+effective Unsupervised Volumetric Interpolation framework, UVI-Net. This
+framework facilitates temporal interpolation without the need for any
+intermediate frames, distinguishing it from the majority of other existing
+unsupervised methods. Experiments on benchmark datasets demonstrate significant
+improvements across diverse evaluation metrics compared to unsupervised and
+supervised baselines. Remarkably, our approach achieves this superior
+performance even when trained with a dataset as small as one, highlighting its
+exceptional robustness and efficiency in scenarios with sparse supervision.
+This positions UVI-Net as a compelling alternative for 4D medical imaging,
+particularly in settings where data availability is limited. The source code is
+available at https://github.com/jungeun122333/UVI-Net.",eess.IV,"['eess.IV', 'cs.AI', 'cs.CV', 'cs.LG']"
+ERMVP: Communication-Efficient and Collaboration-Robust Multi-Vehicle Perception in Challenging Environments,Jingyu Zhang · Kun Yang · Yilei Wang · Hanqi Wang · Peng Sun · Liang Song, ,https://arxiv.org/abs/2307.13929v3,,2307.13929v3.pdf,Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception,"Multi-agent collaborative perception as a potential application for
+vehicle-to-everything communication could significantly improve the perception
+performance of autonomous vehicles over single-agent perception. However,
+several challenges remain in achieving pragmatic information sharing in this
+emerging research. In this paper, we propose SCOPE, a novel collaborative
+perception framework that aggregates the spatio-temporal awareness
+characteristics across on-road agents in an end-to-end manner. Specifically,
+SCOPE has three distinct strengths: i) it considers effective semantic cues of
+the temporal context to enhance current representations of the target agent;
+ii) it aggregates perceptually critical spatial information from heterogeneous
+agents and overcomes localization errors via multi-scale feature interactions;
+iii) it integrates multi-source representations of the target agent based on
+their complementary contributions by an adaptive fusion paradigm. To thoroughly
+evaluate SCOPE, we consider both real-world and simulated scenarios of
+collaborative 3D object detection tasks on three datasets. Extensive
+experiments demonstrate the superiority of our approach and the necessity of
+the proposed components.",cs.CV,['cs.CV']
+HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding,Trong-Thuan Nguyen · Pha Nguyen · Khoa Luu,https://uark-cviu.github.io/ASPIRe/,https://arxiv.org/abs/2312.03050,,2312.03050.pdf,HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding,"Visual interactivity understanding within visual scenes presents a
+significant challenge in computer vision. Existing methods focus on complex
+interactivities while leveraging a simple relationship model. These methods,
+however, struggle with a diversity of appearance, situation, position,
+interaction, and relation in videos. This limitation hinders the ability to
+fully comprehend the interplay within the complex visual dynamics of subjects.
+In this paper, we delve into interactivities understanding within visual
+content by deriving scene graph representations from dense interactivities
+among humans and objects. To achieve this goal, we first present a new dataset
+containing Appearance-Situation-Position-Interaction-Relation predicates, named
+ASPIRe, offering an extensive collection of videos marked by a wide range of
+interactivities. Then, we propose a new approach named Hierarchical
+Interlacement Graph (HIG), which leverages a unified layer and graph within a
+hierarchical structure to provide deep insights into scene changes across five
+distinct tasks. Our approach demonstrates superior performance to other methods
+through extensive experiments conducted in various scenarios.",cs.CV,['cs.CV']
+A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions,Jack Urbanek · Florian Bordes · Pietro Astolfi · Mary Williamson · Vasu Sharma · Adriana Romero-Soriano, ,https://arxiv.org/abs/2312.08578,,2312.08578.pdf,A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions,"Curation methods for massive vision-language datasets trade off between
+dataset size and quality. However, even the highest quality of available
+curated captions are far too short to capture the rich visual detail in an
+image. To show the value of dense and highly-aligned image-text pairs, we
+collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
+images human-annotated with mask-aligned descriptions averaging above 1000
+words each. With precise and reliable captions associated with specific parts
+of an image, we can evaluate vision-language models' (VLMs) understanding of
+image content with a novel task that matches each caption with its
+corresponding subcrop. As current models are often limited to 77 text tokens,
+we also introduce a summarized version (sDCI) in which each caption length is
+limited. We show that modern techniques that make progress on standard
+benchmarks do not correspond with significant improvement on our sDCI based
+benchmark. Lastly, we finetune CLIP using sDCI and show significant
+improvements over the baseline despite a small training set. By releasing the
+first human annotated dense image captioning dataset, we hope to enable the
+development of new benchmarks or fine-tuning recipes for the next generation of
+VLMs to come.",cs.CV,['cs.CV']
+Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion,Lucas Nunes · Rodrigo Marcuzzi · Benedikt Mersch · Jens Behley · Cyrill Stachniss,https://github.com/PRBonn/LiDiff,https://arxiv.org/html/2403.13470v1,,2403.13470v1.pdf,Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion,"Computer vision techniques play a central role in the perception stack of
+autonomous vehicles. Such methods are employed to perceive the vehicle
+surroundings given sensor data. 3D LiDAR sensors are commonly used to collect
+sparse 3D point clouds from the scene. However, compared to human perception,
+such systems struggle to deduce the unseen parts of the scene given those
+sparse point clouds. In this matter, the scene completion task aims at
+predicting the gaps in the LiDAR measurements to achieve a more complete scene
+representation. Given the promising results of recent diffusion models as
+generative models for images, we propose extending them to achieve scene
+completion from a single 3D LiDAR scan. Previous works used diffusion models
+over range images extracted from LiDAR data, directly applying image-based
+diffusion methods. Distinctly, we propose to directly operate on the points,
+reformulating the noising and denoising diffusion process such that it can
+efficiently work at scene scale. Together with our approach, we propose a
+regularization loss to stabilize the noise predicted during the denoising
+process. Our experimental evaluation shows that our method can complete the
+scene given a single LiDAR scan as input, producing a scene with more details
+compared to state-of-the-art scene completion methods. We believe that our
+proposed diffusion process formulation can support further research in
+diffusion models applied to scene-scale point cloud data.",cs.CV,['cs.CV']
+Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis,Willi Menapace · Aliaksandr Siarohin · Ivan Skorokhodov · Ekaterina Deyneka · Tsai-Shien Chen · Anil Kag · Yuwei Fang · Aleksei Stoliar · Elisa Ricci · Jian Ren · Sergey Tulyakov, ,https://arxiv.org/abs/2402.14797,,2402.14797.pdf,Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis,"Contemporary models for generating images show remarkable quality and
+versatility. Swayed by these advantages, the research community repurposes them
+to generate videos. Since video content is highly redundant, we argue that
+naively bringing advances of image models to the video generation domain
+reduces motion fidelity, visual quality and impairs scalability. In this work,
+we build Snap Video, a video-first model that systematically addresses these
+challenges. To do that, we first extend the EDM framework to take into account
+spatially and temporally redundant pixels and naturally support video
+generation. Second, we show that a U-Net - a workhorse behind image generation
+- scales poorly when generating videos, requiring significant computational
+overhead. Hence, we propose a new transformer-based architecture that trains
+3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
+to efficiently train a text-to-video model with billions of parameters for the
+first time, reach state-of-the-art results on a number of benchmarks, and
+generate videos with substantially higher quality, temporal consistency, and
+motion complexity. The user studies showed that our model was favored by a
+large margin over the most recent methods. See our website at
+https://snap-research.github.io/snapvideo/.",cs.CV,"['cs.CV', 'cs.AI']"
+ADFactory: An Effective Framework for Generalizing Optical Flow with NeRF,Han Ling · Quansen Sun · Yinghui Sun · Xian Xu · Xingfeng Li, ,https://arxiv.org/abs/2311.04246,,2311.04246.pdf,ADFactory: An Effective Framework for Generalizing Optical Flow with Nerf,"A significant challenge facing current optical flow methods is the difficulty
+in generalizing them well to the real world. This is mainly due to the high
+cost of hand-crafted datasets, and existing self-supervised methods are limited
+by indirect loss and occlusions, resulting in fuzzy outcomes. To address this
+challenge, we introduce a novel optical flow training framework: automatic data
+factory (ADF). ADF only requires RGB images as input to effectively train the
+optical flow network on the target data domain. Specifically, we use advanced
+Nerf technology to reconstruct scenes from photo groups collected by a
+monocular camera, and then calculate optical flow labels between camera pose
+pairs based on the rendering results. To eliminate erroneous labels caused by
+defects in the scene reconstructed by Nerf, we screened the generated labels
+from multiple aspects, such as optical flow matching accuracy, radiation field
+confidence, and depth consistency. The filtered labels can be directly used for
+network supervision. Experimentally, the generalization ability of ADF on KITTI
+surpasses existing self-supervised optical flow and monocular scene flow
+algorithms. In addition, ADF achieves impressive results in real-world
+zero-point generalization evaluations and surpasses most supervised methods.",cs.CV,['cs.CV']
+Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection,Chengjie Wang · wenbing zhu · Bin-Bin Gao · Zhenye Gan · Jiangning Zhang · Zhihao Gu · Bruce Qian · Mingang Chen · Lizhuang Ma, ,https://arxiv.org/abs/2403.12580,,2403.12580.pdf,Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection,"Industrial anomaly detection (IAD) has garnered significant attention and
+experienced rapid development. However, the recent development of IAD approach
+has encountered certain difficulties due to dataset limitations. On the one
+hand, most of the state-of-the-art methods have achieved saturation (over 99%
+in AUROC) on mainstream datasets such as MVTec, and the differences of methods
+cannot be well distinguished, leading to a significant gap between public
+datasets and actual application scenarios. On the other hand, the research on
+various new practical anomaly detection settings is limited by the scale of the
+dataset, posing a risk of overfitting in evaluation results. Therefore, we
+propose a large-scale, Real-world, and multi-view Industrial Anomaly Detection
+dataset, named Real-IAD, which contains 150K high-resolution images of 30
+different objects, an order of magnitude larger than existing datasets. It has
+a larger range of defect area and ratio proportions, making it more challenging
+than previous datasets. To make the dataset closer to real application
+scenarios, we adopted a multi-view shooting method and proposed sample-level
+evaluation metrics. In addition, beyond the general unsupervised anomaly
+detection setting, we propose a new setting for Fully Unsupervised Industrial
+Anomaly Detection (FUIAD) based on the observation that the yield rate in
+industrial production is usually greater than 60%, which has more practical
+application value. Finally, we report the results of popular IAD methods on the
+Real-IAD dataset, providing a highly challenging benchmark to promote the
+development of the IAD field.",cs.CV,['cs.CV']
+Multiview Aerial Visual RECognition (MAVREC) Dataset: Can Multi-view Improve Aerial Visual Perception?,Aritra Dutta · Srijan Das · Jacob Nielsen · RAJATSUBHRA CHAKRABORTY · Mubarak Shah, ,https://arxiv.org/abs/2312.04548,,2312.04548.pdf,Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?,"Despite the commercial abundance of UAVs, aerial data acquisition remains
+challenging, and the existing Asia and North America-centric open-source UAV
+datasets are small-scale or low-resolution and lack diversity in scene
+contextuality. Additionally, the color content of the scenes, solar-zenith
+angle, and population density of different geographies influence the data
+diversity. These two factors conjointly render suboptimal aerial-visual
+perception of the deep neural network (DNN) models trained primarily on the
+ground-view data, including the open-world foundational models.
+  To pave the way for a transformative era of aerial detection, we present
+Multiview Aerial Visual RECognition or MAVREC, a video dataset where we record
+synchronized scenes from different perspectives -- ground camera and
+drone-mounted camera. MAVREC consists of around 2.5 hours of industry-standard
+2.7K resolution video sequences, more than 0.5 million frames, and 1.1 million
+annotated bounding boxes. This makes MAVREC the largest ground and aerial-view
+dataset, and the fourth largest among all drone-based datasets across all
+modalities and tasks. Through our extensive benchmarking on MAVREC, we
+recognize that augmenting object detectors with ground-view images from the
+corresponding geographical location is a superior pre-training strategy for
+aerial detection. Building on this strategy, we benchmark MAVREC with a
+curriculum-based semi-supervised object detection approach that leverages
+labeled (ground and aerial) and unlabeled (only aerial) images to enhance the
+aerial detection. We publicly release the MAVREC dataset:
+https://mavrec.github.io.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'I.4.0; I.4.8; I.5.1; I.5.4; I.2.10']"
+SpatialTracker: Tracking Any 2D Pixels in 3D Space,Yuxi Xiao · Qianqian Wang · Shangzhan Zhang · Nan Xue · Sida Peng · Yujun Shen · Xiaowei Zhou, ,https://arxiv.org/abs/2404.04319,,2404.04319.pdf,SpatialTracker: Tracking Any 2D Pixels in 3D Space,"Recovering dense and long-range pixel motion in videos is a challenging
+problem. Part of the difficulty arises from the 3D-to-2D projection process,
+leading to occlusions and discontinuities in the 2D motion domain. While 2D
+motion can be intricate, we posit that the underlying 3D motion can often be
+simple and low-dimensional. In this work, we propose to estimate point
+trajectories in 3D space to mitigate the issues caused by image projection. Our
+method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth
+estimators, represents the 3D content of each frame efficiently using a
+triplane representation, and performs iterative updates using a transformer to
+estimate 3D trajectories. Tracking in 3D allows us to leverage
+as-rigid-as-possible (ARAP) constraints while simultaneously learning a
+rigidity embedding that clusters pixels into different rigid parts. Extensive
+evaluation shows that our approach achieves state-of-the-art tracking
+performance both qualitatively and quantitatively, particularly in challenging
+scenarios such as out-of-plane rotation.",cs.CV,['cs.CV']
+SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective,Yu-Bang Zheng · Xile Zhao · Junhua Zeng · Chao Li · Qibin Zhao · Heng-Chao Li · Ting-Zhu Huang,https://yubangzheng.github.io,,https://zhaoxile.github.io/index.html,,,,,nan
+LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge,Gongwei Chen · Leyang Shen · Rui Shao · Xiang Deng · Liqiang Nie, ,https://arxiv.org/abs/2311.11860,,2311.11860.pdf,LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge,"Multimodal Large Language Models (MLLMs) have endowed LLMs with the ability
+to perceive and understand multi-modal signals. However, most of the existing
+MLLMs mainly adopt vision encoders pretrained on coarsely aligned image-text
+pairs, leading to insufficient extraction and reasoning of visual knowledge. To
+address this issue, we devise a dual-Level vIsual knOwledge eNhanced Multimodal
+Large Language Model (LION), which empowers the MLLM by injecting visual
+knowledge in two levels. 1) Progressive incorporation of fine-grained
+spatial-aware visual knowledge. We design a vision aggregator cooperated with
+region-level vision-language (VL) tasks to incorporate fine-grained
+spatial-aware visual knowledge into the MLLM. To alleviate the conflict between
+image-level and region-level VL tasks during incorporation, we devise a
+dedicated stage-wise instruction-tuning strategy with mixture-of-adapters. This
+progressive incorporation scheme contributes to the mutual promotion between
+these two kinds of VL tasks. 2) Soft prompting of high-level semantic visual
+evidence. We facilitate the MLLM with high-level semantic visual evidence by
+leveraging diverse image tags. To mitigate the potential influence caused by
+imperfect predicted tags, we propose a soft prompting method by embedding a
+learnable token into the tailored text instruction. Comprehensive experiments
+on several multi-modal benchmarks demonstrate the superiority of our model
+(e.g., improvement of 5% accuracy on VSR and 3% CIDEr on TextCaps over
+InstructBLIP, 5% accuracy on RefCOCOg over Kosmos-2).",cs.CV,['cs.CV']
+LEAD: Learning Decomposition for Source-free Universal Domain Adaptation,Sanqing Qu · Tianpei Zou · Lianghua He · Florian Röhrbein · Alois Knoll · Guang Chen · Changjun Jiang,https://github.com/ispc-lab/LEAD,https://arxiv.org/abs/2403.03421,,2403.03421.pdf,LEAD: Learning Decomposition for Source-free Universal Domain Adaptation,"Universal Domain Adaptation (UniDA) targets knowledge transfer in the
+presence of both covariate and label shifts. Recently, Source-free Universal
+Domain Adaptation (SF-UniDA) has emerged to achieve UniDA without access to
+source data, which tends to be more practical due to data protection policies.
+The main challenge lies in determining whether covariate-shifted samples belong
+to target-private unknown categories. Existing methods tackle this either
+through hand-crafted thresholding or by developing time-consuming iterative
+clustering strategies. In this paper, we propose a new idea of LEArning
+Decomposition (LEAD), which decouples features into source-known and -unknown
+components to identify target-private data. Technically, LEAD initially
+leverages the orthogonal decomposition analysis for feature decomposition.
+Then, LEAD builds instance-level decision boundaries to adaptively identify
+target-private data. Extensive experiments across various UniDA scenarios have
+demonstrated the effectiveness and superiority of LEAD. Notably, in the OPDA
+scenario on VisDA dataset, LEAD outperforms GLC by 3.5% overall H-score and
+reduces 75% time to derive pseudo-labeling decision boundaries. Besides, LEAD
+is also appealing in that it is complementary to most existing methods. The
+code is available at https://github.com/ispc-lab/LEAD.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations,Evonne Ng · Javier Romero · Timur Bagautdinov · Shaojie Bai · Trevor Darrell · Angjoo Kanazawa · Alexander Richard, ,https://arxiv.org/abs/2401.01885,,2401.01885.pdf,From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations,"We present a framework for generating full-bodied photorealistic avatars that
+gesture according to the conversational dynamics of a dyadic interaction. Given
+speech audio, we output multiple possibilities of gestural motion for an
+individual, including face, body, and hands. The key behind our method is in
+combining the benefits of sample diversity from vector quantization with the
+high-frequency details obtained through diffusion to generate more dynamic,
+expressive motion. We visualize the generated motion using highly
+photorealistic avatars that can express crucial nuances in gestures (e.g.
+sneers and smirks). To facilitate this line of research, we introduce a
+first-of-its-kind multi-view conversational dataset that allows for
+photorealistic reconstruction. Experiments show our model generates appropriate
+and diverse gestures, outperforming both diffusion- and VQ-only methods.
+Furthermore, our perceptual evaluation highlights the importance of
+photorealism (vs. meshes) in accurately assessing subtle motion details in
+conversational gestures. Code and dataset available online.",cs.CV,['cs.CV']
+Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture,Fei Wang · Dan Guo · Kun Li · Zhun Zhong · Meng Wang, ,https://arxiv.org/abs/2403.07347,,2403.07347.pdf,Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture,"Video Motion Magnification (VMM) aims to reveal subtle and imperceptible
+motion information of objects in the macroscopic world. Prior methods directly
+model the motion field from the Eulerian perspective by Representation Learning
+that separates shape and texture or Multi-domain Learning from phase
+fluctuations. Inspired by the frequency spectrum, we observe that the
+low-frequency components with stable energy always possess spatial structure
+and less noise, making them suitable for modeling the subtle motion field. To
+this end, we present FD4MM, a new paradigm of Frequency Decoupling for Motion
+Magnification with a Multi-level Isomorphic Architecture to capture multi-level
+high-frequency details and a stable low-frequency structure (motion field) in
+video space. Since high-frequency details and subtle motions are susceptible to
+information degradation due to their inherent subtlety and unavoidable external
+interference from noise, we carefully design Sparse High/Low-pass Filters to
+enhance the integrity of details and motion structures, and a Sparse Frequency
+Mixer to promote seamless recoupling. Besides, we innovatively design a
+contrastive regularization for this task to strengthen the model's ability to
+discriminate irrelevant features, reducing undesired motion magnification.
+Extensive experiments on both Real-world and Synthetic Datasets show that our
+FD4MM outperforms SOTA methods. Meanwhile, FD4MM reduces FLOPs by 1.63$\times$
+and boosts inference speed by 1.68$\times$ than the latest method. Our code is
+available at https://github.com/Jiafei127/FD4MM.",cs.CV,['cs.CV']
+LLM-AR: When Large Language Model Meets Skeleton-Based Action Recognition,Haoxuan Qu · Yujun Cai · Jun Liu, ,https://arxiv.org/abs/2404.00532,,2404.00532.pdf,LLMs are Good Action Recognizers,"Skeleton-based action recognition has attracted lots of research attention.
+Recently, to build an accurate skeleton-based action recognizer, a variety of
+works have been proposed. Among them, some works use large model architectures
+as backbones of their recognizers to boost the skeleton data representation
+capability, while some other works pre-train their recognizers on external data
+to enrich the knowledge. In this work, we observe that large language models
+which have been extensively used in various natural language processing tasks
+generally hold both large model architectures and rich implicit knowledge.
+Motivated by this, we propose a novel LLM-AR framework, in which we investigate
+treating the Large Language Model as an Action Recognizer. In our framework, we
+propose a linguistic projection process to project each input action signal
+(i.e., each skeleton sequence) into its ``sentence format'' (i.e., an ``action
+sentence''). Moreover, we also incorporate our framework with several designs
+to further facilitate this linguistic projection process. Extensive experiments
+demonstrate the efficacy of our proposed framework.",cs.CV,['cs.CV']
+Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis,Xin Zhou · Dingkang Liang · Wei Xu · Xingkui Zhu · Yihan Xu · Zhikang Zou · Xiang Bai, ,https://arxiv.org/abs/2403.01439,,2403.01439.pdf,Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis,"Point cloud analysis has achieved outstanding performance by transferring
+point cloud pre-trained models. However, existing methods for model adaptation
+usually update all model parameters, i.e., full fine-tuning paradigm, which is
+inefficient as it relies on high computational costs (e.g., training GPU
+memory) and massive storage space. In this paper, we aim to study
+parameter-efficient transfer learning for point cloud analysis with an ideal
+trade-off between task performance and parameter efficiency. To achieve this
+goal, we freeze the parameters of the default pre-trained models and then
+propose the Dynamic Adapter, which generates a dynamic scale for each token,
+considering the token significance to the downstream task. We further
+seamlessly integrate Dynamic Adapter with Prompt Tuning (DAPT) by constructing
+Internal Prompts, capturing the instance-specific features for interaction.
+Extensive experiments conducted on five challenging datasets demonstrate that
+the proposed DAPT achieves superior performance compared to the full
+fine-tuning counterparts while significantly reducing the trainable parameters
+and training GPU memory by 95% and 35%, respectively. Code is available at
+https://github.com/LMD0311/DAPT.",cs.CV,['cs.CV']
+Link-Context Learning for Multimodal LLMs,Yan Tai · Weichen Fan · Zhao Zhang · Ziwei Liu, ,https://arxiv.org/abs/2308.07891,,2308.07891.pdf,Link-Context Learning for Multimodal LLMs,"The ability to learn from context with novel concepts, and deliver
+appropriate responses are essential in human conversations. Despite current
+Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
+trained on mega-scale datasets, recognizing unseen images or understanding
+novel concepts in a training-free manner remains a challenge. In-Context
+Learning (ICL) explores training-free few-shot learning, where models are
+encouraged to ``learn to learn"" from limited tasks and generalize to unseen
+tasks. In this work, we propose link-context learning (LCL), which emphasizes
+""reasoning from cause and effect"" to augment the learning capabilities of
+MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
+relationship between the support set and the query set. By providing
+demonstrations with causal links, LCL guides the model to discern not only the
+analogy but also the underlying causal associations between data points, which
+empowers MLLMs to recognize unseen images and understand novel concepts more
+effectively. To facilitate the evaluation of this novel approach, we introduce
+the ISEKAI dataset, comprising exclusively of unseen generated image-label
+pairs designed for link-context learning. Extensive experiments show that our
+LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
+over vanilla MLLMs. Code and data will be released at
+https://github.com/isekai-portal/Link-Context-Learning.",cs.CV,"['cs.CV', 'cs.CL']"
+Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models,Jingyao Xu · Yuetong Lu · Yandong Li · Siyang Lu · Dongdong Wang · Xiang Wei, ,https://arxiv.org/abs/2404.15081,,2404.15081.pdf,Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models,"Diffusion models (DMs) embark a new era of generative modeling and offer more
+opportunities for efficient generating high-quality and realistic data samples.
+However, their widespread use has also brought forth new challenges in model
+security, which motivates the creation of more effective adversarial attackers
+on DMs to understand its vulnerability. We propose CAAT, a simple but generic
+and efficient approach that does not require costly training to effectively
+fool latent diffusion models (LDMs). The approach is based on the observation
+that cross-attention layers exhibits higher sensitivity to gradient change,
+allowing for leveraging subtle perturbations on published images to
+significantly corrupt the generated images. We show that a subtle perturbation
+on an image can significantly impact the cross-attention layers, thus changing
+the mapping between text and image during the fine-tuning of customized
+diffusion models. Extensive experiments demonstrate that CAAT is compatible
+with diverse diffusion models and outperforms baseline attack methods in a more
+effective (more noise) and efficient (twice as fast as Anti-DreamBooth and
+Mist) manner.",cs.CV,"['cs.CV', 'cs.CR', 'cs.LG']"
+Robust Depth Enhancement via Polarization Prompt Fusion Tuning,Kei IKEMURA · Yiming Huang · Felix Heide · Zhaoxiang Zhang · Qifeng Chen · Chenyang Lei,https://lastbasket.github.io/PPFT/,https://arxiv.org/abs/2404.04318,,2404.04318.pdf,Robust Depth Enhancement via Polarization Prompt Fusion Tuning,"Existing depth sensors are imperfect and may provide inaccurate depth values
+in challenging scenarios, such as in the presence of transparent or reflective
+objects. In this work, we present a general framework that leverages
+polarization imaging to improve inaccurate depth measurements from various
+depth sensors. Previous polarization-based depth enhancement methods focus on
+utilizing pure physics-based formulas for a single sensor. In contrast, our
+method first adopts a learning-based strategy where a neural network is trained
+to estimate a dense and complete depth map from polarization data and a sensor
+depth map from different sensors. To further improve the performance, we
+propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively
+utilize RGB-based models pre-trained on large-scale datasets, as the size of
+the polarization dataset is limited to train a strong model from scratch. We
+conducted extensive experiments on a public dataset, and the results
+demonstrate that the proposed method performs favorably compared to existing
+depth enhancement baselines. Code and demos are available at
+https://lastbasket.github.io/PPFT/.",cs.CV,"['cs.CV', 'cs.AI']"
+Shadows Don’t Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for now,Ayush Sarkar · Hanlin Mai · Amitabh Mahapatra · David Forsyth · Svetlana Lazebnik · Anand Bhattad,https://projective-geometry.github.io,https://arxiv.org/abs/2311.17138,,2311.17138.pdf,Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for now,"Generative models can produce impressively realistic images. This paper
+demonstrates that generated images have geometric features different from those
+of real images. We build a set of collections of generated images, prequalified
+to fool simple, signal-based classifiers into believing they are real. We then
+show that prequalified generated images can be identified reliably by
+classifiers that only look at geometric properties. We use three such
+classifiers. All three classifiers are denied access to image pixels, and look
+only at derived geometric features. The first classifier looks at the
+perspective field of the image, the second looks at lines detected in the
+image, and the third looks at relations between detected objects and shadows.
+Our procedure detects generated images more reliably than SOTA local signal
+based detectors, for images from a number of distinct generators. Saliency maps
+suggest that the classifiers can identify geometric problems reliably. We
+conclude that current generators cannot reliably reproduce geometric properties
+of real images.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image,Chong Bao · Yinda Zhang · Yuan Li · Xiyu Zhang · Bangbang Yang · Hujun Bao · Marc Pollefeys · Guofeng Zhang · Zhaopeng Cui,https://zju3dv.github.io/geneavatar/,https://arxiv.org/abs/2404.02152,,2404.02152.pdf,GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image,"Recently, we have witnessed the explosive growth of various volumetric
+representations in modeling animatable head avatars. However, due to the
+diversity of frameworks, there is no practical method to support high-level
+applications like 3D head avatar editing across different representations. In
+this paper, we propose a generic avatar editing approach that can be
+universally applied to various 3DMM driving volumetric head avatars. To achieve
+this goal, we design a novel expression-aware modification generative model,
+which enables lift 2D editing from a single image to a consistent 3D
+modification field. To ensure the effectiveness of the generative modification
+process, we develop several techniques, including an expression-dependent
+modification distillation scheme to draw knowledge from the large-scale head
+avatar model and 2D facial texture editing tools, implicit latent space
+guidance to enhance model convergence, and a segmentation-based loss reweight
+strategy for fine-grained texture inversion. Extensive experiments demonstrate
+that our method delivers high-quality and consistent results across multiple
+expression and viewpoints. Project page: https://zju3dv.github.io/geneavatar/",cs.CV,['cs.CV']
+MarkovGen: Structured Prediction for Efficient Text-to-Image Generation,Sadeep Jayasumana · Daniel Glasner · Srikumar Ramalingam · Andreas Veit · Ayan Chakrabarti · Sanjiv Kumar, ,https://arxiv.org/abs/2308.10997,,2308.10997.pdf,MarkovGen: Structured Prediction for Efficient Text-to-Image Generation,"Modern text-to-image generation models produce high-quality images that are
+both photorealistic and faithful to the text prompts. However, this quality
+comes at significant computational cost: nearly all of these models are
+iterative and require running sampling multiple times with large models. This
+iterative process is needed to ensure that different regions of the image are
+not only aligned with the text prompt, but also compatible with each other. In
+this work, we propose a light-weight approach to achieving this compatibility
+between different regions of an image, using a Markov Random Field (MRF) model.
+We demonstrate the effectiveness of this method on top of the latent
+token-based Muse text-to-image model. The MRF richly encodes the compatibility
+among image tokens at different spatial locations to improve quality and
+significantly reduce the required number of Muse sampling steps. Inference with
+the MRF is significantly cheaper, and its parameters can be quickly learned
+through back-propagation by modeling MRF inference as a differentiable
+neural-network layer. Our full model, MarkovGen, uses this proposed MRF model
+to both speed up Muse by 1.5X and produce higher quality images by decreasing
+undesirable image artifacts.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+LAENeRF: Local Appearance Editing for Neural Radiance Fields,Lukas Radl · Michael Steiner · Andreas Kurz · Markus Steinberger,https://r4dl.github.io/LAENeRF/,https://arxiv.org/abs/2312.09913,,2312.09913.pdf,LAENeRF: Local Appearance Editing for Neural Radiance Fields,"Due to the omnipresence of Neural Radiance Fields (NeRFs), the interest
+towards editable implicit 3D representations has surged over the last years.
+However, editing implicit or hybrid representations as used for NeRFs is
+difficult due to the entanglement of appearance and geometry encoded in the
+model parameters. Despite these challenges, recent research has shown first
+promising steps towards photorealistic and non-photorealistic appearance edits.
+The main open issues of related work include limited interactivity, a lack of
+support for local edits and large memory requirements, rendering them less
+useful in practice. We address these limitations with LAENeRF, a unified
+framework for photorealistic and non-photorealistic appearance editing of
+NeRFs. To tackle local editing, we leverage a voxel grid as starting point for
+region selection. We learn a mapping from expected ray terminations to final
+output color, which can optionally be supervised by a style loss, resulting in
+a framework which can perform photorealistic and non-photorealistic appearance
+editing of selected regions. Relying on a single point per ray for our mapping,
+we limit memory requirements and enable fast optimization. To guarantee
+interactivity, we compose the output color using a set of learned, modifiable
+base colors, composed with additive layer mixing. Compared to concurrent work,
+LAENeRF enables recoloring and stylization while keeping processing time low.
+Furthermore, we demonstrate that our approach surpasses baseline methods both
+quantitatively and qualitatively.",cs.CV,['cs.CV']
+EgoGen: An Egocentric Synthetic Data Generator,Gen Li · Kaifeng Zhao · Siwei Zhang · Xiaozhong Lyu · Mihai Dusmanu · Yan Zhang · Marc Pollefeys · Siyu Tang,https://ego-gen.github.io,https://arxiv.org/abs/2401.08739,,2401.08739.pdf,EgoGen: An Egocentric Synthetic Data Generator,"Understanding the world in first-person view is fundamental in Augmented
+Reality (AR). This immersive perspective brings dramatic visual changes and
+unique challenges compared to third-person views. Synthetic data has empowered
+third-person-view vision models, but its application to embodied egocentric
+perception tasks remains largely unexplored. A critical challenge lies in
+simulating natural human movements and behaviors that effectively steer the
+embodied cameras to capture a faithful egocentric representation of the 3D
+world. To address this challenge, we introduce EgoGen, a new synthetic data
+generator that can produce accurate and rich ground-truth training data for
+egocentric perception tasks. At the heart of EgoGen is a novel human motion
+synthesis model that directly leverages egocentric visual inputs of a virtual
+human to sense the 3D environment. Combined with collision-avoiding motion
+primitives and a two-stage reinforcement learning approach, our motion
+synthesis model offers a closed-loop solution where the embodied perception and
+movement of the virtual human are seamlessly coupled. Compared to previous
+works, our model eliminates the need for a pre-defined global path, and is
+directly applicable to dynamic environments. Combined with our easy-to-use and
+scalable data generation pipeline, we demonstrate EgoGen's efficacy in three
+tasks: mapping and localization for head-mounted cameras, egocentric camera
+tracking, and human mesh recovery from egocentric views. EgoGen will be fully
+open-sourced, offering a practical solution for creating realistic egocentric
+training data and aiming to serve as a useful tool for egocentric computer
+vision research. Refer to our project page: https://ego-gen.github.io/.",cs.CV,"['cs.CV', 'cs.AI']"
+D3T: Distinctive Dual-Domain Teacher Zigzagging Across RGB-Thermal Gap for Domain-Adaptive Object Detection,Dinh Phat Do · Taehoon Kim · JAEMIN NA · Jiwon Kim · Keonho LEE · Kyunghwan Cho · Wonjun Hwang,https://github.com/EdwardDo69/D3T,https://arxiv.org/abs/2403.09359,,2403.09359.pdf,D3T: Distinctive Dual-Domain Teacher Zigzagging Across RGB-Thermal Gap for Domain-Adaptive Object Detection,"Domain adaptation for object detection typically entails transferring
+knowledge from one visible domain to another visible domain. However, there are
+limited studies on adapting from the visible to the thermal domain, because the
+domain gap between the visible and thermal domains is much larger than
+expected, and traditional domain adaptation can not successfully facilitate
+learning in this situation. To overcome this challenge, we propose a
+Distinctive Dual-Domain Teacher (D3T) framework that employs distinct training
+paradigms for each domain. Specifically, we segregate the source and target
+training sets for building dual-teachers and successively deploy exponential
+moving average to the student model to individual teachers of each domain. The
+framework further incorporates a zigzag learning method between dual teachers,
+facilitating a gradual transition from the visible to thermal domains during
+training. We validate the superiority of our method through newly designed
+experimental protocols with well-known thermal datasets, i.e., FLIR and KAIST.
+Source code is available at https://github.com/EdwardDo69/D3T .",cs.CV,"['cs.CV', 'cs.AI']"
+Bayesian Diffusion Models for 3D Shape Reconstruction,Haiyang Xu · Yu lei · Zeyuan Chen · Xiang Zhang · Yue Zhao · Yilin Wang · Zhuowen Tu, ,https://arxiv.org/abs/2403.06973,,2403.06973.pdf,Bayesian Diffusion Models for 3D Shape Reconstruction,"We present Bayesian Diffusion Models (BDM), a prediction algorithm that
+performs effective Bayesian inference by tightly coupling the top-down (prior)
+information with the bottom-up (data-driven) procedure via joint diffusion
+processes. We show the effectiveness of BDM on the 3D shape reconstruction
+task. Compared to prototypical deep learning data-driven approaches trained on
+paired (supervised) data-labels (e.g. image-point clouds) datasets, our BDM
+brings in rich prior information from standalone labels (e.g. point clouds) to
+improve the bottom-up 3D reconstruction. As opposed to the standard Bayesian
+frameworks where explicit prior and likelihood are required for the inference,
+BDM performs seamless information fusion via coupled diffusion processes with
+learned gradient computation networks. The specialty of our BDM lies in its
+capability to engage the active and effective information exchange and fusion
+of the top-down and bottom-up processes where each itself is a diffusion
+process. We demonstrate state-of-the-art results on both synthetic and
+real-world benchmarks for 3D shape reconstruction.",cs.CV,"['cs.CV', 'cs.LG']"
+Domain Separation Graph Neural Networks for Saliency Object Ranking,Zijian Wu · Jun Lu · Jing Han · Lianfa Bai · Yi Zhang · Zhuang Zhao · Siyang Song, ,,https://www.nature.com/articles/s41598-024-61105-3,,,,,nan
+DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision,Lu Ling · Yichen Sheng · Zhi Tu · Wentian Zhao · Cheng Xin · Kun Wan · Lantao Yu · Qianyu Guo · Zixun Yu · Yawen Lu · Xuanmao Li · Xingpeng Sun · Rohan Ashok · Aniruddha Mukherjee · Hao Kang · Xiangrui Kong · Gang Hua · Tianyi Zhang · Bedrich Benes · Aniket Bera, ,https://arxiv.org/abs/2312.16256,,2312.16256.pdf,DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision,"We have witnessed significant progress in deep learning-based 3D vision,
+ranging from neural radiance field (NeRF) based 3D representation learning to
+applications in novel view synthesis (NVS). However, existing scene-level
+datasets for deep learning-based 3D vision, limited to either synthetic
+environments or a narrow selection of real-world scenes, are quite
+insufficient. This insufficiency not only hinders a comprehensive benchmark of
+existing methods but also caps what could be explored in deep learning-based 3D
+analysis. To address this critical gap, we present DL3DV-10K, a large-scale
+scene dataset, featuring 51.2 million frames from 10,510 videos captured from
+65 types of point-of-interest (POI) locations, covering both bounded and
+unbounded scenes, with different levels of reflection, transparency, and
+lighting. We conducted a comprehensive benchmark of recent NVS methods on
+DL3DV-10K, which revealed valuable insights for future research in NVS. In
+addition, we have obtained encouraging results in a pilot study to learn
+generalizable NeRF from DL3DV-10K, which manifests the necessity of a
+large-scale scene-level dataset to forge a path toward a foundation model for
+learning 3D representation. Our DL3DV-10K dataset, benchmark results, and
+models will be publicly accessible at https://dl3dv-10k.github.io/DL3DV-10K/.",cs.CV,"['cs.CV', 'cs.AI']"
+Fitting Flats to Flats,Gabriel Dogadov · Ugo Finnendahl · Marc Alexa, ,,https://github.com/gdogadov,,,,,nan
+MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild,Zeren Jiang · Chen Guo · Manuel Kaufmann · Tianjian Jiang · Julien Valentin · Otmar Hilliges · Jie Song, ,,https://dl.acm.org/doi/10.1145/3581783.3611978,,,,,nan
+Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving,Yuqi Wang · Jiawei He · Lue Fan · Hongxin Li · Yuntao Chen · Zhaoxiang Zhang,https://drive-wm.github.io,https://arxiv.org/abs/2311.17918,,2311.17918.pdf,Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving,"In autonomous driving, predicting future events in advance and evaluating the
+foreseeable risks empowers autonomous vehicles to better plan their actions,
+enhancing safety and efficiency on the road. To this end, we propose Drive-WM,
+the first driving world model compatible with existing end-to-end planning
+models. Through a joint spatial-temporal modeling facilitated by view
+factorization, our model generates high-fidelity multiview videos in driving
+scenes. Building on its powerful generation ability, we showcase the potential
+of applying the world model for safe driving planning for the first time.
+Particularly, our Drive-WM enables driving into multiple futures based on
+distinct driving maneuvers, and determines the optimal trajectory according to
+the image-based rewards. Evaluation on real-world driving datasets verifies
+that our method could generate high-quality, consistent, and controllable
+multiview videos, opening up possibilities for real-world simulations and safe
+planning.",cs.CV,['cs.CV']
+DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving,Chen Min · Dawei Zhao · Liang Xiao · Jian Zhao · Xinli Xu · Zheng Zhu · Lei Jin · Jianshu Li · Yulan Guo · Junliang Xing · Liping Jing · Yiming Nie · Bin Dai, ,,https://paperswithcode.com/paper/driveworld-4d-pre-trained-scene-understanding,,,,,nan
+ZONE: Zero-Shot Instruction-Guided Local Editing,Shanglin Li · Bohan Zeng · Yutang Feng · Sicheng Gao · Xuhui Liu · Jiaming Liu · Li Lin · Xu Tang · Yao Hu · Jianzhuang Liu · Baochang Zhang, ,https://arxiv.org/abs/2312.16794,,2312.16794.pdf,ZONE: Zero-Shot Instruction-Guided Local Editing,"Recent advances in vision-language models like Stable Diffusion have shown
+remarkable power in creative image synthesis and editing.However, most existing
+text-to-image editing methods encounter two obstacles: First, the text prompt
+needs to be carefully crafted to achieve good results, which is not intuitive
+or user-friendly. Second, they are insensitive to local edits and can
+irreversibly affect non-edited regions, leaving obvious editing traces. To
+tackle these problems, we propose a Zero-shot instructiON-guided local image
+Editing approach, termed ZONE. We first convert the editing intent from the
+user-provided instruction (e.g., ""make his tie blue"") into specific image
+editing regions through InstructPix2Pix. We then propose a Region-IoU scheme
+for precise image layer extraction from an off-the-shelf segment model. We
+further develop an edge smoother based on FFT for seamless blending between the
+layer and the image.Our method allows for arbitrary manipulation of a specific
+region with a single instruction while preserving the rest. Extensive
+experiments demonstrate that our ZONE achieves remarkable local editing results
+and user-friendliness, outperforming state-of-the-art methods. Code is
+available at https://github.com/lsl001006/ZONE.",cs.CV,['cs.CV']
+DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing,Kaiwen Zhang · Yifan Zhou · Xudong XU · Bo Dai · Xingang Pan,https://kevin-thu.github.io/DiffMorpher_page,https://arxiv.org/abs/2312.07409,,2312.07409.pdf,DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing,"Diffusion models have achieved remarkable image generation quality surpassing
+previous generative models. However, a notable limitation of diffusion models,
+in comparison to GANs, is their difficulty in smoothly interpolating between
+two image samples, due to their highly unstructured latent space. Such a smooth
+interpolation is intriguing as it naturally serves as a solution for the image
+morphing task with many applications. In this work, we present DiffMorpher, the
+first approach enabling smooth and natural image interpolation using diffusion
+models. Our key idea is to capture the semantics of the two images by fitting
+two LoRAs to them respectively, and interpolate between both the LoRA
+parameters and the latent noises to ensure a smooth semantic transition, where
+correspondence automatically emerges without the need for annotation. In
+addition, we propose an attention interpolation and injection technique and a
+new sampling schedule to further enhance the smoothness between consecutive
+images. Extensive experiments demonstrate that DiffMorpher achieves starkly
+better image morphing effects than previous methods across a variety of object
+categories, bridging a critical functional gap that distinguished diffusion
+models from GANs.",cs.CV,['cs.CV']
+InstructDiffusion: A Generalist Modeling Interface for Vision Tasks,Zigang Geng · Binxin Yang · Tiankai Hang · Chen Li · Shuyang Gu · Ting Zhang · Jianmin Bao · Zheng Zhang · Houqiang Li · Han Hu · Dong Chen · Baining Guo, ,https://arxiv.org/abs/2309.03895,,2309.03895.pdf,InstructDiffusion: A Generalist Modeling Interface for Vision Tasks,"We present InstructDiffusion, a unifying and generic framework for aligning
+computer vision tasks with human instructions. Unlike existing approaches that
+integrate prior knowledge and pre-define the output space (e.g., categories and
+coordinates) for each vision task, we cast diverse vision tasks into a
+human-intuitive image-manipulating process whose output space is a flexible and
+interactive pixel space. Concretely, the model is built upon the diffusion
+process and is trained to predict pixels according to user instructions, such
+as encircling the man's left shoulder in red or applying a blue mask to the
+left car. InstructDiffusion could handle a variety of vision tasks, including
+understanding tasks (such as segmentation and keypoint detection) and
+generative tasks (such as editing and enhancement). It even exhibits the
+ability to handle unseen tasks and outperforms prior methods on novel datasets.
+This represents a significant step towards a generalist modeling interface for
+vision tasks, advancing artificial general intelligence in the field of
+computer vision.",cs.CV,['cs.CV']
+Loose Inertial Poser: Motion Capture with IMU-attached Loose-Wear Jacket,Chengxu Zuo · Yiming Wang · Lishuang Zhan · Shihui Guo · Xinyu Yi · Feng Xu · Yipeng Qin, ,https://arxiv.org/abs/2308.16682,,2308.16682.pdf,DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion,"Motion capture from a limited number of body-worn sensors, such as inertial
+measurement units (IMUs) and pressure insoles, has important applications in
+health, human performance, and entertainment. Recent work has focused on
+accurately reconstructing whole-body motion from a specific sensor
+configuration using six IMUs. While a common goal across applications is to use
+the minimal number of sensors to achieve required accuracy, the optimal
+arrangement of the sensors might differ from application to application. We
+propose a single diffusion model, DiffusionPoser, which reconstructs human
+motion in real-time from an arbitrary combination of sensors, including IMUs
+placed at specified locations, and, pressure insoles. Unlike existing methods,
+our model grants users the flexibility to determine the number and arrangement
+of sensors tailored to the specific activity of interest, without the need for
+retraining. A novel autoregressive inferencing scheme ensures real-time motion
+reconstruction that closely aligns with measured sensor signals. The generative
+nature of DiffusionPoser ensures realistic behavior, even for
+degrees-of-freedom not directly measured. Qualitative results can be found on
+our website: https://diffusionposer.github.io/.",cs.CV,['cs.CV']
+Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology,Andrew Song · Richard J. Chen · Tong Ding · Drew F. K. Williamson · Guillaume Jaume · Faisal Mahmood, ,https://arxiv.org/abs/2405.11643,,2405.11643.pdf,Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology,"Representation learning of pathology whole-slide images (WSIs) has been has
+primarily relied on weak supervision with Multiple Instance Learning (MIL).
+However, the slide representations resulting from this approach are highly
+tailored to specific clinical tasks, which limits their expressivity and
+generalization, particularly in scenarios with limited data. Instead, we
+hypothesize that morphological redundancy in tissue can be leveraged to build a
+task-agnostic slide representation in an unsupervised fashion. To this end, we
+introduce PANTHER, a prototype-based approach rooted in the Gaussian mixture
+model that summarizes the set of WSI patches into a much smaller set of
+morphological prototypes. Specifically, each patch is assumed to have been
+generated from a mixture distribution, where each mixture component represents
+a morphological exemplar. Utilizing the estimated mixture parameters, we then
+construct a compact slide representation that can be readily used for a wide
+range of downstream tasks. By performing an extensive evaluation of PANTHER on
+subtyping and survival tasks using 13 datasets, we show that 1) PANTHER
+outperforms or is on par with supervised MIL baselines and 2) the analysis of
+morphological prototypes brings new qualitative and quantitative insights into
+model interpretability.",cs.CV,"['cs.CV', 'cs.LG', 'stat.AP']"
+FairRAG: Fair Human Generation via Fair Retrieval Augmentation,Robik Shrestha · Yang Zou · Qiuyu Chen · Zhiheng Li · Yusheng Xie · Siqi Deng, ,https://arxiv.org/abs/2403.19964,,2403.19964.pdf,FairRAG: Fair Human Generation via Fair Retrieval Augmentation,"Existing text-to-image generative models reflect or even amplify societal
+biases ingrained in their training data. This is especially concerning for
+human image generation where models are biased against certain demographic
+groups. Existing attempts to rectify this issue are hindered by the inherent
+limitations of the pre-trained models and fail to substantially improve
+demographic diversity. In this work, we introduce Fair Retrieval Augmented
+Generation (FairRAG), a novel framework that conditions pre-trained generative
+models on reference images retrieved from an external image database to improve
+fairness in human generation. FairRAG enables conditioning through a
+lightweight linear module that projects reference images into the textual
+space. To enhance fairness, FairRAG applies simple-yet-effective debiasing
+strategies, providing images from diverse demographic groups during the
+generative process. Extensive experiments demonstrate that FairRAG outperforms
+existing methods in terms of demographic diversity, image-text alignment, and
+image fidelity while incurring minimal computational overhead during inference.",cs.CV,"['cs.CV', 'cs.CY', 'cs.LG']"
+Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction,Guillaume Jaume · Anurag Vaidya · Richard J. Chen · Drew F. K. Williamson · Paul Pu Liang · Faisal Mahmood, ,https://arxiv.org/abs/2404.08027,,2404.08027.pdf,SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction,"Multi-modal learning that combines pathological images with genomic data has
+significantly enhanced the accuracy of survival prediction. Nevertheless,
+existing methods have not fully utilized the inherent hierarchical structure
+within both whole slide images (WSIs) and transcriptomic data, from which
+better intra-modal representations and inter-modal integration could be
+derived. Moreover, many existing studies attempt to improve multi-modal
+representations through attention mechanisms, which inevitably lead to high
+complexity when processing high-dimensional WSIs and transcriptomic data.
+Recently, a structured state space model named Mamba emerged as a promising
+approach for its superior performance in modeling long sequences with low
+complexity. In this study, we propose Mamba with multi-grained multi-modal
+interaction (SurvMamba) for survival prediction. SurvMamba is implemented with
+a Hierarchical Interaction Mamba (HIM) module that facilitates efficient
+intra-modal interactions at different granularities, thereby capturing more
+detailed local features as well as rich global representations. In addition, an
+Interaction Fusion Mamba (IFM) module is used for cascaded inter-modal
+interactive fusion, yielding more comprehensive features for survival
+prediction. Comprehensive evaluations on five TCGA datasets demonstrate that
+SurvMamba outperforms other existing methods in terms of performance and
+computational cost.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'q-bio.QM']"
+End-to-End Spatio-Temporal Action Localisation with Video Transformers,Alexey Gritsenko · Xuehan Xiong · Josip Djolonga · Mostafa Dehghani · Chen Sun · Mario Lučić · Cordelia Schmid · Anurag Arnab, ,,https://openreview.net/forum?id=Va4t6R8cGG,,,,,nan
+MCNet: Rethinking the Core Ingredients for Accurate and Efficient Homography Estimation,Haokai Zhu · Si-Yuan Cao · Jianxin Hu · Sitong Zuo · Beinan Yu · Jiacheng Ying · Junwei Li · Hui-Liang Shen,https://github.com/zjuzhk/MCNet,,https://www.youtube.com/watch?v=mcRa7BsZrOE,,,,,nan
+SODA: Bottleneck Diffusion Models for Representation Learning,Drew Hudson · Daniel Zoran · Mateusz Malinowski · Andrew Lampinen · Andrew Jaegle · James McClelland · Loic Matthey · Felix Hill · Alexander Lerchner, ,https://arxiv.org/abs/2311.17901,,2311.17901.pdf,SODA: Bottleneck Diffusion Models for Representation Learning,"We introduce SODA, a self-supervised diffusion model, designed for
+representation learning. The model incorporates an image encoder, which
+distills a source view into a compact representation, that, in turn, guides the
+generation of related novel views. We show that by imposing a tight bottleneck
+between the encoder and a denoising decoder, and leveraging novel view
+synthesis as a self-supervised objective, we can turn diffusion models into
+strong representation learners, capable of capturing visual semantics in an
+unsupervised manner. To the best of our knowledge, SODA is the first diffusion
+model to succeed at ImageNet linear-probe classification, and, at the same
+time, it accomplishes reconstruction, editing and synthesis tasks across a wide
+range of datasets. Further investigation reveals the disentangled nature of its
+emergent latent space, that serves as an effective interface to control and
+manipulate the model's produced images. All in all, we aim to shed light on the
+exciting and promising potential of diffusion models, not only for image
+generation, but also for learning rich and robust representations.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+EasyDrag: Efficient Point-based Manipulation on Diffusion Models,Xingzhong Hou · Boxiao Liu · Yi Zhang · Jihao Liu · Yu Liu · Haihang You, ,,https://github.com/Yujun-Shi/DragDiffusion,,,,,nan
+Segment and Caption Anything,Xiaoke Huang · Jianfeng Wang · Yansong Tang · Zheng Zhang · Han Hu · Jiwen Lu · Lijuan Wang · Zicheng Liu,https://xk-huang.github.io/segment-caption-anything/,https://arxiv.org/abs/2312.00869,,2312.00869.pdf,Segment and Caption Anything,"We propose a method to efficiently equip the Segment Anything Model (SAM)
+with the ability to generate regional captions. SAM presents strong
+generalizability to segment anything while is short for semantic understanding.
+By introducing a lightweight query-based feature mixer, we align the
+region-specific features with the embedding space of language models for later
+caption generation. As the number of trainable parameters is small (typically
+in the order of tens of millions), it costs less computation, less memory
+usage, and less communication bandwidth, resulting in both fast and scalable
+training. To address the scarcity problem of regional caption data, we propose
+to first pre-train our model on objection detection and segmentation tasks. We
+call this step weak supervision pretraining since the pre-training data only
+contains category names instead of full-sentence descriptions. The weak
+supervision pretraining allows us to leverage many publicly available object
+detection and segmentation datasets. We conduct extensive experiments to
+demonstrate the superiority of our method and validate each design choice. This
+work serves as a stepping stone towards scaling up regional captioning data and
+sheds light on exploring efficient ways to augment SAM with regional semantics.
+The project page, along with the associated code, can be accessed via
+https://xk-huang.github.io/segment-caption-anything/.",cs.CV,['cs.CV']
+6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation,Li Xu · Haoxuan Qu · Yujun Cai · Jun Liu, ,https://arxiv.org/abs/2401.00029,,2401.00029.pdf,6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation,"Estimating the 6D object pose from a single RGB image often involves noise
+and indeterminacy due to challenges such as occlusions and cluttered
+backgrounds. Meanwhile, diffusion models have shown appealing performance in
+generating high-quality images from random noise with high indeterminacy
+through step-by-step denoising. Inspired by their denoising capability, we
+propose a novel diffusion-based framework (6D-Diff) to handle the noise and
+indeterminacy in object pose estimation for better performance. In our
+framework, to establish accurate 2D-3D correspondence, we formulate 2D
+keypoints detection as a reverse diffusion (denoising) process. To facilitate
+such a denoising process, we design a Mixture-of-Cauchy-based forward diffusion
+process and condition the reverse process on the object features. Extensive
+experiments on the LM-O and YCB-V datasets demonstrate the effectiveness of our
+framework.",cs.CV,['cs.CV']
+UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes,David Rozenberszki · Or Litany · Angela Dai,https://rozdavid.github.io/unscene3d,https://ar5iv.labs.arxiv.org/html/2312.11557,,2312.11557.pdf,SAI3D: Segment Any Instance in 3D Scenes,"Advancements in 3D instance segmentation have traditionally been tethered to
+the availability of annotated datasets, limiting their application to a narrow
+spectrum of object categories. Recent efforts have sought to harness
+vision-language models like CLIP for open-set semantic reasoning, yet these
+methods struggle to distinguish between objects of the same categories and rely
+on specific prompts that are not universally applicable. In this paper, we
+introduce SAI3D, a novel zero-shot 3D instance segmentation approach that
+synergistically leverages geometric priors and semantic cues derived from
+Segment Anything Model (SAM). Our method partitions a 3D scene into geometric
+primitives, which are then progressively merged into 3D instance segmentations
+that are consistent with the multi-view SAM masks. Moreover, we design a
+hierarchical region-growing algorithm with a dynamic thresholding mechanism,
+which largely improves the robustness of finegrained 3D scene parsing.Empirical
+evaluations on ScanNet, Matterport3D and the more challenging ScanNet++
+datasets demonstrate the superiority of our approach. Notably, SAI3D
+outperforms existing open-vocabulary baselines and even surpasses
+fully-supervised methods in class-agnostic segmentation on ScanNet++. Our
+project page is at https://yd-yin.github.io/SAI3D.",cs.CV,['cs.CV']
+Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation,Yi Zhang · Meng-Hao Guo · Miao Wang · Shi-Min Hu, ,https://arxiv.org/abs/2403.08426,,2403.08426.pdf,Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation,"The pre-trained vision-language model, exemplified by CLIP, advances
+zero-shot semantic segmentation by aligning visual features with class
+embeddings through a transformer decoder to generate semantic masks. Despite
+its effectiveness, prevailing methods within this paradigm encounter
+challenges, including overfitting on seen classes and small fragmentation in
+masks. To mitigate these issues, we propose a Language-Driven Visual Consensus
+(LDVC) approach, fostering improved alignment of semantic and visual
+information.Specifically, we leverage class embeddings as anchors due to their
+discrete and abstract nature, steering vision features toward class embeddings.
+Moreover, to circumvent noisy alignments from the vision part due to its
+redundant nature, we introduce route attention into self-attention for finding
+visual consensus, thereby enhancing semantic consistency within the same
+object. Equipped with a vision-language prompting strategy, our approach
+significantly boosts the generalization capacity of segmentation models for
+unseen classes. Experimental results underscore the effectiveness of our
+approach, showcasing mIoU gains of 4.5 on the PASCAL VOC 2012 and 3.6 on the
+COCO-Stuff 164k for unseen classes compared with the state-of-the-art methods.",cs.CV,"['cs.CV', 'cs.AI']"
+Selective nonlinearities removal from digital signals,Krzysztof Maliszewski · Magdalena Urbanska · Varvara Vetrova · Sylwia Kolenderska, ,https://arxiv.org/abs/2403.09731,,2403.09731.pdf,Selective nonlinearities removal from digital signals,"Many instruments performing optical and non-optical imaging and sensing, such
+as Optical Coherence Tomography (OCT), Magnetic Resonance Imaging or
+Fourier-transform spectrometry, produce digital signals containing modulations,
+sine-like components, which only after Fourier transformation give information
+about the structure or characteristics of the investigated object. Due to the
+fundamental physics-related limitations of such methods, the distribution of
+these signal components is often nonlinear and, when not properly compensated,
+leads to the resolution, precision or quality drop in the final image. Here, we
+propose an innovative approach that has the potential to allow cleaning of the
+signal from the nonlinearities but most of all, it now allows to switch the
+given order off, leaving all others intact. The latter provides a tool for more
+in-depth analysis of the nonlinearity-inducing properties of the investigated
+object, which can lead to applications in early disease detection or more
+sensitive sensing of chemical compounds. We consider OCT signals and
+nonlinearities up to the third order. In our approach, we propose two neural
+networks: one to remove solely the second-order nonlinearity and the other for
+removing solely the third-order nonlinearity. The input of the networks is a
+novel two-dimensional data structure with all the information needed for the
+network to infer a nonlinearity-free signal. We describe the developed networks
+and present the results for second-order and third-order nonlinearity removal
+in OCT data representing the images of various objects: a mirror, glass, and
+fruits.",eess.IV,"['eess.IV', 'physics.data-an', 'physics.optics']"
+Efficient Model Stealing Defense with Noise Transition Matrix,Dong-Dong Wu · Chilin Fu · Weichang Wu · Wenwen Xia · Xiaolu Zhang · JUN ZHOU · Min-Ling Zhang, ,https://arxiv.org/abs/2309.01838,,2309.01838.pdf,Efficient Defense Against Model Stealing Attacks on Convolutional Neural Networks,"Model stealing attacks have become a serious concern for deep learning
+models, where an attacker can steal a trained model by querying its black-box
+API. This can lead to intellectual property theft and other security and
+privacy risks. The current state-of-the-art defenses against model stealing
+attacks suggest adding perturbations to the prediction probabilities. However,
+they suffer from heavy computations and make impracticable assumptions about
+the adversary. They often require the training of auxiliary models. This can be
+time-consuming and resource-intensive which hinders the deployment of these
+defenses in real-world applications. In this paper, we propose a simple yet
+effective and efficient defense alternative. We introduce a heuristic approach
+to perturb the output probabilities. The proposed defense can be easily
+integrated into models without additional training. We show that our defense is
+effective in defending against three state-of-the-art stealing attacks. We
+evaluate our approach on large and quantized (i.e., compressed) Convolutional
+Neural Networks (CNNs) trained on several vision datasets. Our technique
+outperforms the state-of-the-art defenses with a $\times37$ faster inference
+latency without requiring any additional model and with a low impact on the
+model's performance. We validate that our defense is also effective for
+quantized CNNs targeting edge devices.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CR']"
+Unsupervised Universal Image Segmentation,XuDong Wang · Dantong Niu · Xinyang Han · Long Lian · Roei Herzig · Trevor Darrell, ,https://arxiv.org/abs/2312.17243,,2312.17243.pdf,Unsupervised Universal Image Segmentation,"Several unsupervised image segmentation approaches have been proposed which
+eliminate the need for dense manually-annotated segmentation masks; current
+models separately handle either semantic segmentation (e.g., STEGO) or
+class-agnostic instance segmentation (e.g., CutLER), but not both (i.e.,
+panoptic segmentation). We propose an Unsupervised Universal Segmentation model
+(U2Seg) adept at performing various image segmentation tasks -- instance,
+semantic and panoptic -- using a novel unified framework. U2Seg generates
+pseudo semantic labels for these segmentation tasks via leveraging
+self-supervised models followed by clustering; each cluster represents
+different semantic and/or instance membership of pixels. We then self-train the
+model on these pseudo semantic labels, yielding substantial performance gains
+over specialized methods tailored to each task: a +2.6 AP$^{\text{box}}$ boost
+vs. CutLER in unsupervised instance segmentation on COCO and a +7.0 PixelAcc
+increase (vs. STEGO) in unsupervised semantic segmentation on COCOStuff.
+Moreover, our method sets up a new baseline for unsupervised panoptic
+segmentation, which has not been previously explored. U2Seg is also a strong
+pretrained model for few-shot segmentation, surpassing CutLER by +5.0
+AP$^{\text{mask}}$ when trained on a low-data regime, e.g., only 1% COCO
+labels. We hope our simple yet effective method can inspire more research on
+unsupervised universal image segmentation.",cs.CV,['cs.CV']
+HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation,Xin Huang · Ruizhi Shao · Qi Zhang · Hongwen Zhang · Ying Feng · Yebin Liu · Qing Wang,https://humannorm.github.io,https://arxiv.org/abs/2310.01406,,2310.01406.pdf,HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation,"Recent text-to-3D methods employing diffusion models have made significant
+advancements in 3D human generation. However, these approaches face challenges
+due to the limitations of text-to-image diffusion models, which lack an
+understanding of 3D structures. Consequently, these methods struggle to achieve
+high-quality human generation, resulting in smooth geometry and cartoon-like
+appearances. In this paper, we propose HumanNorm, a novel approach for
+high-quality and realistic 3D human generation. The main idea is to enhance the
+model's 2D perception of 3D geometry by learning a normal-adapted diffusion
+model and a normal-aligned diffusion model. The normal-adapted diffusion model
+can generate high-fidelity normal maps corresponding to user prompts with
+view-dependent and body-aware text. The normal-aligned diffusion model learns
+to generate color images aligned with the normal maps, thereby transforming
+physical geometry details into realistic appearance. Leveraging the proposed
+normal diffusion model, we devise a progressive geometry generation strategy
+and a multi-step Score Distillation Sampling (SDS) loss to enhance the
+performance of 3D human generation. Comprehensive experiments substantiate
+HumanNorm's ability to generate 3D humans with intricate geometry and realistic
+appearances. HumanNorm outperforms existing text-to-3D methods in both geometry
+and texture quality. The project page of HumanNorm is
+https://humannorm.github.io/.",cs.CV,['cs.CV']
+SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation,Kejia Yin · Varshanth Rao · Ruowei Jiang · Xudong Liu · Parham Aarabi · David B. Lindell, ,https://arxiv.org/abs/2405.18322,,2405.18322.pdf,SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation,"Self-supervised landmark estimation is a challenging task that demands the
+formation of locally distinct feature representations to identify sparse facial
+landmarks in the absence of annotated data. To tackle this task, existing
+state-of-the-art (SOTA) methods (1) extract coarse features from backbones that
+are trained with instance-level self-supervised learning (SSL) paradigms, which
+neglect the dense prediction nature of the task, (2) aggregate them into
+memory-intensive hypercolumn formations, and (3) supervise lightweight
+projector networks to naively establish full local correspondences among all
+pairs of spatial features. In this paper, we introduce SCE-MAE, a framework
+that (1) leverages the MAE, a region-level SSL method that naturally better
+suits the landmark prediction task, (2) operates on the vanilla feature map
+instead of on expensive hypercolumns, and (3) employs a Correspondence
+Approximation and Refinement Block (CARB) that utilizes a simple density peak
+clustering algorithm and our proposed Locality-Constrained Repellence Loss to
+directly hone only select local correspondences. We demonstrate through
+extensive experiments that SCE-MAE is highly effective and robust,
+outperforming existing SOTA methods by large margins of approximately 20%-44%
+on the landmark matching and approximately 9%-15% on the landmark detection
+tasks.",cs.CV,"['cs.CV', 'cs.AI']"
+Beyond Textual Constraints: Learning Novel Diffusion Conditions with Fewer Examples,Yuyang Yu · Bangzhen Liu · Chenxi Zheng · Xuemiao Xu · Huaidong Zhang · Shengfeng He,https://github.com/Yuyan9Yu/BeyondTextConstraint,https://arxiv.org/abs/2307.16424,,2307.16424.pdf,MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning,"Equipping a deep model the abaility of few-shot learning, i.e., learning
+quickly from only few examples, is a core challenge for artificial
+intelligence. Gradient-based meta-learning approaches effectively address the
+challenge by learning how to learn novel tasks. Its key idea is learning a deep
+model in a bi-level optimization manner, where the outer-loop process learns a
+shared gradient descent algorithm (i.e., its hyperparameters), while the
+inner-loop process leverage it to optimize a task-specific model by using only
+few labeled data. Although these existing methods have shown superior
+performance, the outer-loop process requires calculating second-order
+derivatives along the inner optimization path, which imposes considerable
+memory burdens and the risk of vanishing gradients. Drawing inspiration from
+recent progress of diffusion models, we find that the inner-loop gradient
+descent process can be actually viewed as a reverse process (i.e., denoising)
+of diffusion where the target of denoising is model weights but the origin
+data. Based on this fact, in this paper, we propose to model the gradient
+descent optimizer as a diffusion model and then present a novel
+task-conditional diffusion-based meta-learning, called MetaDiff, that
+effectively models the optimization process of model weights from Gaussion
+noises to target weights in a denoising manner. Thanks to the training
+efficiency of diffusion models, our MetaDiff do not need to differentiate
+through the inner-loop path such that the memory burdens and the risk of
+vanishing gradients can be effectvely alleviated. Experiment results show that
+our MetaDiff outperforms the state-of-the-art gradient-based meta-learning
+family in few-shot learning tasks.",cs.LG,['cs.LG']
+Joint2Human: High-quality 3D Human Generation via Compact Spherical Embedding of 3D Joints,Muxin Zhang · Qiao Feng · Zhuo Su · Chao Wen · Zhou Xue · Kun Li, ,https://arxiv.org/abs/2312.08591,,2312.08591.pdf,Joint2Human: High-quality 3D Human Generation via Compact Spherical Embedding of 3D Joints,"3D human generation is increasingly significant in various applications.
+However, the direct use of 2D generative methods in 3D generation often results
+in losing local details, while methods that reconstruct geometry from generated
+images struggle with global view consistency. In this work, we introduce
+Joint2Human, a novel method that leverages 2D diffusion models to generate
+detailed 3D human geometry directly, ensuring both global structure and local
+details. To achieve this, we employ the Fourier occupancy field (FOF)
+representation, enabling the direct generation of 3D shapes as preliminary
+results with 2D generative models. With the proposed high-frequency enhancer
+and the multi-view recarving strategy, our method can seamlessly integrate the
+details from different views into a uniform global shape. To better utilize the
+3D human prior and enhance control over the generated geometry, we introduce a
+compact spherical embedding of 3D joints. This allows for an effective guidance
+of pose during the generation process. Additionally, our method can generate 3D
+humans guided by textual inputs. Our experimental results demonstrate the
+capability of our method to ensure global structure, local details, high
+resolution, and low computational cost simultaneously. More results and the
+code can be found on our project page at
+http://cic.tju.edu.cn/faculty/likun/projects/Joint2Human.",cs.CV,['cs.CV']
+Rethinking Generalizable Face Anti-spoofing via Hierarchical Prototype-guided Distribution Refinement in Hyperbolic Space,Chengyang Hu · Ke-Yue Zhang · Taiping Yao · Shouhong Ding · Lizhuang Ma, ,https://arxiv.org/abs/2308.09107,,2308.09107.pdf,Hyperbolic Face Anti-Spoofing,"Learning generalized face anti-spoofing (FAS) models against presentation
+attacks is essential for the security of face recognition systems. Previous FAS
+methods usually encourage models to extract discriminative features, of which
+the distances within the same class (bonafide or attack) are pushed close while
+those between bonafide and attack are pulled away. However, these methods are
+designed based on Euclidean distance, which lacks generalization ability for
+unseen attack detection due to poor hierarchy embedding ability. According to
+the evidence that different spoofing attacks are intrinsically hierarchical, we
+propose to learn richer hierarchical and discriminative spoofing cues in
+hyperbolic space. Specifically, for unimodal FAS learning, the feature
+embeddings are projected into the Poincar\'e ball, and then the hyperbolic
+binary logistic regression layer is cascaded for classification. To further
+improve generalization, we conduct hyperbolic contrastive learning for the
+bonafide only while relaxing the constraints on diverse spoofing attacks. To
+alleviate the vanishing gradient problem in hyperbolic space, a new feature
+clipping method is proposed to enhance the training stability of hyperbolic
+models. Besides, we further design a multimodal FAS framework with Euclidean
+multimodal feature decomposition and hyperbolic multimodal feature fusion &
+classification. Extensive experiments on three benchmark datasets (i.e., WMCA,
+PADISI-Face, and SiW-M) with diverse attack types demonstrate that the proposed
+method can bring significant improvement compared to the Euclidean baselines on
+unseen attack detection. In addition, the proposed framework is also
+generalized well on four benchmark datasets (i.e., MSU-MFSD, IDIAP
+REPLAY-ATTACK, CASIA-FASD, and OULU-NPU) with a limited number of attack types.",cs.CV,['cs.CV']
+NARUTO: Neural Active Reconstruction from Uncertain Target Observations,Ziyue Feng · Huangying Zhan · Zheng Chen · Qingan Yan · Xiangyu Xu · Changjiang Cai · Bing Li · Qilun Zhu · Yi Xu,https://oppo-us-research.github.io/NARUTO-website/,https://arxiv.org/abs/2402.18771v2,,2402.18771v2.pdf,NARUTO: Neural Active Reconstruction from Uncertain Target Observations,"We present NARUTO, a neural active reconstruction system that combines a
+hybrid neural representation with uncertainty learning, enabling high-fidelity
+surface reconstruction. Our approach leverages a multi-resolution hash-grid as
+the mapping backbone, chosen for its exceptional convergence speed and capacity
+to capture high-frequency local features.The centerpiece of our work is the
+incorporation of an uncertainty learning module that dynamically quantifies
+reconstruction uncertainty while actively reconstructing the environment. By
+harnessing learned uncertainty, we propose a novel uncertainty aggregation
+strategy for goal searching and efficient path planning. Our system
+autonomously explores by targeting uncertain observations and reconstructs
+environments with remarkable completeness and fidelity. We also demonstrate the
+utility of this uncertainty-aware approach by enhancing SOTA neural SLAM
+systems through an active ray sampling strategy. Extensive evaluations of
+NARUTO in various environments, using an indoor scene simulator, confirm its
+superior performance and state-of-the-art status in active reconstruction, as
+evidenced by its impressive results on benchmark datasets like Replica and
+MP3D.",cs.CV,"['cs.CV', 'cs.RO']"
+CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning,Shiyu Tian · Hongxin Wei · Yiqun Wang · Lei Feng, ,,https://dblp.org/rec/journals/corr/abs-2303-10365,,,,,nan
+Generative Proxemics: A Prior for 3D Social Interaction from Images,Vickie Ye · Vickie Ye · Georgios Pavlakos · Michael J. Black · Angjoo Kanazawa,https://muelea.github.io/buddi/,https://arxiv.org/abs/2306.09337,,2306.09337.pdf,Generative Proxemics: A Prior for 3D Social Interaction from Images,"Social interaction is a fundamental aspect of human behavior and
+communication. The way individuals position themselves in relation to others,
+also known as proxemics, conveys social cues and affects the dynamics of social
+interaction. Reconstructing such interaction from images presents challenges
+because of mutual occlusion and the limited availability of large training
+datasets. To address this, we present a novel approach that learns a prior over
+the 3D proxemics two people in close social interaction and demonstrate its use
+for single-view 3D reconstruction. We start by creating 3D training data of
+interacting people using image datasets with contact annotations. We then model
+the proxemics using a novel denoising diffusion model called BUDDI that learns
+the joint distribution over the poses of two people in close social
+interaction. Sampling from our generative proxemics model produces realistic 3D
+human interactions, which we validate through a perceptual study. We use BUDDI
+in reconstructing two people in close proximity from a single image without any
+contact annotation via an optimization approach that uses the diffusion model
+as a prior. Our approach recovers accurate and plausible 3D social interactions
+from noisy initial estimates, outperforming state-of-the-art methods. Our code,
+data, and model are availableat our project website at: muelea.github.io/buddi.",cs.CV,['cs.CV']
+Learning Degradation Independent Representations for Camera ISP Pipelines,Yanhui Guo · Fangzhou Luo · Xiaolin Wu, ,https://arxiv.org/abs/2307.00761v3,,2307.00761v3.pdf,Learning Degradation-Independent Representations for Camera ISP Pipelines,"Image signal processing (ISP) pipeline plays a fundamental role in digital
+cameras, which converts raw Bayer sensor data to RGB images. However,
+ISP-generated images usually suffer from imperfections due to the compounded
+degradations that stem from sensor noises, demosaicing noises, compression
+artifacts, and possibly adverse effects of erroneous ISP hyperparameter
+settings such as ISO and gamma values. In a general sense, these ISP
+imperfections can be considered as degradations. The highly complex mechanisms
+of ISP degradations, some of which are even unknown, pose great challenges to
+the generalization capability of deep neural networks (DNN) for image
+restoration and to their adaptability to downstream tasks. To tackle the
+issues, we propose a novel DNN approach to learn degradation-independent
+representations (DiR) through the refinement of a self-supervised learned
+baseline representation. The proposed DiR learning technique has remarkable
+domain generalization capability and consequently, it outperforms
+state-of-the-art methods across various downstream tasks, including blind image
+restoration, object detection, and instance segmentation, as verified in our
+experiments.",cs.CV,['cs.CV']
+VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation,XuDong Wang · Ishan Misra · Ziyun Zeng · Rohit Girdhar · Trevor Darrell, ,https://arxiv.org/abs/2308.14710,,2308.14710.pdf,VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation,"Existing approaches to unsupervised video instance segmentation typically
+rely on motion estimates and experience difficulties tracking small or
+divergent motions. We present VideoCutLER, a simple method for unsupervised
+multi-instance video segmentation without using motion-based learning signals
+like optical flow or training on natural videos. Our key insight is that using
+high-quality pseudo masks and a simple video synthesis method for model
+training is surprisingly sufficient to enable the resulting video model to
+effectively segment and track multiple instances across video frames. We show
+the first competitive unsupervised learning results on the challenging
+YouTubeVIS-2019 benchmark, achieving 50.7% APvideo^50 , surpassing the previous
+state-of-the-art by a large margin. VideoCutLER can also serve as a strong
+pretrained model for supervised video instance segmentation tasks, exceeding
+DINO by 15.9% on YouTubeVIS-2019 in terms of APvideo.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching,Xinghui Li · Jingyi Lu · Kai Han · Victor Adrian Prisacariu, ,https://arxiv.org/abs/2310.17569,,2310.17569.pdf,SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching,"In this paper, we address the challenge of matching semantically similar
+keypoints across image pairs. Existing research indicates that the intermediate
+output of the UNet within the Stable Diffusion (SD) can serve as robust image
+feature maps for such a matching task. We demonstrate that by employing a basic
+prompt tuning technique, the inherent potential of Stable Diffusion can be
+harnessed, resulting in a significant enhancement in accuracy over previous
+approaches. We further introduce a novel conditional prompting module that
+conditions the prompt on the local details of the input image pairs, leading to
+a further improvement in performance. We designate our approach as SD4Match,
+short for Stable Diffusion for Semantic Matching. Comprehensive evaluations of
+SD4Match on the PF-Pascal, PF-Willow, and SPair-71k datasets show that it sets
+new benchmarks in accuracy across all these datasets. Particularly, SD4Match
+outperforms the previous state-of-the-art by a margin of 12 percentage points
+on the challenging SPair-71k dataset.",cs.CV,"['cs.CV', 'cs.LG']"
+PoNQ: a Neural QEM-based Mesh Representation,Nissim Maruani · Maks Ovsjanikov · Pierre Alliez · Mathieu Desbrun,https://nissmar.github.io/projects/ponq/,https://arxiv.org/abs/2403.12870,,2403.12870.pdf,PoNQ: a Neural QEM-based Mesh Representation,"Although polygon meshes have been a standard representation in geometry
+processing, their irregular and combinatorial nature hinders their suitability
+for learning-based applications. In this work, we introduce a novel learnable
+mesh representation through a set of local 3D sample Points and their
+associated Normals and Quadric error metrics (QEM) w.r.t. the underlying shape,
+which we denote PoNQ. A global mesh is directly derived from PoNQ by
+efficiently leveraging the knowledge of the local quadric errors. Besides
+marking the first use of QEM within a neural shape representation, our
+contribution guarantees both topological and geometrical properties by ensuring
+that a PoNQ mesh does not self-intersect and is always the boundary of a
+volume. Notably, our representation does not rely on a regular grid, is
+supervised directly by the target surface alone, and also handles open surfaces
+with boundaries and/or sharp features. We demonstrate the efficacy of PoNQ
+through a learning-based mesh prediction from SDF grids and show that our
+method surpasses recent state-of-the-art techniques in terms of both surface
+and edge-based metrics.",cs.CV,['cs.CV']
+M&M VTO: Multi-Garment Virtual Try-On and Editing,Luyang Zhu · Yingwei Li · Nan Liu · Hao Peng · Dawei Yang · Ira Kemelmacher-Shlizerman,https://mmvto.github.io/,https://arxiv.org/abs/2405.07472,,2405.07472.pdf,GaussianVTON: 3D Human Virtual Try-ON via Multi-Stage Gaussian Splatting Editing with Image Prompting,"The increasing prominence of e-commerce has underscored the importance of
+Virtual Try-On (VTON). However, previous studies predominantly focus on the 2D
+realm and rely heavily on extensive data for training. Research on 3D VTON
+primarily centers on garment-body shape compatibility, a topic extensively
+covered in 2D VTON. Thanks to advances in 3D scene editing, a 2D diffusion
+model has now been adapted for 3D editing via multi-viewpoint editing. In this
+work, we propose GaussianVTON, an innovative 3D VTON pipeline integrating
+Gaussian Splatting (GS) editing with 2D VTON. To facilitate a seamless
+transition from 2D to 3D VTON, we propose, for the first time, the use of only
+images as editing prompts for 3D editing. To further address issues, e.g., face
+blurring, garment inaccuracy, and degraded viewpoint quality during editing, we
+devise a three-stage refinement strategy to gradually mitigate potential
+issues. Furthermore, we introduce a new editing strategy termed Edit Recall
+Reconstruction (ERR) to tackle the limitations of previous editing strategies
+in leading to complex geometric changes. Our comprehensive experiments
+demonstrate the superiority of GaussianVTON, offering a novel perspective on 3D
+VTON while also establishing a novel starting point for image-prompting 3D
+scene editing.",cs.CV,['cs.CV']
+One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls,Minghui Hu · Jianbin Zheng · Chuanxia Zheng · Chaoyue Wang · Dacheng Tao · Tat-Jen Cham, ,https://arxiv.org/abs/2311.15744,,2311.15744.pdf,One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls,"It is well known that many open-released foundational diffusion models have
+difficulty in generating images that substantially depart from average
+brightness, despite such images being present in the training data. This is due
+to an inconsistency: while denoising starts from pure Gaussian noise during
+inference, the training noise schedule retains residual data even in the final
+timestep distribution, due to difficulties in numerical conditioning in
+mainstream formulation, leading to unintended bias during inference. To
+mitigate this issue, certain $\epsilon$-prediction models are combined with an
+ad-hoc offset-noise methodology. In parallel, some contemporary models have
+adopted zero-terminal SNR noise schedules together with
+$\mathbf{v}$-prediction, which necessitate major alterations to pre-trained
+models. However, such changes risk destabilizing a large multitude of
+community-driven applications anchored on these pre-trained models. In light of
+this, our investigation revisits the fundamental causes, leading to our
+proposal of an innovative and principled remedy, called One More Step (OMS). By
+integrating a compact network and incorporating an additional simple yet
+effective step during inference, OMS elevates image fidelity and harmonizes the
+dichotomy between training and inference, while preserving original model
+parameters. Once trained, various pre-trained diffusion models with the same
+latent domain can share the same OMS module.",cs.CV,['cs.CV']
+Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features,Thomas Wimmer · Peter Wonka · Maks Ovsjanikov,https://wimmerth.github.io/back-to-3d.html,https://arxiv.org/abs/2311.18113,,2311.18113.pdf,Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features,"With the immense growth of dataset sizes and computing resources in recent
+years, so-called foundation models have become popular in NLP and vision tasks.
+In this work, we propose to explore foundation models for the task of keypoint
+detection on 3D shapes. A unique characteristic of keypoint detection is that
+it requires semantic and geometric awareness while demanding high localization
+accuracy. To address this problem, we propose, first, to back-project features
+from large pre-trained 2D vision models onto 3D shapes and employ them for this
+task. We show that we obtain robust 3D features that contain rich semantic
+information and analyze multiple candidate features stemming from different 2D
+foundation models. Second, we employ a keypoint candidate optimization module
+which aims to match the average observed distribution of keypoints on the shape
+and is guided by the back-projected features. The resulting approach achieves a
+new state of the art for few-shot keypoint detection on the KeyPointNet
+dataset, almost doubling the performance of the previous best methods.",cs.CV,"['cs.CV', 'cs.GR']"
+Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining,Xiang Chen · Jinshan Pan · Jiangxin Dong,https://github.com/cschenxiang/NeRD-Rain,https://arxiv.org/abs/2404.01547v1,,2404.01547v1.pdf,Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining,"How to effectively explore multi-scale representations of rain streaks is
+important for image deraining. In contrast to existing Transformer-based
+methods that depend mostly on single-scale rain appearance, we develop an
+end-to-end multi-scale Transformer that leverages the potentially useful
+features in various scales to facilitate high-quality image reconstruction. To
+better explore the common degradation representations from spatially-varying
+rain streaks, we incorporate intra-scale implicit neural representations based
+on pixel coordinates with the degraded inputs in a closed-loop design, enabling
+the learned features to facilitate rain removal and improve the robustness of
+the model in complex scenarios. To ensure richer collaborative representation
+from different scales, we embed a simple yet effective inter-scale
+bidirectional feedback operation into our multi-scale Transformer by performing
+coarse-to-fine and fine-to-coarse information communication. Extensive
+experiments demonstrate that our approach, named as NeRD-Rain, performs
+favorably against the state-of-the-art ones on both synthetic and real-world
+benchmark datasets. The source code and trained models are available at
+https://github.com/cschenxiang/NeRD-Rain.",cs.CV,['cs.CV']
+InstanceDiffusion: Instance-level Control for Image Generation,XuDong Wang · Trevor Darrell · Sai Saketh Rambhatla · Rohit Girdhar · Ishan Misra, ,https://arxiv.org/abs/2402.03290,,2402.03290.pdf,InstanceDiffusion: Instance-level Control for Image Generation,"Text-to-image diffusion models produce high quality images but do not offer
+control over individual instances in the image. We introduce InstanceDiffusion
+that adds precise instance-level control to text-to-image diffusion models.
+InstanceDiffusion supports free-form language conditions per instance and
+allows flexible ways to specify instance locations such as simple single
+points, scribbles, bounding boxes or intricate instance segmentation masks, and
+combinations thereof. We propose three major changes to text-to-image models
+that enable precise instance-level control. Our UniFusion block enables
+instance-level conditions for text-to-image models, the ScaleU block improves
+image fidelity, and our Multi-instance Sampler improves generations for
+multiple instances. InstanceDiffusion significantly surpasses specialized
+state-of-the-art models for each location condition. Notably, on the COCO
+dataset, we outperform previous state-of-the-art by 20.4% AP$_{50}^\text{box}$
+for box inputs, and 25.4% IoU for mask inputs.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Constructing and Exploring Intermediate Domains in Mixed Domain Semi-supervised Medical Image Segmentation,Qinghe Ma · Jian Zhang · Lei Qi · Qian Yu · Yinghuan Shi · Yang Gao, ,https://arxiv.org/abs/2404.08951,,2404.08951.pdf,Constructing and Exploring Intermediate Domains in Mixed Domain Semi-supervised Medical Image Segmentation,"Both limited annotation and domain shift are prevalent challenges in medical
+image segmentation. Traditional semi-supervised segmentation and unsupervised
+domain adaptation methods address one of these issues separately. However, the
+coexistence of limited annotation and domain shift is quite common, which
+motivates us to introduce a novel and challenging scenario: Mixed Domain
+Semi-supervised medical image Segmentation (MiDSS). In this scenario, we handle
+data from multiple medical centers, with limited annotations available for a
+single domain and a large amount of unlabeled data from multiple domains. We
+found that the key to solving the problem lies in how to generate reliable
+pseudo labels for the unlabeled data in the presence of domain shift with
+labeled data. To tackle this issue, we employ Unified Copy-Paste (UCP) between
+images to construct intermediate domains, facilitating the knowledge transfer
+from the domain of labeled data to the domains of unlabeled data. To fully
+utilize the information within the intermediate domain, we propose a symmetric
+Guidance training strategy (SymGD), which additionally offers direct guidance
+to unlabeled data by merging pseudo labels from intermediate samples.
+Subsequently, we introduce a Training Process aware Random Amplitude MixUp
+(TP-RAM) to progressively incorporate style-transition components into
+intermediate samples. Compared with existing state-of-the-art approaches, our
+method achieves a notable 13.57% improvement in Dice score on Prostate dataset,
+as demonstrated on three public datasets. Our code is available at
+https://github.com/MQinghe/MiDSS .",cs.CV,"['cs.CV', 'cs.LG']"
+NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors,Yannan He · Garvita Tiwari · Tolga Birdal · Jan Lenssen · Gerard Pons-Moll, ,https://arxiv.org/abs/2403.03122v1,,2403.03122v1.pdf,NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors,"Faithfully modeling the space of articulations is a crucial task that allows
+recovery and generation of realistic poses, and remains a notorious challenge.
+To this end, we introduce Neural Riemannian Distance Fields (NRDFs),
+data-driven priors modeling the space of plausible articulations, represented
+as the zero-level-set of a neural field in a high-dimensional
+product-quaternion space. To train NRDFs only on positive examples, we
+introduce a new sampling algorithm, ensuring that the geodesic distances follow
+a desired distribution, yielding a principled distance field learning paradigm.
+We then devise a projection algorithm to map any random pose onto the level-set
+by an adaptive-step Riemannian optimizer, adhering to the product manifold of
+joint rotations at all times. NRDFs can compute the Riemannian gradient via
+backpropagation and by mathematical analogy, are related to Riemannian flow
+matching, a recent generative model. We conduct a comprehensive evaluation of
+NRDF against other pose priors in various downstream tasks, i.e., pose
+generation, image-based pose estimation, and solving inverse kinematics,
+highlighting NRDF's superior performance. Besides humans, NRDF's versatility
+extends to hand and animal poses, as it can effectively represent any
+articulation.",cs.CV,['cs.CV']
+Privacy-Preserving Face Recognition Using Trainable Feature Subtraction,Yuxi Mi · Zhizhou Zhong · Yuge Huang · Jiazhen Ji · Jianqing Xu · Jun Wang · ShaoMing Wang · Shouhong Ding · Shuigeng Zhou,https://github.com/Tencent/TFace/tree/master/recognition/tasks/minusface,https://arxiv.org/abs/2403.12457,,,Privacy-Preserving Face Recognition Using Trainable Feature Subtraction,"The widespread adoption of face recognition has led to increasing privacy
+concerns, as unauthorized access to face images can expose sensitive personal
+information. This paper explores face image protection against viewing and
+recovery attacks. Inspired by image compression, we propose creating a visually
+uninformative face image through feature subtraction between an original face
+and its model-produced regeneration. Recognizable identity features within the
+image are encouraged by co-training a recognition model on its high-dimensional
+feature representation. To enhance privacy, the high-dimensional representation
+is crafted through random channel shuffling, resulting in randomized
+recognizable images devoid of attacker-leverageable texture details. We distill
+our methodologies into a novel privacy-preserving face recognition method,
+MinusFace. Experiments demonstrate its high recognition accuracy and effective
+privacy protection. Its code is available at https://github.com/Tencent/TFace.",cs.CV,['cs.CV']
+Generating Human Motion in 3D Scenes from Text Descriptions,Zhi Cen · Huaijin Pi · Sida Peng · Zehong Shen · Minghui Yang · Shuai Zhu · Hujun Bao · Xiaowei Zhou,https://zju3dv.github.io/text_scene_motion/,https://arxiv.org/html/2405.07784v1,,2405.07784v1.pdf,Generating Human Motion in 3D Scenes from Text Descriptions,"Generating human motions from textual descriptions has gained growing
+research interest due to its wide range of applications. However, only a few
+works consider human-scene interactions together with text conditions, which is
+crucial for visual and physical realism. This paper focuses on the task of
+generating human motions in 3D indoor scenes given text descriptions of the
+human-scene interactions. This task presents challenges due to the
+multi-modality nature of text, scene, and motion, as well as the need for
+spatial reasoning. To address these challenges, we propose a new approach that
+decomposes the complex problem into two more manageable sub-problems: (1)
+language grounding of the target object and (2) object-centric motion
+generation. For language grounding of the target object, we leverage the power
+of large language models. For motion generation, we design an object-centric
+scene representation for the generative model to focus on the target object,
+thereby reducing the scene complexity and facilitating the modeling of the
+relationship between human motions and the object. Experiments demonstrate the
+better motion quality of our approach compared to baselines and validate our
+design choices.",cs.CV,['cs.CV']
+HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting,Xian Liu · Xiaohang Zhan · Jiaxiang Tang · Ying Shan · Gang Zeng · Dahua Lin · Xihui Liu · Ziwei Liu,https://alvinliu0.github.io/projects/HumanGaussian,https://arxiv.org/abs/2311.17061,,2311.17061.pdf,HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting,"Realistic 3D human generation from text prompts is a desirable yet
+challenging task. Existing methods optimize 3D representations like mesh or
+neural fields via score distillation sampling (SDS), which suffers from
+inadequate fine details or excessive training time. In this paper, we propose
+an efficient yet effective framework, HumanGaussian, that generates
+high-quality 3D humans with fine-grained geometry and realistic appearance. Our
+key insight is that 3D Gaussian Splatting is an efficient renderer with
+periodic Gaussian shrinkage or growing, where such adaptive density control can
+be naturally guided by intrinsic human structures. Specifically, 1) we first
+propose a Structure-Aware SDS that simultaneously optimizes human appearance
+and geometry. The multi-modal score function from both RGB and depth space is
+leveraged to distill the Gaussian densification and pruning process. 2)
+Moreover, we devise an Annealed Negative Prompt Guidance by decomposing SDS
+into a noisier generative score and a cleaner classifier score, which well
+addresses the over-saturation issue. The floating artifacts are further
+eliminated based on Gaussian size in a prune-only phase to enhance generation
+smoothness. Extensive experiments demonstrate the superior efficiency and
+competitive quality of our framework, rendering vivid 3D humans under diverse
+scenarios. Project Page: https://alvinliu0.github.io/projects/HumanGaussian",cs.CV,['cs.CV']
+"See, Say, and Segment: Correcting False Premises with LMMs",Tsung-Han Wu · Giscard Biamby · David Chan · Lisa Dunlap · Ritwik Gupta · XuDong Wang · Trevor Darrell · Joseph Gonzalez,https://see-say-segment.github.io/,https://arxiv.org/html/2312.08366v1,,2312.08366v1.pdf,"See, Say, and Segment: Teaching LMMs to Overcome False Premises","Current open-source Large Multimodal Models (LMMs) excel at tasks such as
+open-vocabulary language grounding and segmentation but can suffer under false
+premises when queries imply the existence of something that is not actually
+present in the image. We observe that existing methods that fine-tune an LMM to
+segment images significantly degrade their ability to reliably determine
+(""see"") if an object is present and to interact naturally with humans (""say""),
+a form of catastrophic forgetting. In this work, we propose a cascading and
+joint training approach for LMMs to solve this task, avoiding catastrophic
+forgetting of previous skills. Our resulting model can ""see"" by detecting
+whether objects are present in an image, ""say"" by telling the user if they are
+not, proposing alternative queries or correcting semantic errors in the query,
+and finally ""segment"" by outputting the mask of the desired objects if they
+exist. Additionally, we introduce a novel False Premise Correction benchmark
+dataset, an extension of existing RefCOCO(+/g) referring segmentation datasets
+(which we call FP-RefCOCO(+/g)). The results show that our method not only
+detects false premises up to 55% better than existing approaches, but under
+false premise conditions produces relative cIOU improvements of more than 31%
+over baselines, and produces natural language feedback judged helpful up to 67%
+of the time.",cs.CV,['cs.CV']
+Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios,Jie Xu · Yazhou Ren · Xiaolong Wang · Lei Feng · Zheng Zhang · Gang Niu · Xiaofeng Zhu,https://github.com/SubmissionsIn/MVCAN,,https://submissionsin.github.io/,,,,,nan
+Learned representation-guided diffusion models for large-image generation,Alexandros Graikos · Srikar Yellapragada · Minh-Quan Le · Saarthak Kapse · Prateek Prasanna · Joel Saltz · Dimitris Samaras,https://histodiffusion.github.io/docs/publications/cvpr_24,https://arxiv.org/abs/2312.07330,,2312.07330.pdf,Learned representation-guided diffusion models for large-image generation,"To synthesize high-fidelity samples, diffusion models typically require
+auxiliary data to guide the generation process. However, it is impractical to
+procure the painstaking patch-level annotation effort required in specialized
+domains like histopathology and satellite imagery; it is often performed by
+domain experts and involves hundreds of millions of patches. Modern-day
+self-supervised learning (SSL) representations encode rich semantic and visual
+information. In this paper, we posit that such representations are expressive
+enough to act as proxies to fine-grained human labels. We introduce a novel
+approach that trains diffusion models conditioned on embeddings from SSL. Our
+diffusion models successfully project these features back to high-quality
+histopathology and remote sensing images. In addition, we construct larger
+images by assembling spatially consistent patches inferred from SSL embeddings,
+preserving long-range dependencies. Augmenting real data by generating
+variations of real images improves downstream classifier accuracy for
+patch-level and larger, image-scale classification tasks. Our models are
+effective even on datasets not encountered during training, demonstrating their
+robustness and generalizability. Generating images from learned embeddings is
+agnostic to the source of the embeddings. The SSL embeddings used to generate a
+large image can either be extracted from a reference image, or sampled from an
+auxiliary model conditioned on any related modality (e.g. class labels, text,
+genomic data). As proof of concept, we introduce the text-to-large image
+synthesis paradigm where we successfully synthesize large pathology and
+satellite images out of text descriptions.",cs.CV,['cs.CV']
+PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis,Zhengyao Lv · Yuxiang Wei · Wangmeng Zuo · Kwan-Yee K. Wong, ,https://arxiv.org/abs/2403.01852,,2403.01852.pdf,PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis,"Recent advancements in large-scale pre-trained text-to-image models have led
+to remarkable progress in semantic image synthesis. Nevertheless, synthesizing
+high-quality images with consistent semantics and layout remains a challenge.
+In this paper, we propose the adaPtive LAyout-semantiC fusion modulE (PLACE)
+that harnesses pre-trained models to alleviate the aforementioned issues.
+Specifically, we first employ the layout control map to faithfully represent
+layouts in the feature space. Subsequently, we combine the layout and semantic
+features in a timestep-adaptive manner to synthesize images with realistic
+details. During fine-tuning, we propose the Semantic Alignment (SA) loss to
+further enhance layout alignment. Additionally, we introduce the Layout-Free
+Prior Preservation (LFP) loss, which leverages unlabeled data to maintain the
+priors of pre-trained models, thereby improving the visual quality and semantic
+consistency of synthesized images. Extensive experiments demonstrate that our
+approach performs favorably in terms of visual quality, semantic consistency,
+and layout alignment. The source code and model are available at
+https://github.com/cszy98/PLACE/tree/main.",cs.CV,['cs.CV']
+Regressor-Segmenter Mutual Prompt Learning for Crowd Counting,Mingyue Guo · Li Yuan · Zhaoyi Yan · Binghui Chen · Yaowei Wang · Qixiang Ye, ,https://arxiv.org/abs/2312.01711v2,,2312.01711v2.pdf,Regressor-Segmenter Mutual Prompt Learning for Crowd Counting,"Crowd counting has achieved significant progress by training regressors to
+predict instance positions. In heavily crowded scenarios, however, regressors
+are challenged by uncontrollable annotation variance, which causes density map
+bias and context information inaccuracy. In this study, we propose mutual
+prompt learning (mPrompt), which leverages a regressor and a segmenter as
+guidance for each other, solving bias and inaccuracy caused by annotation
+variance while distinguishing foreground from background. In specific, mPrompt
+leverages point annotations to tune the segmenter and predict pseudo head masks
+in a way of point prompt learning. It then uses the predicted segmentation
+masks, which serve as spatial constraint, to rectify biased point annotations
+as context prompt learning. mPrompt defines a way of mutual information
+maximization from prompt learning, mitigating the impact of annotation variance
+while improving model accuracy. Experiments show that mPrompt significantly
+reduces the Mean Average Error (MAE), demonstrating the potential to be general
+framework for down-stream vision tasks.",cs.CV,['cs.CV']
+SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting,Zhijing Shao · Wang Zhaolong · Zhuang Li · Duotun Wang · Xiangru Lin · Yu Zhang · Mingming Fan · Zeyu Wang,https://initialneil.github.io/SplattingAvatar,https://arxiv.org/abs/2403.05087,,2403.05087.pdf,SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting,"We present SplattingAvatar, a hybrid 3D representation of photorealistic
+human avatars with Gaussian Splatting embedded on a triangle mesh, which
+renders over 300 FPS on a modern GPU and 30 FPS on a mobile device. We
+disentangle the motion and appearance of a virtual human with explicit mesh
+geometry and implicit appearance modeling with Gaussian Splatting. The
+Gaussians are defined by barycentric coordinates and displacement on a triangle
+mesh as Phong surfaces. We extend lifted optimization to simultaneously
+optimize the parameters of the Gaussians while walking on the triangle mesh.
+SplattingAvatar is a hybrid representation of virtual humans where the mesh
+represents low-frequency motion and surface deformation, while the Gaussians
+take over the high-frequency geometry and detailed appearance. Unlike existing
+deformation methods that rely on an MLP-based linear blend skinning (LBS) field
+for motion, we control the rotation and translation of the Gaussians directly
+by mesh, which empowers its compatibility with various animation techniques,
+e.g., skeletal animation, blend shapes, and mesh editing. Trainable from
+monocular videos for both full-body and head avatars, SplattingAvatar shows
+state-of-the-art rendering quality across multiple datasets.",cs.GR,"['cs.GR', 'cs.CV']"
+FastMAC: Stochastic Spectral Sampling of Correspondence Graph,Yifei Zhang · Hao Zhao · Hongyang Li · Siheng Chen,https://github.com/Forrest-110/FastMAC,https://arxiv.org/abs/2403.08770,,2403.08770.pdf,FastMAC: Stochastic Spectral Sampling of Correspondence Graph,"3D correspondence, i.e., a pair of 3D points, is a fundamental concept in
+computer vision. A set of 3D correspondences, when equipped with compatibility
+edges, forms a correspondence graph. This graph is a critical component in
+several state-of-the-art 3D point cloud registration approaches, e.g., the one
+based on maximal cliques (MAC). However, its properties have not been well
+understood. So we present the first study that introduces graph signal
+processing into the domain of correspondence graph. We exploit the generalized
+degree signal on correspondence graph and pursue sampling strategies that
+preserve high-frequency components of this signal. To address time-consuming
+singular value decomposition in deterministic sampling, we resort to a
+stochastic approximate sampling strategy. As such, the core of our method is
+the stochastic spectral sampling of correspondence graph. As an application, we
+build a complete 3D registration algorithm termed as FastMAC, that reaches
+real-time speed while leading to little to none performance drop. Through
+extensive experiments, we validate that FastMAC works for both indoor and
+outdoor benchmarks. For example, FastMAC can accelerate MAC by 80 times while
+maintaining high registration success rate on KITTI. Codes are publicly
+available at https://github.com/Forrest-110/FastMAC.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Fairy: Fast Parallellized Instruction-Guided Video-to-Video Synthesis,Bichen Wu · Ching-Yao Chuang · Xiaoyan Wang · Yichen Jia · Kapil Krishnakumar · Tong Xiao · Feng Liang · Licheng Yu · Peter Vajda, ,https://arxiv.org/abs/2312.13834,,2312.13834.pdf,Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis,"In this paper, we introduce Fairy, a minimalist yet robust adaptation of
+image-editing diffusion models, enhancing them for video editing applications.
+Our approach centers on the concept of anchor-based cross-frame attention, a
+mechanism that implicitly propagates diffusion features across frames, ensuring
+superior temporal coherence and high-fidelity synthesis. Fairy not only
+addresses limitations of previous models, including memory and processing
+speed. It also improves temporal consistency through a unique data augmentation
+strategy. This strategy renders the model equivariant to affine transformations
+in both source and target images. Remarkably efficient, Fairy generates
+120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds,
+outpacing prior works by at least 44x. A comprehensive user study, involving
+1000 generated samples, confirms that our approach delivers superior quality,
+decisively outperforming established methods.",cs.CV,['cs.CV']
+MMA: Multi-Modal Adapter for Vision-Language Models,Lingxiao Yang · Ru-Yuan Zhang · Yanchen Wang · Xiaohua Xie, ,https://arxiv.org/abs/2405.15684,,2405.15684.pdf,Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models,"To bridge the gap between vision and language modalities, Multimodal Large
+Language Models (MLLMs) usually learn an adapter that converts visual inputs to
+understandable tokens for Large Language Models (LLMs). However, most adapters
+generate consistent visual tokens, regardless of the specific objects of
+interest mentioned in the prompt. Since these adapters distribute equal
+attention to every detail in the image and focus on the entire scene, they may
+increase the cognitive load for LLMs, particularly when processing complex
+scenes. To alleviate this problem, we propose prompt-aware adapters. These
+adapters are designed with the capability to dynamically embed visual inputs
+based on the specific focus of the prompt. Specifically, prompt-aware adapters
+utilize both global and local textual features to capture the most relevant
+visual clues from the prompt at both coarse and fine granularity levels. This
+approach significantly enhances the ability of LLMs to understand and interpret
+visual content. Experiments on various visual question answering tasks, such as
+counting and position reasoning, demonstrate the effectiveness of prompt-aware
+adapters.",cs.CV,"['cs.CV', 'cs.AI']"
+RoDLA: Benchmarking the Robustness of Document Layout Analysis Models,Yufan Chen · Jiaming Zhang · Kunyu Peng · Junwei Zheng · Ruiping Liu · Philip H.S. Torr · Rainer Stiefelhagen,https://yufanchen96.github.io/projects/RoDLA/,https://arxiv.org/abs/2403.14442,,2403.14442.pdf,RoDLA: Benchmarking the Robustness of Document Layout Analysis Models,"Before developing a Document Layout Analysis (DLA) model in real-world
+applications, conducting comprehensive robustness testing is essential.
+However, the robustness of DLA models remains underexplored in the literature.
+To address this, we are the first to introduce a robustness benchmark for DLA
+models, which includes 450K document images of three datasets. To cover
+realistic corruptions, we propose a perturbation taxonomy with 36 common
+document perturbations inspired by real-world document processing.
+Additionally, to better understand document perturbation impacts, we propose
+two metrics, Mean Perturbation Effect (mPE) for perturbation assessment and
+Mean Robustness Degradation (mRD) for robustness evaluation. Furthermore, we
+introduce a self-titled model, i.e., Robust Document Layout Analyzer (RoDLA),
+which improves attention mechanisms to boost extraction of robust features.
+Experiments on the proposed benchmarks (PubLayNet-P, DocLayNet-P, and
+M$^6$Doc-P) demonstrate that RoDLA obtains state-of-the-art mRD scores of
+115.7, 135.4, and 150.4, respectively. Compared to previous methods, RoDLA
+achieves notable improvements in mAP of +3.8%, +7.1% and +12.1%, respectively.",cs.CV,['cs.CV']
+LAKE-RED: Camouflaged Images Generation by Latent Background Knowledge Retrieval-Augmented Diffusion,Pancheng Zhao · Peng Xu · Pengda Qin · Deng-Ping Fan · Zhicheng Zhang · Guoli Jia · Bowen Zhou · Jufeng Yang, ,https://arxiv.org/abs/2404.00292,,2404.00292.pdf,LAKE-RED: Camouflaged Images Generation by Latent Background Knowledge Retrieval-Augmented Diffusion,"Camouflaged vision perception is an important vision task with numerous
+practical applications. Due to the expensive collection and labeling costs,
+this community struggles with a major bottleneck that the species category of
+its datasets is limited to a small number of object species. However, the
+existing camouflaged generation methods require specifying the background
+manually, thus failing to extend the camouflaged sample diversity in a low-cost
+manner. In this paper, we propose a Latent Background Knowledge
+Retrieval-Augmented Diffusion (LAKE-RED) for camouflaged image generation. To
+our knowledge, our contributions mainly include: (1) For the first time, we
+propose a camouflaged generation paradigm that does not need to receive any
+background inputs. (2) Our LAKE-RED is the first knowledge retrieval-augmented
+method with interpretability for camouflaged generation, in which we propose an
+idea that knowledge retrieval and reasoning enhancement are separated
+explicitly, to alleviate the task-specific challenges. Moreover, our method is
+not restricted to specific foreground targets or backgrounds, offering a
+potential for extending camouflaged vision perception to more diverse domains.
+(3) Experimental results demonstrate that our method outperforms the existing
+approaches, generating more realistic camouflage images.",cs.CV,['cs.CV']
+Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding,Peng Jin · Ryuichi Takanobu · Cai Zhang · Xiaochun Cao · Li Yuan,https://github.com/PKU-YuanGroup/Chat-UniVi,https://arxiv.org/abs/2311.08046,,2311.08046.pdf,Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding,"Large language models have demonstrated impressive universal capabilities
+across a wide range of open-ended tasks and have extended their utility to
+encompass multimodal conversations. However, existing methods encounter
+challenges in effectively handling both image and video understanding,
+particularly with limited visual tokens. In this work, we introduce Chat-UniVi,
+a Unified Vision-language model capable of comprehending and engaging in
+conversations involving images and videos through a unified visual
+representation. Specifically, we employ a set of dynamic visual tokens to
+uniformly represent images and videos. This representation framework empowers
+the model to efficiently utilize a limited number of visual tokens to
+simultaneously capture the spatial details necessary for images and the
+comprehensive temporal relationship required for videos. Moreover, we leverage
+a multi-scale representation, enabling the model to perceive both high-level
+semantic concepts and low-level visual details. Notably, Chat-UniVi is trained
+on a mixed dataset containing both images and videos, allowing direct
+application to tasks involving both mediums without requiring any
+modifications. Extensive experimental results demonstrate that Chat-UniVi
+consistently outperforms even existing methods exclusively designed for either
+images or videos. Code is available at
+https://github.com/PKU-YuanGroup/Chat-UniVi.",cs.CV,['cs.CV']
+HAVE-FUN: Human Avatar Reconstruction from Few-Shot Unconstrained Images,Xihe Yang · Xingyu Chen · Daiheng Gao · Finn Wong · Xiaoguang Han · Baoyuan Wang, ,https://arxiv.org/abs/2311.15672,,2311.15672.pdf,HAVE-FUN: Human Avatar Reconstruction from Few-Shot Unconstrained Images,"As for human avatar reconstruction, contemporary techniques commonly
+necessitate the acquisition of costly data and struggle to achieve satisfactory
+results from a small number of casual images. In this paper, we investigate
+this task from a few-shot unconstrained photo album. The reconstruction of
+human avatars from such data sources is challenging because of limited data
+amount and dynamic articulated poses. For handling dynamic data, we integrate a
+skinning mechanism with deep marching tetrahedra (DMTet) to form a drivable
+tetrahedral representation, which drives arbitrary mesh topologies generated by
+the DMTet for the adaptation of unconstrained images. To effectively mine
+instructive information from few-shot data, we devise a two-phase optimization
+method with few-shot reference and few-shot guidance. The former focuses on
+aligning avatar identity with reference images, while the latter aims to
+generate plausible appearances for unseen regions. Overall, our framework,
+called HaveFun, can undertake avatar reconstruction, rendering, and animation.
+Extensive experiments on our developed benchmarks demonstrate that HaveFun
+exhibits substantially superior performance in reconstructing the human body
+and hand. Project website: https://seanchenxy.github.io/HaveFunWeb/.",cs.CV,['cs.CV']
+BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP,Jiawang Bai · Kuofeng Gao · Shaobo Min · Shu-Tao Xia · Zhifeng Li · Wei Liu, ,https://arxiv.org/abs/2311.16194,,2311.16194.pdf,BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP,"Contrastive Vision-Language Pre-training, known as CLIP, has shown promising
+effectiveness in addressing downstream image recognition tasks. However, recent
+works revealed that the CLIP model can be implanted with a downstream-oriented
+backdoor. On downstream tasks, one victim model performs well on clean samples
+but predicts a specific target class whenever a specific trigger is present.
+For injecting a backdoor, existing attacks depend on a large amount of
+additional data to maliciously fine-tune the entire pre-trained CLIP model,
+which makes them inapplicable to data-limited scenarios. In this work,
+motivated by the recent success of learnable prompts, we address this problem
+by injecting a backdoor into the CLIP model in the prompt learning stage. Our
+method named BadCLIP is built on a novel and effective mechanism in backdoor
+attacks on CLIP, i.e., influencing both the image and text encoders with the
+trigger. It consists of a learnable trigger applied to images and a
+trigger-aware context generator, such that the trigger can change text features
+via trigger-aware prompts, resulting in a powerful and generalizable attack.
+Extensive experiments conducted on 11 datasets verify that the clean accuracy
+of BadCLIP is similar to those of advanced prompt learning methods and the
+attack success rate is higher than 99% in most cases. BadCLIP is also
+generalizable to unseen classes, and shows a strong generalization capability
+under cross-dataset and cross-domain settings.",cs.CV,['cs.CV']
+PromptKD: Unsupervised Prompt Distillation for Vision-Language Models,Zheng Li · Xiang Li · xinyi fu · Xin Zhang · Weiqiang Wang · Shuo Chen · Jian Yang,https://zhengli97.github.io/PromptKD/,https://arxiv.org/abs/2403.02781v3,,2403.02781v3.pdf,PromptKD: Unsupervised Prompt Distillation for Vision-Language Models,"Prompt learning has emerged as a valuable technique in enhancing
+vision-language models (VLMs) such as CLIP for downstream tasks in specific
+domains. Existing work mainly focuses on designing various learning forms of
+prompts, neglecting the potential of prompts as effective distillers for
+learning from larger teacher models. In this paper, we introduce an
+unsupervised domain prompt distillation framework, which aims to transfer the
+knowledge of a larger teacher model to a lightweight target model through
+prompt-driven imitation using unlabeled domain images. Specifically, our
+framework consists of two distinct stages. In the initial stage, we pre-train a
+large CLIP teacher model using domain (few-shot) labels. After pre-training, we
+leverage the unique decoupled-modality characteristics of CLIP by pre-computing
+and storing the text features as class vectors only once through the teacher
+text encoder. In the subsequent stage, the stored class vectors are shared
+across teacher and student image encoders for calculating the predicted logits.
+Further, we align the logits of both the teacher and student models via KL
+divergence, encouraging the student image encoder to generate similar
+probability distributions to the teacher through the learnable prompts. The
+proposed prompt distillation process eliminates the reliance on labeled data,
+enabling the algorithm to leverage a vast amount of unlabeled images within the
+domain. Finally, the well-trained student image encoders and pre-stored text
+features (class vectors) are utilized for inference. To our best knowledge, we
+are the first to (1) perform unsupervised domain-specific prompt-driven
+knowledge distillation for CLIP, and (2) establish a practical pre-storing
+mechanism of text features as shared class vectors between teacher and student.
+Extensive experiments on 11 datasets demonstrate the effectiveness of our
+method.",cs.CV,['cs.CV']
+IBD-SLAM: Learning Image-Based Depth Fusion for Generalizable SLAM,Minghao Yin · Shangzhe Wu · Kai Han, ,https://arxiv.org/html/2405.03413v2,,2405.03413v2.pdf,SL-SLAM: A robust visual-inertial SLAM based deep feature extraction and matching,"This paper explores how deep learning techniques can improve visual-based
+SLAM performance in challenging environments. By combining deep feature
+extraction and deep matching methods, we introduce a versatile hybrid visual
+SLAM system designed to enhance adaptability in challenging scenarios, such as
+low-light conditions, dynamic lighting, weak-texture areas, and severe jitter.
+Our system supports multiple modes, including monocular, stereo,
+monocular-inertial, and stereo-inertial configurations. We also perform
+analysis how to combine visual SLAM with deep learning methods to enlighten
+other researches. Through extensive experiments on both public datasets and
+self-sampled data, we demonstrate the superiority of the SL-SLAM system over
+traditional approaches. The experimental results show that SL-SLAM outperforms
+state-of-the-art SLAM algorithms in terms of localization accuracy and tracking
+robustness. For the benefit of community, we make public the source code at
+https://github.com/zzzzxxxx111/SLslam.",cs.RO,['cs.RO']
+GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering,Abdullah J Hamdi · Luke Melas-Kyriazi · Jinjie Mai · Guocheng Qian · Ruoshi Liu · Carl Vondrick · Bernard Ghanem · Andrea Vedaldi, ,https://arxiv.org/abs/2402.10128,,2402.10128.pdf,GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering,"Advancements in 3D Gaussian Splatting have significantly accelerated 3D
+reconstruction and generation. However, it may require a large number of
+Gaussians, which creates a substantial memory footprint. This paper introduces
+GES (Generalized Exponential Splatting), a novel representation that employs
+Generalized Exponential Function (GEF) to model 3D scenes, requiring far fewer
+particles to represent a scene and thus significantly outperforming Gaussian
+Splatting methods in efficiency with a plug-and-play replacement ability for
+Gaussian-based utilities. GES is validated theoretically and empirically in
+both principled 1D setup and realistic 3D scenes.
+  It is shown to represent signals with sharp edges more accurately, which are
+typically challenging for Gaussians due to their inherent low-pass
+characteristics. Our empirical analysis demonstrates that GEF outperforms
+Gaussians in fitting natural-occurring signals (e.g. squares, triangles, and
+parabolic signals), thereby reducing the need for extensive splitting
+operations that increase the memory footprint of Gaussian Splatting. With the
+aid of a frequency-modulated loss, GES achieves competitive performance in
+novel-view synthesis benchmarks while requiring less than half the memory
+storage of Gaussian Splatting and increasing the rendering speed by up to 39%.
+The code is available on the project website https://abdullahamdi.com/ges .",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Hybrid Functional Maps for Crease-Aware Non-Isometric Shape Matching,Lennart Bastian · Yizheng Xie · Nassir Navab · Zorah Lähner, ,https://arxiv.org/abs/2312.03678,,2312.03678.pdf,Hybrid Functional Maps for Crease-Aware Non-Isometric Shape Matching,"Non-isometric shape correspondence remains a fundamental challenge in
+computer vision. Traditional methods using Laplace-Beltrami operator (LBO)
+eigenmodes face limitations in characterizing high-frequency extrinsic shape
+changes like bending and creases. We propose a novel approach of combining the
+non-orthogonal extrinsic basis of eigenfunctions of the elastic thin-shell
+hessian with the intrinsic ones of the LBO, creating a hybrid spectral space in
+which we construct functional maps. To this end, we present a theoretical
+framework to effectively integrate non-orthogonal basis functions into
+descriptor- and learning-based functional map methods. Our approach can be
+incorporated easily into existing functional map pipelines across varying
+applications and is able to handle complex deformations beyond isometries. We
+show extensive evaluations across various supervised and unsupervised settings
+and demonstrate significant improvements. Notably, our approach achieves up to
+15% better mean geodesic error for non-isometric correspondence settings and up
+to 45% improvement in scenarios with topological noise.",cs.CV,['cs.CV']
+DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction,Junwen Xiong · Peng Zhang · Tao You · Chuanyue Li · Wei Huang · Yufei Zha,https://github.com/junwenxiong/diff_sal,https://arxiv.org/abs/2403.01226,,2403.01226.pdf,DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction,"Audio-visual saliency prediction can draw support from diverse modality
+complements, but further performance enhancement is still challenged by
+customized architectures as well as task-specific loss functions. In recent
+studies, denoising diffusion models have shown more promising in unifying task
+frameworks owing to their inherent ability of generalization. Following this
+motivation, a novel Diffusion architecture for generalized audio-visual
+Saliency prediction (DiffSal) is proposed in this work, which formulates the
+prediction problem as a conditional generative task of the saliency map by
+utilizing input audio and video as the conditions. Based on the spatio-temporal
+audio-visual features, an extra network Saliency-UNet is designed to perform
+multi-modal attention modulation for progressive refinement of the ground-truth
+saliency map from the noisy map. Extensive experiments demonstrate that the
+proposed DiffSal can achieve excellent performance across six challenging
+audio-visual benchmarks, with an average relative improvement of 6.3\% over the
+previous state-of-the-art results by six metrics.",cs.CV,['cs.CV']
+Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models,Zijin Yang · Kai Zeng · Kejiang Chen · Han Fang · Weiming Zhang · Nenghai Yu, ,https://arxiv.org/abs/2404.04956,,2404.04956.pdf,Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models,"Ethical concerns surrounding copyright protection and inappropriate content
+generation pose challenges for the practical implementation of diffusion
+models. One effective solution involves watermarking the generated images.
+However, existing methods often compromise the model performance or require
+additional training, which is undesirable for operators and users. To address
+this issue, we propose Gaussian Shading, a diffusion model watermarking
+technique that is both performance-lossless and training-free, while serving
+the dual purpose of copyright protection and tracing of offending content. Our
+watermark embedding is free of model parameter modifications and thus is
+plug-and-play. We map the watermark to latent representations following a
+standard Gaussian distribution, which is indistinguishable from latent
+representations obtained from the non-watermarked diffusion model. Therefore we
+can achieve watermark embedding with lossless performance, for which we also
+provide theoretical proof. Furthermore, since the watermark is intricately
+linked with image semantics, it exhibits resilience to lossy processing and
+erasure attempts. The watermark can be extracted by Denoising Diffusion
+Implicit Models (DDIM) inversion and inverse sampling. We evaluate Gaussian
+Shading on multiple versions of Stable Diffusion, and the results demonstrate
+that Gaussian Shading not only is performance-lossless but also outperforms
+existing methods in terms of robustness.",cs.CV,"['cs.CV', 'cs.CR']"
+Benchmarking Audio Visual Segmentation for Long-Untrimmed Videos,Chen Liu · Peike Li · Qingtao Yu · Hongwei Sheng · Dadong Wang · Lincheng Li · Xin Yu, ,https://arxiv.org/abs/2307.16620,,2307.16620.pdf,Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics,"The audio-visual segmentation (AVS) task aims to segment sounding objects
+from a given video. Existing works mainly focus on fusing audio and visual
+features of a given video to achieve sounding object masks. However, we
+observed that prior arts are prone to segment a certain salient object in a
+video regardless of the audio information. This is because sounding objects are
+often the most salient ones in the AVS dataset. Thus, current AVS methods might
+fail to localize genuine sounding objects due to the dataset bias. In this
+work, we present an audio-visual instance-aware segmentation approach to
+overcome the dataset bias. In a nutshell, our method first localizes potential
+sounding objects in a video by an object segmentation network, and then
+associates the sounding object candidates with the given audio. We notice that
+an object could be a sounding object in one video but a silent one in another
+video. This would bring ambiguity in training our object segmentation network
+as only sounding objects have corresponding segmentation masks. We thus propose
+a silent object-aware segmentation objective to alleviate the ambiguity.
+Moreover, since the category information of audio is unknown, especially for
+multiple sounding sources, we propose to explore the audio-visual semantic
+correlation and then associate audio with potential objects. Specifically, we
+attend predicted audio category scores to potential instance masks and these
+scores will highlight corresponding sounding instances while suppressing
+inaudible ones. When we enforce the attended instance masks to resemble the
+ground-truth mask, we are able to establish audio-visual semantics correlation.
+Experimental results on the AVS benchmarks demonstrate that our method can
+effectively segment sounding objects without being biased to salient objects.",cs.SD,"['cs.SD', 'cs.CV', 'eess.AS']"
+Modular Blind Video Quality Assessment,Wen Wen · Mu Li · Yabin ZHANG · Yiting Liao · Junlin Li · Li zhang · Kede Ma, ,https://arxiv.org/abs/2402.19276,,2402.19276.pdf,Modular Blind Video Quality Assessment,"Blind video quality assessment (BVQA) plays a pivotal role in evaluating and
+improving the viewing experience of end-users across a wide range of
+video-based platforms and services. Contemporary deep learning-based models
+primarily analyze video content in its aggressively subsampled format, while
+being blind to the impact of the actual spatial resolution and frame rate on
+video quality. In this paper, we propose a modular BVQA model and a method of
+training it to improve its modularity. Our model comprises a base quality
+predictor, a spatial rectifier, and a temporal rectifier, responding to the
+visual content and distortion, spatial resolution, and frame rate changes on
+video quality, respectively. During training, spatial and temporal rectifiers
+are dropped out with some probabilities to render the base quality predictor a
+standalone BVQA model, which should work better with the rectifiers. Extensive
+experiments on both professionally-generated content and user-generated content
+video databases show that our quality model achieves superior or comparable
+performance to current methods. Additionally, the modularity of our model
+offers an opportunity to analyze existing video quality databases in terms of
+their spatial and temporal complexity.",eess.IV,"['eess.IV', 'cs.CV']"
+Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation,Ji-Jia Wu · Andy Chia-Hao Chang · Chieh-Yu Chuang · Chun-Pei Chen · Yu-Lun Liu · Min-Hung Chen · Hou-Ning Hu · Yung-Yu Chuang · Yen-Yu Lin, ,https://arxiv.org/abs/2404.04231,,2404.04231.pdf,Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation,"This paper addresses text-supervised semantic segmentation, aiming to learn a
+model capable of segmenting arbitrary visual concepts within images by using
+only image-text pairs without dense annotations. Existing methods have
+demonstrated that contrastive learning on image-text pairs effectively aligns
+visual segments with the meanings of texts. We notice that there is a
+discrepancy between text alignment and semantic segmentation: A text often
+consists of multiple semantic concepts, whereas semantic segmentation strives
+to create semantically homogeneous segments. To address this issue, we propose
+a novel framework, Image-Text Co-Decomposition (CoDe), where the paired image
+and text are jointly decomposed into a set of image regions and a set of word
+segments, respectively, and contrastive learning is developed to enforce
+region-word alignment. To work with a vision-language model, we present a
+prompt learning mechanism that derives an extra representation to highlight an
+image segment or a word segment of interest, with which more effective features
+can be extracted from that segment. Comprehensive experimental results
+demonstrate that our method performs favorably against existing text-supervised
+semantic segmentation methods on six benchmark datasets.",cs.CV,['cs.CV']
+Detector-Free Structure from Motion,Xingyi He · Jiaming Sun · Yifan Wang · Sida Peng · Qixing Huang · Hujun Bao · Xiaowei Zhou, ,https://arxiv.org/abs/2306.15669,,2306.15669.pdf,Detector-Free Structure from Motion,"We propose a new structure-from-motion framework to recover accurate camera
+poses and point clouds from unordered images. Traditional SfM systems typically
+rely on the successful detection of repeatable keypoints across multiple views
+as the first step, which is difficult for texture-poor scenes, and poor
+keypoint detection may break down the whole SfM system. We propose a new
+detector-free SfM framework to draw benefits from the recent success of
+detector-free matchers to avoid the early determination of keypoints, while
+solving the multi-view inconsistency issue of detector-free matchers.
+Specifically, our framework first reconstructs a coarse SfM model from
+quantized detector-free matches. Then, it refines the model by a novel
+iterative refinement pipeline, which iterates between an attention-based
+multi-view matching module to refine feature tracks and a geometry refinement
+module to improve the reconstruction accuracy. Experiments demonstrate that the
+proposed framework outperforms existing detector-based SfM systems on common
+benchmark datasets. We also collect a texture-poor SfM dataset to demonstrate
+the capability of our framework to reconstruct texture-poor scenes. Based on
+this framework, we take $\textit{first place}$ in Image Matching Challenge
+2023.",cs.CV,['cs.CV']
+Simple Semantic-Aided Few-Shot Learning,Hai Zhang · Junzhe Xu · Shanlin Jiang · Zhenan He,https://github.com/zhangdoudou123/SemFew,https://arxiv.org/abs/2311.18649,,2311.18649.pdf,Simple Semantic-Aided Few-Shot Learning,"Learning from a limited amount of data, namely Few-Shot Learning, stands out
+as a challenging computer vision task. Several works exploit semantics and
+design complicated semantic fusion mechanisms to compensate for rare
+representative features within restricted data. However, relying on naive
+semantics such as class names introduces biases due to their brevity, while
+acquiring extensive semantics from external knowledge takes a huge time and
+effort. This limitation severely constrains the potential of semantics in
+Few-Shot Learning. In this paper, we design an automatic way called Semantic
+Evolution to generate high-quality semantics. The incorporation of high-quality
+semantics alleviates the need for complex network structures and learning
+algorithms used in previous works. Hence, we employ a simple two-layer network
+termed Semantic Alignment Network to transform semantics and visual features
+into robust class prototypes with rich discriminative features for few-shot
+classification. The experimental results show our framework outperforms all
+previous methods on six benchmarks, demonstrating a simple network with
+high-quality semantics can beat intricate multi-modal modules on few-shot
+classification tasks. Code is available at
+https://github.com/zhangdoudou123/SemFew.",cs.CV,['cs.CV']
+iToF-flow-based High Frame Rate Depth Imaging,Yu Meng · Zhou Xue · Xu Chang · Xuemei Hu · Tao Yue, ,https://arxiv.org/abs/2306.17618,,2306.17618.pdf,Polarimetric iToF: Measuring High-Fidelity Depth through Scattering Media,"Indirect time-of-flight (iToF) imaging allows us to capture dense depth
+information at a low cost. However, iToF imaging often suffers from multipath
+interference (MPI) artifacts in the presence of scattering media, resulting in
+severe depth-accuracy degradation. For instance, iToF cameras cannot measure
+depth accurately through fog because ToF active illumination scatters back to
+the sensor before reaching the farther target surface. In this work, we propose
+a polarimetric iToF imaging method that can capture depth information robustly
+through scattering media. Our observations on the principle of indirect ToF
+imaging and polarization of light allow us to formulate a novel computational
+model of scattering-aware polarimetric phase measurements that enables us to
+correct MPI errors. We first devise a scattering-aware polarimetric iToF model
+that can estimate the phase of unpolarized backscattered light. We then combine
+the optical filtering of polarization and our computational modeling of
+unpolarized backscattered light via scattering analysis of phase and amplitude.
+This allows us to tackle the MPI problem by estimating the scattering energy
+through the participating media. We validate our method on an experimental
+setup using a customized off-the-shelf iToF camera. Our method outperforms
+baseline methods by a significant margin by means of our scattering model and
+polarimetric phase measurements.",cs.CV,['cs.CV']
+Perceptual-Oriented Video Frame Interpolation Via Asymmetric Synergistic Blending,Guangyang Wu · Xin Tao · Changlin Li · Wenyi Wang · Xiaohong Liu · Qingqing Zheng, ,https://arxiv.org/abs/2404.06692,,2404.06692.pdf,Perception-Oriented Video Frame Interpolation via Asymmetric Blending,"Previous methods for Video Frame Interpolation (VFI) have encountered
+challenges, notably the manifestation of blur and ghosting effects. These
+issues can be traced back to two pivotal factors: unavoidable motion errors and
+misalignment in supervision. In practice, motion estimates often prove to be
+error-prone, resulting in misaligned features. Furthermore, the reconstruction
+loss tends to bring blurry results, particularly in misaligned regions. To
+mitigate these challenges, we propose a new paradigm called PerVFI
+(Perception-oriented Video Frame Interpolation). Our approach incorporates an
+Asymmetric Synergistic Blending module (ASB) that utilizes features from both
+sides to synergistically blend intermediate features. One reference frame
+emphasizes primary content, while the other contributes complementary
+information. To impose a stringent constraint on the blending process, we
+introduce a self-learned sparse quasi-binary mask which effectively mitigates
+ghosting and blur artifacts in the output. Additionally, we employ a
+normalizing flow-based generator and utilize the negative log-likelihood loss
+to learn the conditional distribution of the output, which further facilitates
+the generation of clear and fine details. Experimental results validate the
+superiority of PerVFI, demonstrating significant improvements in perceptual
+quality compared to existing methods. Codes are available at
+\url{https://github.com/mulns/PerVFI}",cs.CV,['cs.CV']
+SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos,Changan Chen · Kumar Ashutosh · Rohit Girdhar · David Harwath · Kristen Grauman,https://vision.cs.utexas.edu/projects/soundingactions/,https://arxiv.org/abs/2404.05206,,,SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos,"We propose a novel self-supervised embedding to learn how actions sound from
+narrated in-the-wild egocentric videos. Whereas existing methods rely on
+curated data with known audio-visual correspondence, our multimodal
+contrastive-consensus coding (MC3) embedding reinforces the associations
+between audio, language, and vision when all modality pairs agree, while
+diminishing those associations when any one pair does not. We show our approach
+can successfully discover how the long tail of human actions sound from
+egocentric video, outperforming an array of recent multimodal embedding
+techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal
+tasks.",cs.CV,"['cs.CV', 'cs.MM', 'cs.SD', 'eess.AS']"
+Dynamic LiDAR Re-simulation using Compositional Neural Fields,Hanfeng Wu · Xingxing Zuo · Stefan Leutenegger · Or Litany · Konrad Schindler · Shengyu Huang, ,https://arxiv.org/abs/2312.05247,,2312.05247.pdf,Dynamic LiDAR Re-simulation using Compositional Neural Fields,"We introduce DyNFL, a novel neural field-based approach for high-fidelity
+re-simulation of LiDAR scans in dynamic driving scenes. DyNFL processes LiDAR
+measurements from dynamic environments, accompanied by bounding boxes of moving
+objects, to construct an editable neural field. This field, comprising
+separately reconstructed static background and dynamic objects, allows users to
+modify viewpoints, adjust object positions, and seamlessly add or remove
+objects in the re-simulated scene. A key innovation of our method is the neural
+field composition technique, which effectively integrates reconstructed neural
+assets from various scenes through a ray drop test, accounting for occlusions
+and transparent surfaces. Our evaluation with both synthetic and real-world
+environments demonstrates that DyNFL substantially improves dynamic scene LiDAR
+simulation, offering a combination of physical fidelity and flexible editing
+capabilities.",cs.CV,['cs.CV']
+GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding,Zi-Ting Chou · Sheng-Yu Huang · I-Jieh Liu · Yu-Chiang Frank Wang,https://timchou-ntu.github.io/gsnerf/,https://arxiv.org/abs/2403.03608,,2403.03608.pdf,GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding,"Utilizing multi-view inputs to synthesize novel-view images, Neural Radiance
+Fields (NeRF) have emerged as a popular research topic in 3D vision. In this
+work, we introduce a Generalizable Semantic Neural Radiance Field (GSNeRF),
+which uniquely takes image semantics into the synthesis process so that both
+novel view images and the associated semantic maps can be produced for unseen
+scenes. Our GSNeRF is composed of two stages: Semantic Geo-Reasoning and
+Depth-Guided Visual rendering. The former is able to observe multi-view image
+inputs to extract semantic and geometry features from a scene. Guided by the
+resulting image geometry information, the latter performs both image and
+semantic rendering with improved performances. Our experiments not only confirm
+that GSNeRF performs favorably against prior works on both novel-view image and
+semantic segmentation synthesis but the effectiveness of our sampling strategy
+for visual rendering is further verified.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,Kunchang Li · Yali Wang · Yinan He · Yizhuo Li · Yi Wang · Yi Liu · Zun Wang · Jilan Xu · Guo Chen · Ping Luo · Limin Wang · Yu Qiao,https://github.com/OpenGVLab/Ask-Anything/tree/main/video_chat2,https://arxiv.org/abs/2311.17005,,2311.17005.pdf,MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,"With the rapid development of Multi-modal Large Language Models (MLLMs), a
+number of diagnostic benchmarks have recently emerged to evaluate the
+comprehension capabilities of these models. However, most benchmarks
+predominantly assess spatial understanding in the static image tasks, while
+overlooking temporal understanding in the dynamic video tasks. To alleviate
+this issue, we introduce a comprehensive Multi-modal Video understanding
+Benchmark, namely MVBench, which covers 20 challenging video tasks that cannot
+be effectively solved with a single frame. Specifically, we first introduce a
+novel static-to-dynamic method to define these temporal-related tasks. By
+transforming various static tasks into dynamic ones, we enable the systematic
+generation of video tasks that require a broad spectrum of temporal skills,
+ranging from perception to cognition. Then, guided by the task definition, we
+automatically convert public video annotations into multiple-choice QA to
+evaluate each task. On one hand, such a distinct paradigm allows us to build
+MVBench efficiently, without much manual intervention. On the other hand, it
+guarantees evaluation fairness with ground-truth video annotations, avoiding
+the biased scoring of LLMs. Moreover, we further develop a robust video MLLM
+baseline, i.e., VideoChat2, by progressive multi-modal training with diverse
+instruction-tuning data. The extensive results on our MVBench reveal that, the
+existing MLLMs are far from satisfactory in temporal understanding, while our
+VideoChat2 largely surpasses these leading models by over 15% on MVBench. All
+models and data are available at https://github.com/OpenGVLab/Ask-Anything.",cs.CV,['cs.CV']
+Unsupervised Gaze Representation Learning from Multi-view Face Images,Yiwei Bao · Feng Lu, ,https://arxiv.org/abs/2309.04506,,2309.04506.pdf,Unsupervised Gaze-aware Contrastive Learning with Subject-specific Condition,"Appearance-based gaze estimation has shown great promise in many applications
+by using a single general-purpose camera as the input device. However, its
+success is highly depending on the availability of large-scale well-annotated
+gaze datasets, which are sparse and expensive to collect. To alleviate this
+challenge we propose ConGaze, a contrastive learning-based framework that
+leverages unlabeled facial images to learn generic gaze-aware representations
+across subjects in an unsupervised way. Specifically, we introduce the
+gaze-specific data augmentation to preserve the gaze-semantic features and
+maintain the gaze consistency, which are proven to be crucial for effective
+contrastive gaze representation learning. Moreover, we devise a novel
+subject-conditional projection module that encourages a share feature extractor
+to learn gaze-aware and generic representations. Our experiments on three
+public gaze estimation datasets show that ConGaze outperforms existing
+unsupervised learning solutions by 6.7% to 22.5%; and achieves 15.1% to 24.6%
+improvement over its supervised learning-based counterpart in cross-dataset
+evaluations.",cs.CV,['cs.CV']
+DIOD: Self-Distillation Meets Object Discovery,Sandra Kara · Hejer AMMAR · Julien Denize · Florian Chabot · Quoc Cuong PHAM, ,https://arxiv.org/abs/2311.02633,,2311.02633.pdf,The Background Also Matters: Background-Aware Motion-Guided Objects Discovery,"Recent works have shown that objects discovery can largely benefit from the
+inherent motion information in video data. However, these methods lack a proper
+background processing, resulting in an over-segmentation of the non-object
+regions into random segments. This is a critical limitation given the
+unsupervised setting, where object segments and noise are not distinguishable.
+To address this limitation we propose BMOD, a Background-aware Motion-guided
+Objects Discovery method. Concretely, we leverage masks of moving objects
+extracted from optical flow and design a learning mechanism to extend them to
+the true foreground composed of both moving and static objects. The background,
+a complementary concept of the learned foreground class, is then isolated in
+the object discovery process. This enables a joint learning of the objects
+discovery task and the object/non-object separation. The conducted experiments
+on synthetic and real-world datasets show that integrating our background
+handling with various cutting-edge methods brings each time a considerable
+improvement. Specifically, we improve the objects discovery performance with a
+large margin, while establishing a strong baseline for object/non-object
+separation.",cs.CV,['cs.CV']
+$\textbf{LaRE}^2$: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detection,Yunpeng Luo · Junlong Du · Ke Yan · Shouhong Ding, ,https://arxiv.org/abs/2403.17465,,2403.17465.pdf,LaRE^2: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detection,"The evolution of Diffusion Models has dramatically improved image generation
+quality, making it increasingly difficult to differentiate between real and
+generated images. This development, while impressive, also raises significant
+privacy and security concerns. In response to this, we propose a novel Latent
+REconstruction error guided feature REfinement method (LaRE^2) for detecting
+the diffusion-generated images. We come up with the Latent Reconstruction Error
+(LaRE), the first reconstruction-error based feature in the latent space for
+generated image detection. LaRE surpasses existing methods in terms of feature
+extraction efficiency while preserving crucial cues required to differentiate
+between the real and the fake. To exploit LaRE, we propose an Error-Guided
+feature REfinement module (EGRE), which can refine the image feature guided by
+LaRE to enhance the discriminativeness of the feature. Our EGRE utilizes an
+align-then-refine mechanism, which effectively refines the image feature for
+generated-image detection from both spatial and channel perspectives. Extensive
+experiments on the large-scale GenImage benchmark demonstrate the superiority
+of our LaRE^2, which surpasses the best SoTA method by up to 11.9%/12.1%
+average ACC/AP across 8 different image generators. LaRE also surpasses
+existing methods in terms of feature extraction cost, delivering an impressive
+speed enhancement of 8 times.",cs.CV,"['cs.CV', 'cs.AI']"
+MindBridge: A Cross-Subject Brain Decoding Framework,Shizun Wang · Songhua Liu · Zhenxiong Tan · Xinchao Wang,https://littlepure2333.github.io/MindBridge/,https://arxiv.org/abs/2404.07850,,2404.07850.pdf,MindBridge: A Cross-Subject Brain Decoding Framework,"Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli
+from acquired brain signals, primarily utilizing functional magnetic resonance
+imaging (fMRI). Currently, brain decoding is confined to a
+per-subject-per-model paradigm, limiting its applicability to the same
+individual for whom the decoding model is trained. This constraint stems from
+three key challenges: 1) the inherent variability in input dimensions across
+subjects due to differences in brain size; 2) the unique intrinsic neural
+patterns, influencing how different individuals perceive and process sensory
+information; 3) limited data availability for new subjects in real-world
+scenarios hampers the performance of decoding models. In this paper, we present
+a novel approach, MindBridge, that achieves cross-subject brain decoding by
+employing only one model. Our proposed framework establishes a generic paradigm
+capable of addressing these challenges by introducing biological-inspired
+aggregation function and novel cyclic fMRI reconstruction mechanism for
+subject-invariant representation learning. Notably, by cycle reconstruction of
+fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as
+pseudo data augmentation. Within the framework, we also devise a novel
+reset-tuning method for adapting a pretrained model to a new subject.
+Experimental results demonstrate MindBridge's ability to reconstruct images for
+multiple subjects, which is competitive with dedicated subject-specific models.
+Furthermore, with limited data for a new subject, we achieve a high level of
+decoding accuracy, surpassing that of subject-specific models. This advancement
+in cross-subject brain decoding suggests promising directions for wider
+applications in neuroscience and indicates potential for more efficient
+utilization of limited fMRI data in real-world scenarios. Project page:
+https://littlepure2333.github.io/MindBridge",cs.CV,"['cs.CV', 'cs.AI']"
+Capturing Closely Interacted Two-Person Motions with Reaction Priors,Qi Fang · Yinghui Fan · Yanjun Li · Junting Dong · Dingwei Wu · Weidong Zhang · Kang Chen, ,https://arxiv.org/abs/2404.05490,,2404.05490.pdf,Two-Person Interaction Augmentation with Skeleton Priors,"Close and continuous interaction with rich contacts is a crucial aspect of
+human activities (e.g. hugging, dancing) and of interest in many domains like
+activity recognition, motion prediction, character animation, etc. However,
+acquiring such skeletal motion is challenging. While direct motion capture is
+expensive and slow, motion editing/generation is also non-trivial, as complex
+contact patterns with topological and geometric constraints have to be
+retained. To this end, we propose a new deep learning method for two-body
+skeletal interaction motion augmentation, which can generate variations of
+contact-rich interactions with varying body sizes and proportions while
+retaining the key geometric/topological relations between two bodies. Our
+system can learn effectively from a relatively small amount of data and
+generalize to drastically different skeleton sizes. Through exhaustive
+evaluation and comparison, we show it can generate high-quality motions, has
+strong generalizability and outperforms traditional optimization-based methods
+and alternative deep learning solutions.",cs.CV,['cs.CV']
+Text-conditional Attribute Alignment across Latent Spaces for 3D Controllable Face Image Synthesis,FeiFan Xu · Rui Li · Si Wu · Yong Xu · Hau San Wong, ,,https://huggingface.co/papers/2306.17115,,,,,nan
+Purified and Unified Steganographic Network,GuoBiao Li · Sheng Li · Zicong Luo · Zhenxing Qian · Xinpeng Zhang,https://github.com/albblgb/PUSNet,https://arxiv.org/abs/2402.17210,,2402.17210.pdf,Purified and Unified Steganographic Network,"Steganography is the art of hiding secret data into the cover media for
+covert communication. In recent years, more and more deep neural network
+(DNN)-based steganographic schemes are proposed to train steganographic
+networks for secret embedding and recovery, which are shown to be promising.
+Compared with the handcrafted steganographic tools, steganographic networks
+tend to be large in size. It raises concerns on how to imperceptibly and
+effectively transmit these networks to the sender and receiver to facilitate
+the covert communication. To address this issue, we propose in this paper a
+Purified and Unified Steganographic Network (PUSNet). It performs an ordinary
+machine learning task in a purified network, which could be triggered into
+steganographic networks for secret embedding or recovery using different keys.
+We formulate the construction of the PUSNet into a sparse weight filling
+problem to flexibly switch between the purified and steganographic networks. We
+further instantiate our PUSNet as an image denoising network with two
+steganographic networks concealed for secret image embedding and recovery.
+Comprehensive experiments demonstrate that our PUSNet achieves good performance
+on secret image embedding, secret image recovery, and image denoising in a
+single architecture. It is also shown to be capable of imperceptibly carrying
+the steganographic networks in a purified network. Code is available at
+\url{https://github.com/albblgb/PUSNet}",cs.CR,"['cs.CR', 'cs.CV']"
+Synergistic Global-space Camera and Human Reconstruction from Videos,Yizhou Zhao · Tuanfeng Y. Wang · Bhiksha Raj · Min Xu · Jimei Yang · Chun-Hao P. Huang,https://paulchhuang.github.io/synchmr/,https://arxiv.org/abs/2405.14855,,2405.14855.pdf,Synergistic Global-space Camera and Human Reconstruction from Videos,"Remarkable strides have been made in reconstructing static scenes or human
+bodies from monocular videos. Yet, the two problems have largely been
+approached independently, without much synergy. Most visual SLAM methods can
+only reconstruct camera trajectories and scene structures up to scale, while
+most HMR methods reconstruct human meshes in metric scale but fall short in
+reasoning with cameras and scenes. This work introduces Synergistic Camera and
+Human Reconstruction (SynCHMR) to marry the best of both worlds. Specifically,
+we design Human-aware Metric SLAM to reconstruct metric-scale camera poses and
+scene point clouds using camera-frame HMR as a strong prior, addressing depth,
+scale, and dynamic ambiguities. Conditioning on the dense scene recovered, we
+further learn a Scene-aware SMPL Denoiser to enhance world-frame HMR by
+incorporating spatio-temporal coherency and dynamic scene constraints.
+Together, they lead to consistent reconstructions of camera trajectories, human
+meshes, and dense scene point clouds in a common world frame. Project page:
+https://paulchhuang.github.io/synchmr",cs.CV,"['cs.CV', 'cs.AI']"
+VRetouchEr: Learning Cross-frame Feature Interdependence  with Imperfection Flow for Face Retouching in Videos,Wen Xue · Le Jiang · Lianxin Xie · Si Wu · Yong Xu · Hau San Wong, ,,https://ojs.aaai.org/index.php/AAAI/article/view/28404,,,,,nan
+Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning,Xialei Liu · Jiang-Tian Zhai · Andrew Bagdanov · Ke Li · Ming-Ming Cheng, ,,https://www.youtube.com/watch?v=5VfpqIwrbWM,,,,,nan
+HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Construction,Yi ZHOU · Hui Zhang · Jiaqian Yu · yifan yang · Sangil Jung · Seung-In Park · ByungIn Yoo, ,https://arxiv.org/abs/2403.08639,,2403.08639.pdf,HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Construction,"Vectorized High-Definition (HD) map construction requires predictions of the
+category and point coordinates of map elements (e.g. road boundary, lane
+divider, pedestrian crossing, etc.). State-of-the-art methods are mainly based
+on point-level representation learning for regressing accurate point
+coordinates. However, this pipeline has limitations in obtaining element-level
+information and handling element-level failures, e.g. erroneous element shape
+or entanglement between elements. To tackle the above issues, we propose a
+simple yet effective HybrId framework named HIMap to sufficiently learn and
+interact both point-level and element-level information. Concretely, we
+introduce a hybrid representation called HIQuery to represent all map elements,
+and propose a point-element interactor to interactively extract and encode the
+hybrid information of elements, e.g. point position and element shape, into the
+HIQuery. Additionally, we present a point-element consistency constraint to
+enhance the consistency between the point-level and element-level information.
+Finally, the output point-element integrated HIQuery can be directly converted
+into map elements' class, point coordinates, and mask. We conduct extensive
+experiments and consistently outperform previous methods on both nuScenes and
+Argoverse2 datasets. Notably, our method achieves $77.8$ mAP on the nuScenes
+dataset, remarkably superior to previous SOTAs by $8.3$ mAP at least.",cs.CV,['cs.CV']
+Making Vision Transformers Truly Shift-Equivariant,Renan A. Rojas-Gomez · Teck-Yian Lim · Minh Do · Raymond A. Yeh,https://renanrojasg.github.io/shifteq_vit/,,https://www.youtube.com/watch?v=PBNdb93NqiA,,,,,nan
+Dynamic Cues-Assisted Transformer for Robust Point Cloud Registration,Hong Chen · Pei Yan · sihe xiang · Yihua Tan, ,https://arxiv.org/abs/2404.14034,,2404.14034.pdf,PointDifformer: Robust Point Cloud Registration With Neural Diffusion and Transformer,"Point cloud registration is a fundamental technique in 3-D computer vision
+with applications in graphics, autonomous driving, and robotics. However,
+registration tasks under challenging conditions, under which noise or
+perturbations are prevalent, can be difficult. We propose a robust point cloud
+registration approach that leverages graph neural partial differential
+equations (PDEs) and heat kernel signatures. Our method first uses graph neural
+PDE modules to extract high dimensional features from point clouds by
+aggregating information from the 3-D point neighborhood, thereby enhancing the
+robustness of the feature representations. Then, we incorporate heat kernel
+signatures into an attention mechanism to efficiently obtain corresponding
+keypoints. Finally, a singular value decomposition (SVD) module with learnable
+weights is used to predict the transformation between two point clouds.
+Empirical experiments on a 3-D point cloud dataset demonstrate that our
+approach not only achieves state-of-the-art performance for point cloud
+registration but also exhibits better robustness to additive noise or 3-D shape
+perturbations.",cs.CV,['cs.CV']
+Generative Multi-modal Models are Good Class Incremental Learners,Xusheng Cao · Haori Lu · Linlan Huang · Xialei Liu · Ming-Ming Cheng, ,https://arxiv.org/abs/2403.18383,,2403.18383.pdf,Generative Multi-modal Models are Good Class-Incremental Learners,"In class-incremental learning (CIL) scenarios, the phenomenon of catastrophic
+forgetting caused by the classifier's bias towards the current task has long
+posed a significant challenge. It is mainly caused by the characteristic of
+discriminative models. With the growing popularity of the generative
+multi-modal models, we would explore replacing discriminative models with
+generative ones for CIL. However, transitioning from discriminative to
+generative models requires addressing two key challenges. The primary challenge
+lies in transferring the generated textual information into the classification
+of distinct categories. Additionally, it requires formulating the task of CIL
+within a generative framework. To this end, we propose a novel generative
+multi-modal model (GMM) framework for class-incremental learning. Our approach
+directly generates labels for images using an adapted generative model. After
+obtaining the detailed text, we use a text encoder to extract text features and
+employ feature matching to determine the most similar label as the
+classification prediction. In the conventional CIL settings, we achieve
+significantly better results in long-sequence task scenarios. Under the
+Few-shot CIL setting, we have improved by at least 14\% accuracy over all the
+current state-of-the-art methods with significantly less forgetting. Our code
+is available at \url{https://github.com/DoubleClass/GMM}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models,Rongjie Li · Songyang Zhang · Dahua Lin · Kai Chen · Xuming He, ,https://arxiv.org/abs/2404.00906,,2404.00906.pdf,From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models,"Scene graph generation (SGG) aims to parse a visual scene into an
+intermediate graph representation for downstream reasoning tasks. Despite
+recent advancements, existing methods struggle to generate scene graphs with
+novel visual relation concepts. To address this challenge, we introduce a new
+open-vocabulary SGG framework based on sequence generation. Our framework
+leverages vision-language pre-trained models (VLM) by incorporating an
+image-to-graph generation paradigm. Specifically, we generate scene graph
+sequences via image-to-text generation with VLM and then construct scene graphs
+from these sequences. By doing so, we harness the strong capabilities of VLM
+for open-vocabulary SGG and seamlessly integrate explicit relational modeling
+for enhancing the VL tasks. Experimental results demonstrate that our design
+not only achieves superior performance with an open vocabulary but also
+enhances downstream vision-language task performance through explicit relation
+modeling knowledge.",cs.CV,['cs.CV']
+Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation,Shanshan Zhong · Zhongzhan Huang · Shanghua Gao · Wushao Wen · Liang Lin · Marinka Zitnik · Pan Zhou,https://zhongshsh.github.io/CLoT/,https://arxiv.org/abs/2312.02439,,2312.02439.pdf,Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation,"Chain-of-Thought (CoT) guides large language models (LLMs) to reason
+step-by-step, and can motivate their logical reasoning ability. While effective
+for logical tasks, CoT is not conducive to creative problem-solving which often
+requires out-of-box thoughts and is crucial for innovation advancements. In
+this paper, we explore the Leap-of-Thought (LoT) abilities within LLMs -- a
+non-sequential, creative paradigm involving strong associations and knowledge
+leaps. To this end, we study LLMs on the popular Oogiri game which needs
+participants to have good creativity and strong associative thinking for
+responding unexpectedly and humorously to the given image, text, or both, and
+thus is suitable for LoT study. Then to investigate LLMs' LoT ability in the
+Oogiri game, we first build a multimodal and multilingual Oogiri-GO dataset
+which contains over 130,000 samples from the Oogiri game, and observe the
+insufficient LoT ability or failures of most existing LLMs on the Oogiri game.
+Accordingly, we introduce a creative Leap-of-Thought (CLoT) paradigm to improve
+LLM's LoT ability. CLoT first formulates the Oogiri-GO dataset into
+LoT-oriented instruction tuning data to train pretrained LLM for achieving
+certain LoT humor generation and discrimination abilities. Then CLoT designs an
+explorative self-refinement that encourages the LLM to generate more creative
+LoT data via exploring parallels between seemingly unrelated concepts and
+selects high-quality data to train itself for self-refinement. CLoT not only
+excels in humor generation in the Oogiri game but also boosts creative
+abilities in various tasks like cloud guessing game and divergent association
+task. These findings advance our understanding and offer a pathway to improve
+LLMs' creative capacities for innovative applications across domains. The
+dataset, code, and models will be released online.
+https://zhongshsh.github.io/CLoT/.",cs.AI,"['cs.AI', 'cs.CL', 'cs.CV']"
+Enhancing Visual Continual Learning with Language-Guided Supervision,Bolin Ni · Hongbo Zhao · Chenghao Zhang · Ke Hu · Gaofeng Meng · Zhaoxiang Zhang · Shiming Xiang, ,https://arxiv.org/abs/2403.16124,,2403.16124.pdf,Enhancing Visual Continual Learning with Language-Guided Supervision,"Continual learning (CL) aims to empower models to learn new tasks without
+forgetting previously acquired knowledge. Most prior works concentrate on the
+techniques of architectures, replay data, regularization, \etc. However, the
+category name of each class is largely neglected. Existing methods commonly
+utilize the one-hot labels and randomly initialize the classifier head. We
+argue that the scarce semantic information conveyed by the one-hot labels
+hampers the effective knowledge transfer across tasks. In this paper, we
+revisit the role of the classifier head within the CL paradigm and replace the
+classifier with semantic knowledge from pretrained language models (PLMs).
+Specifically, we use PLMs to generate semantic targets for each class, which
+are frozen and serve as supervision signals during training. Such targets fully
+consider the semantic correlation between all classes across tasks. Empirical
+studies show that our approach mitigates forgetting by alleviating
+representation drifting and facilitating knowledge transfer across tasks. The
+proposed method is simple to implement and can seamlessly be plugged into
+existing methods with negligible adjustments. Extensive experiments based on
+eleven mainstream baselines demonstrate the effectiveness and generalizability
+of our approach to various protocols. For example, under the class-incremental
+learning setting on ImageNet-100, our method significantly improves the Top-1
+accuracy by 3.2\% to 6.1\% while reducing the forgetting rate by 2.6\% to
+13.1\%.",cs.CV,['cs.CV']
+Learned Trajectory Embedding for Subspace Clustering,Yaroslava Lochman · Christopher Zach · Carl Olsson,https://ylochman.github.io/trajectory-embedding,,https://link.springer.com/article/10.1007/s44267-024-00043-0,,,,,nan
+Denoising Point Clouds in Latent Space via Graph Convolution and Invertible Neural Network,Aihua Mao · Biao Yan · Zijing Ma · Ying He, ,https://arxiv.org/abs/2401.09721,,2401.09721.pdf,Fast graph-based denoising for point cloud color information,"Point clouds are utilized in various 3D applications such as cross-reality
+(XR) and realistic 3D displays. In some applications, e.g., for live streaming
+using a 3D point cloud, real-time point cloud denoising methods are required to
+enhance the visual quality. However, conventional high-precision denoising
+methods cannot be executed in real time for large-scale point clouds owing to
+the complexity of graph constructions with K nearest neighbors and noise level
+estimation. This paper proposes a fast graph-based denoising (FGBD) for a
+large-scale point cloud. First, high-speed graph construction is achieved by
+scanning a point cloud in various directions and searching adjacent
+neighborhoods on the scanning lines. Second, we propose a fast noise level
+estimation method using eigenvalues of the covariance matrix on a graph.
+Finally, we also propose a new low-cost filter selection method to enhance
+denoising accuracy to compensate for the degradation caused by the acceleration
+algorithms. In our experiments, we succeeded in reducing the processing time
+dramatically while maintaining accuracy relative to conventional denoising
+methods. Denoising was performed at 30fps, with frames containing approximately
+1 million points.",cs.CV,"['cs.CV', 'eess.IV', 'eess.SP']"
+LASO: Language-guided Affordance Segmentation on 3D Object,Yicong Li · Na Zhao · Junbin Xiao · Chun Feng · Xiang Wang · Tat-seng Chua, ,https://arxiv.org/abs/2309.10911,,2309.10911.pdf,Language-Conditioned Affordance-Pose Detection in 3D Point Clouds,"Affordance detection and pose estimation are of great importance in many
+robotic applications. Their combination helps the robot gain an enhanced
+manipulation capability, in which the generated pose can facilitate the
+corresponding affordance task. Previous methods for affodance-pose joint
+learning are limited to a predefined set of affordances, thus limiting the
+adaptability of robots in real-world environments. In this paper, we propose a
+new method for language-conditioned affordance-pose joint learning in 3D point
+clouds. Given a 3D point cloud object, our method detects the affordance region
+and generates appropriate 6-DoF poses for any unconstrained affordance label.
+Our method consists of an open-vocabulary affordance detection branch and a
+language-guided diffusion model that generates 6-DoF poses based on the
+affordance text. We also introduce a new high-quality dataset for the task of
+language-driven affordance-pose joint learning. Intensive experimental results
+demonstrate that our proposed method works effectively on a wide range of
+open-vocabulary affordances and outperforms other baselines by a large margin.
+In addition, we illustrate the usefulness of our method in real-world robotic
+applications. Our code and dataset are publicly available at
+https://3DAPNet.github.io",cs.RO,['cs.RO']
+MonoCD: Monocular 3D Object Detection with Complementary Depths,Longfei Yan · Pei Yan · Shengzhou Xiong · Xuanyu Xiang · Yihua Tan,https://github.com/elvintanhust/MonoCD,https://arxiv.org/abs/2404.03181v1,,2404.03181v1.pdf,MonoCD: Monocular 3D Object Detection with Complementary Depths,"Monocular 3D object detection has attracted widespread attention due to its
+potential to accurately obtain object 3D localization from a single image at a
+low cost. Depth estimation is an essential but challenging subtask of monocular
+3D object detection due to the ill-posedness of 2D to 3D mapping. Many methods
+explore multiple local depth clues such as object heights and keypoints and
+then formulate the object depth estimation as an ensemble of multiple depth
+predictions to mitigate the insufficiency of single-depth information. However,
+the errors of existing multiple depths tend to have the same sign, which
+hinders them from neutralizing each other and limits the overall accuracy of
+combined depth. To alleviate this problem, we propose to increase the
+complementarity of depths with two novel designs. First, we add a new depth
+prediction branch named complementary depth that utilizes global and efficient
+depth clues from the entire image rather than the local clues to reduce the
+correlation of depth predictions. Second, we propose to fully exploit the
+geometric relations between multiple depth clues to achieve complementarity in
+form. Benefiting from these designs, our method achieves higher
+complementarity. Experiments on the KITTI benchmark demonstrate that our method
+achieves state-of-the-art performance without introducing extra data. In
+addition, complementary depth can also be a lightweight and plug-and-play
+module to boost multiple existing monocular 3d object detectors. Code is
+available at https://github.com/elvintanhust/MonoCD.",cs.CV,['cs.CV']
+All Rivers Run to the Sea: Private Learning with Asymmetric Flows,Yue Niu · Ramy E. Ali · Saurav Prakash · Salman Avestimehr, ,https://arxiv.org/abs/2312.05264,,2312.05264.pdf,All Rivers Run to the Sea: Private Learning with Asymmetric Flows,"Data privacy is of great concern in cloud machine-learning service platforms,
+when sensitive data are exposed to service providers. While private computing
+environments (e.g., secure enclaves), and cryptographic approaches (e.g.,
+homomorphic encryption) provide strong privacy protection, their computing
+performance still falls short compared to cloud GPUs. To achieve privacy
+protection with high computing performance, we propose Delta, a new private
+training and inference framework, with comparable model performance as
+non-private centralized training. Delta features two asymmetric data flows: the
+main information-sensitive flow and the residual flow. The main part flows into
+a small model while the residuals are offloaded to a large model. Specifically,
+Delta embeds the information-sensitive representations into a low-dimensional
+space while pushing the information-insensitive part into high-dimension
+residuals. To ensure privacy protection, the low-dimensional
+information-sensitive part is secured and fed to a small model in a private
+environment. On the other hand, the residual part is sent to fast cloud GPUs,
+and processed by a large model. To further enhance privacy and reduce the
+communication cost, Delta applies a random binary quantization technique along
+with a DP-based technique to the residuals before sharing them with the public
+platform. We theoretically show that Delta guarantees differential privacy in
+the public environment and greatly reduces the complexity in the private
+environment. We conduct empirical analyses on CIFAR-10, CIFAR-100 and ImageNet
+datasets and ResNet-18 and ResNet-34, showing that Delta achieves strong
+privacy protection, fast training, and inference without significantly
+compromising the model utility.",cs.CR,"['cs.CR', 'cs.LG']"
+PH-Net: Semi-Supervised Breast Lesion Segmentation via Patch-wise Hardness,Siyao Jiang · Huisi Wu · Junyang Chen · Qin Zhang · Jing Qin, ,,https://link.springer.com/article/10.1007/s11517-023-02970-4,,,,,nan
+Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge,Haoxiang Ma · Modi Shi · Boyang GAO · Di Huang, ,https://arxiv.org/abs/2404.01727v1,,2404.01727v1.pdf,Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge,"We focus on the generalization ability of the 6-DoF grasp detection method in
+this paper. While learning-based grasp detection methods can predict grasp
+poses for unseen objects using the grasp distribution learned from the training
+set, they often exhibit a significant performance drop when encountering
+objects with diverse shapes and structures. To enhance the grasp detection
+methods' generalization ability, we incorporate domain prior knowledge of
+robotic grasping, enabling better adaptation to objects with significant shape
+and structure differences. More specifically, we employ the physical constraint
+regularization during the training phase to guide the model towards predicting
+grasps that comply with the physical rule on grasping. For the unstable grasp
+poses predicted on novel objects, we design a contact-score joint optimization
+using the projection contact map to refine these poses in cluttered scenarios.
+Extensive experiments conducted on the GraspNet-1billion benchmark demonstrate
+a substantial performance gain on the novel object set and the real-world
+grasping experiments also demonstrate the effectiveness of our generalizing
+6-DoF grasp detection method.",cs.RO,"['cs.RO', 'cs.CV']"
+Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline,Xiao Wang · Shiao Wang · Chuanming Tang · Lin Zhu · Bo Jiang · Yonghong Tian · Jin Tang, ,https://arxiv.org/abs/2309.14611,,2309.14611.pdf,Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline,"Tracking using bio-inspired event cameras has drawn more and more attention
+in recent years. Existing works either utilize aligned RGB and event data for
+accurate tracking or directly learn an event-based tracker. The first category
+needs more cost for inference and the second one may be easily influenced by
+noisy events or sparse spatial resolution. In this paper, we propose a novel
+hierarchical knowledge distillation framework that can fully utilize
+multi-modal / multi-view information during training to facilitate knowledge
+transfer, enabling us to achieve high-speed and low-latency visual tracking
+during testing by using only event signals. Specifically, a teacher
+Transformer-based multi-modal tracking framework is first trained by feeding
+the RGB frame and event stream simultaneously. Then, we design a new
+hierarchical knowledge distillation strategy which includes pairwise
+similarity, feature representation, and response maps-based knowledge
+distillation to guide the learning of the student Transformer network.
+Moreover, since existing event-based tracking datasets are all low-resolution
+($346 \times 260$), we propose the first large-scale high-resolution ($1280
+\times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide
+range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc.
+Extensive experiments on both low-resolution (FE240hz, VisEvent, COESOT), and
+our newly proposed high-resolution EventVOT dataset fully validated the
+effectiveness of our proposed method. The dataset, evaluation toolkit, and
+source code are available on
+\url{https://github.com/Event-AHU/EventVOT_Benchmark}",cs.CV,"['cs.CV', 'cs.NE']"
+An Empirical Study of Scaling Law for Scene Text Recognition,Miao Rang · Zhenni Bi · Chuanjian Liu · Yunhe Wang · Kai Han, ,https://arxiv.org/abs/2401.00028,,2401.00028.pdf,An Empirical Study of Scaling Law for OCR,"The laws of model size, data volume, computation and model performance have
+been extensively studied in the field of Natural Language Processing (NLP).
+However, the scaling laws in Optical Character Recognition (OCR) have not yet
+been investigated. To address this, we conducted comprehensive studies that
+involved examining the correlation between performance and the scale of models,
+data volume and computation in the field of text recognition.Conclusively, the
+study demonstrates smooth power laws between performance and model size, as
+well as training data volume, when other influencing factors are held constant.
+Additionally, we have constructed a large-scale dataset called REBU-Syn, which
+comprises 6 million real samples and 18 million synthetic samples. Based on our
+scaling law and new dataset, we have successfully trained a scene text
+recognition model, achieving a new state-ofthe-art on 6 common test benchmarks
+with a top-1 average accuracy of 97.42%. The models and dataset are publicly
+available at https://github.com/large-ocr-model/large-ocr-model.github.io.",cs.CV,['cs.CV']
+Dual-scale Transformer for Large-scale Single-Pixel Imaging,Gang Qu · Ping Wang · Xin Yuan, ,https://arxiv.org/abs/2404.05001,,2404.05001.pdf,Dual-Scale Transformer for Large-Scale Single-Pixel Imaging,"Single-pixel imaging (SPI) is a potential computational imaging technique
+which produces image by solving an illposed reconstruction problem from few
+measurements captured by a single-pixel detector. Deep learning has achieved
+impressive success on SPI reconstruction. However, previous poor reconstruction
+performance and impractical imaging model limit its real-world applications. In
+this paper, we propose a deep unfolding network with hybrid-attention
+Transformer on Kronecker SPI model, dubbed HATNet, to improve the imaging
+quality of real SPI cameras. Specifically, we unfold the computation graph of
+the iterative shrinkagethresholding algorithm (ISTA) into two alternative
+modules: efficient tensor gradient descent and hybrid-attention multiscale
+denoising. By virtue of Kronecker SPI, the gradient descent module can avoid
+high computational overheads rooted in previous gradient descent modules based
+on vectorized SPI. The denoising module is an encoder-decoder architecture
+powered by dual-scale spatial attention for high- and low-frequency aggregation
+and channel attention for global information recalibration. Moreover, we build
+a SPI prototype to verify the effectiveness of the proposed method. Extensive
+experiments on synthetic and real data demonstrate that our method achieves the
+state-of-the-art performance. The source code and pre-trained models are
+available at https://github.com/Gang-Qu/HATNet-SPI.",cs.CV,['cs.CV']
+Learning Intra-view and Cross-view Geometric Knowledge for Stereo Matching,Rui Gong · Weide Liu · ZAIWANG GU · Xulei Yang · Jun Cheng, ,https://arxiv.org/abs/2402.19270,,2402.19270.pdf,Learning Intra-view and Cross-view Geometric Knowledge for Stereo Matching,"Geometric knowledge has been shown to be beneficial for the stereo matching
+task. However, prior attempts to integrate geometric insights into stereo
+matching algorithms have largely focused on geometric knowledge from single
+images while crucial cross-view factors such as occlusion and matching
+uniqueness have been overlooked. To address this gap, we propose a novel
+Intra-view and Cross-view Geometric knowledge learning Network (ICGNet),
+specifically crafted to assimilate both intra-view and cross-view geometric
+knowledge. ICGNet harnesses the power of interest points to serve as a channel
+for intra-view geometric understanding. Simultaneously, it employs the
+correspondences among these points to capture cross-view geometric
+relationships. This dual incorporation empowers the proposed ICGNet to leverage
+both intra-view and cross-view geometric knowledge in its learning process,
+substantially improving its ability to estimate disparities. Our extensive
+experiments demonstrate the superiority of the ICGNet over contemporary leading
+models.",cs.CV,['cs.CV']
+ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks,Kai Han · Yunhe Wang · Jianyuan Guo · Enhua Wu,https://parameternet.github.io/,https://arxiv.org/abs/2306.14525,,2306.14525.pdf,ParameterNet: Parameters Are All You Need,"The large-scale visual pretraining has significantly improve the performance
+of large vision models. However, we observe the \emph{low FLOPs pitfall} that
+the existing low-FLOPs models cannot benefit from large-scale pretraining. In
+this paper, we introduce a novel design principle, termed ParameterNet, aimed
+at augmenting the number of parameters in large-scale visual pretraining models
+while minimizing the increase in FLOPs. We leverage dynamic convolutions to
+incorporate additional parameters into the networks with only a marginal rise
+in FLOPs. The ParameterNet approach allows low-FLOPs networks to take advantage
+of large-scale visual pretraining. Furthermore, we extend the ParameterNet
+concept to the language domain to enhance inference results while preserving
+inference speed. Experiments on the large-scale ImageNet-22K have shown the
+superiority of our ParameterNet scheme. For example, ParameterNet-600M can
+achieve higher accuracy on ImageNet than the widely-used Swin Transformer
+(81.6\% \emph{vs.} 80.9\%) and has much lower FLOPs (0.6G \emph{vs.} 4.5G). In
+the language domain, LLaMA-1B enhanced with ParameterNet achieves 2\% higher
+accuracy over vanilla LLaMA. The code will be released at
+\url{https://parameternet.github.io/}.",cs.CV,['cs.CV']
+Relational Matching for Weakly Semi-Supervised Oriented Object Detection,Wenhao Wu · Hau San Wong · Si Wu · Tianyou Zhang, ,,https://paperswithcode.com/paper/weakly-semi-supervised-object-detection-in,,,,,nan
+A2XP: Towards Private Domain Generalization,Geunhyeok Yu · Hyoseok Hwang,https://airlabkhu.github.io/A2XP/,https://arxiv.org/abs/2311.10339,,2311.10339.pdf,A2XP: Towards Private Domain Generalization,"Deep Neural Networks (DNNs) have become pivotal in various fields, especially
+in computer vision, outperforming previous methodologies. A critical challenge
+in their deployment is the bias inherent in data across different domains, such
+as image style and environmental conditions, leading to domain gaps. This
+necessitates techniques for learning general representations from biased
+training data, known as domain generalization. This paper presents Attend to
+eXpert Prompts (A2XP), a novel approach for domain generalization that
+preserves the privacy and integrity of the network architecture. A2XP consists
+of two phases: Expert Adaptation and Domain Generalization. In the first phase,
+prompts for each source domain are optimized to guide the model towards the
+optimal direction. In the second phase, two embedder networks are trained to
+effectively amalgamate these expert prompts, aiming for an optimal output. Our
+extensive experiments demonstrate that A2XP achieves state-of-the-art results
+over existing non-private domain generalization methods. The experimental
+results validate that the proposed approach not only tackles the domain
+generalization challenge in DNNs but also offers a privacy-preserving,
+efficient solution to the broader field of computer vision.",cs.CV,['cs.CV']
+Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding,Hoang-Quan Nguyen · Thanh-Dat Truong · Xuan-Bac Nguyen · Ashley Dowling · Xin Li · Khoa Luu,https://uark-cviu.github.io/projects/insect_foundation.html,https://arxiv.org/abs/2311.15206,,2311.15206.pdf,Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding,"In precision agriculture, the detection and recognition of insects play an
+essential role in the ability of crops to grow healthy and produce a
+high-quality yield. The current machine vision model requires a large volume of
+data to achieve high performance. However, there are approximately 5.5 million
+different insect species in the world. None of the existing insect datasets can
+cover even a fraction of them due to varying geographic locations and
+acquisition costs. In this paper, we introduce a novel ""Insect-1M"" dataset, a
+game-changing resource poised to revolutionize insect-related foundation model
+training. Covering a vast spectrum of insect species, our dataset, including 1
+million images with dense identification labels of taxonomy hierarchy and
+insect descriptions, offers a panoramic view of entomology, enabling foundation
+models to comprehend visual and semantic information about insects like never
+before. Then, to efficiently establish an Insect Foundation Model, we develop a
+micro-feature self-supervised learning method with a Patch-wise Relevant
+Attention mechanism capable of discerning the subtle differences among insect
+images. In addition, we introduce Description Consistency loss to improve
+micro-feature modeling via insect descriptions. Through our experiments, we
+illustrate the effectiveness of our proposed approach in insect modeling and
+achieve State-of-the-Art performance on standard benchmarks of insect-related
+tasks. Our Insect Foundation Model and Dataset promise to empower the next
+generation of insect-related vision models, bringing them closer to the
+ultimate goal of precision agriculture.",cs.CV,['cs.CV']
+PostureHMR: Posture Transformation for 3D Human Mesh Recovery,Yu-Pei Song · Xiao WU · Zhaoquan Yuan · Jian-Jun Qiao · Qiang Peng, ,https://arxiv.org/abs/2403.12473,,2403.12473.pdf,PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust 3D Human Mesh Recovery,"With the recent advancements in single-image-based human mesh recovery, there
+is a growing interest in enhancing its performance in certain extreme
+scenarios, such as occlusion, while maintaining overall model accuracy.
+Although obtaining accurately annotated 3D human poses under occlusion is
+challenging, there is still a wealth of rich and precise 2D pose annotations
+that can be leveraged. However, existing works mostly focus on directly
+leveraging 2D pose coordinates to estimate 3D pose and mesh. In this paper, we
+present PostoMETRO($\textbf{Pos}$e $\textbf{to}$ken enhanced $\textbf{ME}$sh
+$\textbf{TR}$ansf$\textbf{O}$rmer), which integrates occlusion-resilient 2D
+pose representation into transformers in a token-wise manner. Utilizing a
+specialized pose tokenizer, we efficiently condense 2D pose data to a compact
+sequence of pose tokens and feed them to the transformer together with the
+image tokens. This process not only ensures a rich depiction of texture from
+the image but also fosters a robust integration of pose and image information.
+Subsequently, these combined tokens are queried by vertex and joint tokens to
+decode 3D coordinates of mesh vertices and human joints. Facilitated by the
+robust pose token representation and the effective combination, we are able to
+produce more precise 3D coordinates, even under extreme scenarios like
+occlusion. Experiments on both standard and occlusion-specific benchmarks
+demonstrate the effectiveness of PostoMETRO. Qualitative results further
+illustrate the clarity of how 2D pose can help 3D reconstruction. Code will be
+made available.",cs.CV,['cs.CV']
+InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning,Jing Shi · Wei Xiong · Zhe Lin · HyunJoon Jung, ,https://arxiv.org/html/2403.11284v1,,2403.11284v1.pdf,Fast Personalized Text-to-Image Syntheses With Attention Injection,"Currently, personalized image generation methods mostly require considerable
+time to finetune and often overfit the concept resulting in generated images
+that are similar to custom concepts but difficult to edit by prompts. We
+propose an effective and fast approach that could balance the text-image
+consistency and identity consistency of the generated image and reference
+image. Our method can generate personalized images without any fine-tuning
+while maintaining the inherent text-to-image generation ability of diffusion
+models. Given a prompt and a reference image, we merge the custom concept into
+generated images by manipulating cross-attention and self-attention layers of
+the original diffusion model to generate personalized images that match the
+text description. Comprehensive experiments highlight the superiority of our
+method.",cs.CV,['cs.CV']
+Exact Fusion via Feature Distribution Matching for Few-shot Image Generation,Yingbo Zhou · Yutong Ye · Pengyu Zhang · Xian Wei · Mingsong Chen, ,https://arxiv.org/abs/2307.14638v1,,2307.14638v1.pdf,EqGAN: Feature Equalization Fusion for Few-shot Image Generation,"Due to the absence of fine structure and texture information, existing
+fusion-based few-shot image generation methods suffer from unsatisfactory
+generation quality and diversity. To address this problem, we propose a novel
+feature Equalization fusion Generative Adversarial Network (EqGAN) for few-shot
+image generation. Unlike existing fusion strategies that rely on either deep
+features or local representations, we design two separate branches to fuse
+structures and textures by disentangling encoded features into shallow and deep
+contents. To refine image contents at all feature levels, we equalize the fused
+structure and texture semantics at different scales and supplement the decoder
+with richer information by skip connections. Since the fused structures and
+textures may be inconsistent with each other, we devise a consistent
+equalization loss between the equalized features and the intermediate output of
+the decoder to further align the semantics. Comprehensive experiments on three
+public datasets demonstrate that, EqGAN not only significantly improves
+generation performance with FID score (by up to 32.7%) and LPIPS score (by up
+to 4.19%), but also outperforms the state-of-the-arts in terms of accuracy (by
+up to 1.97%) for downstream classification tasks.",cs.CV,['cs.CV']
+Data Poisoning based Backdoor Attacks to Contrastive Learning,Jinghuai Zhang · Hongbin Liu · Jinyuan Jia · Neil Zhenqiang Gong,https://github.com/jzhang538/CorruptEncoder,,,,,,,nan
+GenesisTex: Adapting Image Denoising Diffusion to Texture Space,Chenjian Gao · Boyan Jiang · Xinghui Li · YingPeng Zhang · Qian Yu,https://cjeen.github.io/GenesisTexPaper/,https://arxiv.org/abs/2403.17782,,2403.17782.pdf,GenesisTex: Adapting Image Denoising Diffusion to Texture Space,"We present GenesisTex, a novel method for synthesizing textures for 3D
+geometries from text descriptions. GenesisTex adapts the pretrained image
+diffusion model to texture space by texture space sampling. Specifically, we
+maintain a latent texture map for each viewpoint, which is updated with
+predicted noise on the rendering of the corresponding viewpoint. The sampled
+latent texture maps are then decoded into a final texture map. During the
+sampling process, we focus on both global and local consistency across multiple
+viewpoints: global consistency is achieved through the integration of style
+consistency mechanisms within the noise prediction network, and low-level
+consistency is achieved by dynamically aligning latent textures. Finally, we
+apply reference-based inpainting and img2img on denser views for texture
+refinement. Our approach overcomes the limitations of slow optimization in
+distillation-based methods and instability in inpainting-based methods.
+Experiments on meshes from various sources demonstrate that our method
+surpasses the baseline methods quantitatively and qualitatively.",cs.CV,"['cs.CV', 'cs.GR']"
+On Scaling up a Multilingual Vision and Language Model,Xi Chen · Josip Djolonga · Piotr Padlewski · Basil Mustafa · Soravit Changpinyo · Jialin Wu · Carlos Riquelme Ruiz · Sebastian Goodman · Xiao Wang · Yi Tay · Siamak Shakeri · Mostafa Dehghani · Daniel Salz · Mario Lučić · Michael Tschannen · Arsha Nagrani · Hexiang Hu · Mandar Joshi · Bo Pang · Ceslee Montgomery · Paulina Pietrzyk · Marvin Ritter · AJ Piergiovanni · Matthias Minderer · Filip Pavetic · Austin Waters · Gang Li · Ibrahim Alabdulmohsin · Lucas Beyer · Julien Amelot · Kenton Lee · Andreas Steiner · Yang Li · Daniel Keysers · Anurag Arnab · Yuanzhong Xu · Keran Rong · Alexander Kolesnikov · Mojtaba Seyedhosseini · Anelia Angelova · Xiaohua Zhai · Neil Houlsby · Radu Soricut, ,https://ar5iv.labs.arxiv.org/html/2312.07533,,2312.07533.pdf,VILA: On Pre-training for Visual Language Models,"Visual language models (VLMs) rapidly progressed with the recent success of
+large language models. There have been growing efforts on visual instruction
+tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
+visual language pre-training process, where the model learns to perform joint
+modeling on both modalities. In this work, we examine the design options for
+VLM pre-training by augmenting LLM towards VLM through step-by-step
+controllable comparisons. We introduce three main findings: (1) freezing LLMs
+during pre-training can achieve decent zero-shot performance, but lack
+in-context learning capability, which requires unfreezing the LLM; (2)
+interleaved pre-training data is beneficial whereas image-text pairs alone are
+not optimal; (3) re-blending text-only instruction data to image-text data
+during instruction fine-tuning not only remedies the degradation of text-only
+tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
+we build VILA, a Visual Language model family that consistently outperforms the
+state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
+and whistles. Multi-modal pre-training also helps unveil appealing properties
+of VILA, including multi-image reasoning, enhanced in-context learning, and
+better world knowledge.",cs.CV,['cs.CV']
+$V_kD:$ Improving knowledge distillation using orthogonal projections,Roy Miles · Ismail Elezi · Jiankang Deng, ,https://arxiv.org/abs/2403.06213,,2403.06213.pdf,$V_kD:$ Improving Knowledge Distillation using Orthogonal Projections,"Knowledge distillation is an effective method for training small and
+efficient deep learning models. However, the efficacy of a single method can
+degenerate when transferring to other tasks, modalities, or even other
+architectures. To address this limitation, we propose a novel constrained
+feature distillation method. This method is derived from a small set of core
+principles, which results in two emerging components: an orthogonal projection
+and a task-specific normalisation. Equipped with both of these components, our
+transformer models can outperform all previous methods on ImageNet and reach up
+to a 4.4% relative improvement over the previous state-of-the-art methods. To
+further demonstrate the generality of our method, we apply it to object
+detection and image generation, whereby we obtain consistent and substantial
+performance improvements over state-of-the-art. Code and models are publicly
+available: https://github.com/roymiles/vkd",cs.CV,"['cs.CV', 'cs.AI']"
+Towards Modern Image Manipulation Localization: A Large-Scale Dataset and Novel Methods,Chenfan Qu · Yiwu Zhong · Chongyu Liu · Guitao Xu · Dezhi Peng · Fengjun Guo · Lianwen Jin, ,https://arxiv.org/abs/2309.01858,,2309.01858.pdf,Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations,"Fine-grained and instance-level recognition methods are commonly trained and
+evaluated on specific domains, in a model per domain scenario. Such an
+approach, however, is impractical in real large-scale applications. In this
+work, we address the problem of universal image embedding, where a single
+universal model is trained and used in multiple domains. First, we leverage
+existing domain-specific datasets to carefully construct a new large-scale
+public benchmark for the evaluation of universal image embeddings, with 241k
+query images, 1.4M index images and 2.8M training images across 8 different
+domains and 349k classes. We define suitable metrics, training and evaluation
+protocols to foster future research in this area. Second, we provide a
+comprehensive experimental evaluation on the new dataset, demonstrating that
+existing approaches and simplistic extensions lead to worse performance than an
+assembly of models trained for each domain separately. Finally, we conducted a
+public research competition on this topic, leveraging industrial datasets,
+which attracted the participation of more than 1k teams worldwide. This
+exercise generated many interesting research ideas and findings which we
+present in detail. Project webpage: https://cmp.felk.cvut.cz/univ_emb/",cs.CV,['cs.CV']
+Permutation Equivariance of Transformers and Its Applications,Hengyuan Xu · Liyao Xiang · Hangyu Ye · Dixi Yao · Pengzhi Chu · Baochun Li,https://github.com/Doby-Xu/ST,https://arxiv.org/abs/2403.05842,,2403.05842.pdf,Hufu: A Modality-Agnositc Watermarking System for Pre-Trained Transformers via Permutation Equivariance,"With the blossom of deep learning models and services, it has become an
+imperative concern to safeguard the valuable model parameters from being
+stolen. Watermarking is considered an important tool for ownership
+verification. However, current watermarking schemes are customized for
+different models and tasks, hard to be integrated as an integrated intellectual
+protection service. We propose Hufu, a modality-agnostic watermarking system
+for pre-trained Transformer-based models, relying on the permutation
+equivariance property of Transformers. Hufu embeds watermark by fine-tuning the
+pre-trained model on a set of data samples specifically permuted, and the
+embedded model essentially contains two sets of weights -- one for normal use
+and the other for watermark extraction which is triggered on permuted inputs.
+The permutation equivariance ensures minimal interference between these two
+sets of model weights and thus high fidelity on downstream tasks. Since our
+method only depends on the model itself, it is naturally modality-agnostic,
+task-independent, and trigger-sample-free. Extensive experiments on the
+state-of-the-art vision Transformers, BERT, and GPT2 have demonstrated Hufu's
+superiority in meeting watermarking requirements including effectiveness,
+efficiency, fidelity, and robustness, showing its great potential to be
+deployed as a uniform ownership verification service for various Transformers.",cs.CR,"['cs.CR', 'cs.AI']"
+CLOAF: CoLlisiOn-Aware Human Flow,Andrey Davydov · Martin Engilberge · Mathieu Salzmann · Pascal Fua,https://arxiv.org/abs/2403.09050,https://arxiv.org/abs/2403.09050,,2403.09050.pdf,CLOAF: CoLlisiOn-Aware Human Flow,"Even the best current algorithms for estimating body 3D shape and pose yield
+results that include body self-intersections. In this paper, we present CLOAF,
+which exploits the diffeomorphic nature of Ordinary Differential Equations to
+eliminate such self-intersections while still imposing body shape constraints.
+We show that, unlike earlier approaches to addressing this issue, ours
+completely eliminates the self-intersections without compromising the accuracy
+of the reconstructions. Being differentiable, CLOAF can be used to fine-tune
+pose and shape estimation baselines to improve their overall performance and
+eliminate self-intersections in their predictions. Furthermore, we demonstrate
+how our CLOAF strategy can be applied to practically any motion field induced
+by the user. CLOAF also makes it possible to edit motion to interact with the
+environment without worrying about potential collision or loss of body-shape
+prior.",cs.CV,['cs.CV']
+A Physics-informed Low-rank Deep Neural Network for Blind and Universal Lens Aberration Correction,Jin Gong · Runzhao Yang · Weihang Zhang · Jinli Suo · Qionghai Dai, ,https://arxiv.org/abs/2310.09528,,2310.09528.pdf,Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks,"In various engineering and applied science applications, repetitive numerical
+simulations of partial differential equations (PDEs) for varying input
+parameters are often required (e.g., aircraft shape optimization over many
+design parameters) and solvers are required to perform rapid execution. In this
+study, we suggest a path that potentially opens up a possibility for
+physics-informed neural networks (PINNs), emerging deep-learning-based solvers,
+to be considered as one such solver. Although PINNs have pioneered a proper
+integration of deep-learning and scientific computing, they require repetitive
+time-consuming training of neural networks, which is not suitable for
+many-query scenarios. To address this issue, we propose a lightweight low-rank
+PINNs containing only hundreds of model parameters and an associated
+hypernetwork-based meta-learning algorithm, which allows efficient
+approximation of solutions of PDEs for varying ranges of PDE input parameters.
+Moreover, we show that the proposed method is effective in overcoming a
+challenging issue, known as ""failure modes"" of PINNs.",cs.LG,"['cs.LG', 'cs.NA', 'math.NA', 'physics.comp-ph']"
+Pre-training Vision Models with Mandelbulb Variations,Benjamin N. Chiche · Yuto Horikawa · Ryo Fujita, ,https://arxiv.org/abs/2403.03346,,2403.03346.pdf,Enhancing Vision-Language Pre-training with Rich Supervisions,"We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
+pre-training paradigm for Vision-Language Models using data from large-scale
+web screenshot rendering. Using web screenshots unlocks a treasure trove of
+visual and textual cues that are not present in using image-text pairs. In S4,
+we leverage the inherent tree-structured hierarchy of HTML elements and the
+spatial localization to carefully design 10 pre-training tasks with large scale
+annotated data. These tasks resemble downstream tasks across different domains
+and the annotations are cheap to obtain. We demonstrate that, compared to
+current screenshot pre-training objectives, our innovative pre-training method
+significantly enhances performance of image-to-text model in nine varied and
+popular downstream tasks - up to 76.1% improvements on Table Detection, and at
+least 1% on Widget Captioning.",cs.CV,['cs.CV']
+Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following,Yutong Feng · Biao Gong · Di Chen · Yujun Shen · Yu Liu · Jingren Zhou, ,https://arxiv.org/abs/2311.17002,,2311.17002.pdf,Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following,"Existing text-to-image (T2I) diffusion models usually struggle in
+interpreting complex prompts, especially those with quantity, object-attribute
+binding, and multi-subject descriptions. In this work, we introduce a semantic
+panel as the middleware in decoding texts to images, supporting the generator
+to better follow instructions. The panel is obtained through arranging the
+visual concepts parsed from the input text by the aid of large language models,
+and then injected into the denoising network as a detailed control signal to
+complement the text condition. To facilitate text-to-panel learning, we come up
+with a carefully designed semantic formatting protocol, accompanied by a
+fully-automatic data preparation pipeline. Thanks to such a design, our
+approach, which we call Ranni, manages to enhance a pre-trained T2I generator
+regarding its textual controllability. More importantly, the introduction of
+the generative middleware brings a more convenient form of interaction (i.e.,
+directly adjusting the elements in the panel or using language instructions)
+and further allows users to finely customize their generation, based on which
+we develop a practical system and showcase its potential in continuous
+generation and chatting-based editing. Our project page is at
+https://ranni-t2i.github.io/Ranni.",cs.CV,['cs.CV']
+Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay,Yuhang Zhou · Zhongyun Hua, ,https://arxiv.org/abs/2404.01828,,2404.01828.pdf,Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay,"Deep neural networks have demonstrated susceptibility to adversarial attacks.
+Adversarial defense techniques often focus on one-shot setting to maintain
+robustness against attack. However, new attacks can emerge in sequences in
+real-world deployment scenarios. As a result, it is crucial for a defense model
+to constantly adapt to new attacks, but the adaptation process can lead to
+catastrophic forgetting of previously defended against attacks. In this paper,
+we discuss for the first time the concept of continual adversarial defense
+under a sequence of attacks, and propose a lifelong defense baseline called
+Anisotropic \& Isotropic Replay (AIR), which offers three advantages: (1)
+Isotropic replay ensures model consistency in the neighborhood distribution of
+new data, indirectly aligning the output preference between old and new tasks.
+(2) Anisotropic replay enables the model to learn a compromise data manifold
+with fresh mixed semantics for further replay constraints and potential future
+attacks. (3) A straightforward regularizer mitigates the 'plasticity-stability'
+trade-off by aligning model output between new and old tasks. Experiment
+results demonstrate that AIR can approximate or even exceed the empirical
+performance upper bounds achieved by Joint Training.",cs.LG,"['cs.LG', 'cs.AI']"
+CDFormer: When Degradation Prediction Embraces Diffusion Model for Blind Image Super-Resolution,Qingguo Liu · Chenyi Zhuang · Pan Gao · Jie Qin, ,https://arxiv.org/abs/2405.07648,,2405.07648.pdf,CDFormer:When Degradation Prediction Embraces Diffusion Model for Blind Image Super-Resolution,"Existing Blind image Super-Resolution (BSR) methods focus on estimating
+either kernel or degradation information, but have long overlooked the
+essential content details. In this paper, we propose a novel BSR approach,
+Content-aware Degradation-driven Transformer (CDFormer), to capture both
+degradation and content representations. However, low-resolution images cannot
+provide enough content details, and thus we introduce a diffusion-based module
+$CDFormer_{diff}$ to first learn Content Degradation Prior (CDP) in both low-
+and high-resolution images, and then approximate the real distribution given
+only low-resolution information. Moreover, we apply an adaptive SR network
+$CDFormer_{SR}$ that effectively utilizes CDP to refine features. Compared to
+previous diffusion-based SR methods, we treat the diffusion model as an
+estimator that can overcome the limitations of expensive sampling time and
+excessive diversity. Experiments show that CDFormer can outperform existing
+methods, establishing a new state-of-the-art performance on various benchmarks
+under blind settings. Codes and models will be available at
+\href{https://github.com/I2-Multimedia-Lab/CDFormer}{https://github.com/I2-Multimedia-Lab/CDFormer}.",cs.CV,"['cs.CV', 'eess.IV']"
+Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration,Tony C. W. MOK · Zi Li · Yunhao Bai · Jianpeng Zhang · Wei Liu · Yan-Jie Zhou · Ke Yan · Dakai Jin · Yu Shi · Xiaoli Yin · Le Lu · Ling Zhang, ,https://arxiv.org/abs/2402.18933,,2402.18933.pdf,Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration,"Establishing dense anatomical correspondence across distinct imaging
+modalities is a foundational yet challenging procedure for numerous medical
+image analysis studies and image-guided radiotherapy. Existing multi-modality
+image registration algorithms rely on statistical-based similarity measures or
+local structural image representations. However, the former is sensitive to
+locally varying noise, while the latter is not discriminative enough to cope
+with complex anatomical structures in multimodal scans, causing ambiguity in
+determining the anatomical correspondence across scans with different
+modalities. In this paper, we propose a modality-agnostic structural
+representation learning method, which leverages Deep Neighbourhood
+Self-similarity (DNS) and anatomy-aware contrastive learning to learn
+discriminative and contrast-invariance deep structural image representations
+(DSIR) without the need for anatomical delineations or pre-aligned training
+images. We evaluate our method on multiphase CT, abdomen MR-CT, and brain MR
+T1w-T2w registration. Comprehensive results demonstrate that our method is
+superior to the conventional local structural representation and
+statistical-based similarity measures in terms of discriminability and
+accuracy.",cs.CV,['cs.CV']
+Towards Accurate and Robust Architectures via Neural Architecture Search,Yuwei Ou · Yuqi Feng · Yanan Sun, ,https://arxiv.org/abs/2405.05502,,2405.05502.pdf,Towards Accurate and Robust Architectures via Neural Architecture Search,"To defend deep neural networks from adversarial attacks, adversarial training
+has been drawing increasing attention for its effectiveness. However, the
+accuracy and robustness resulting from the adversarial training are limited by
+the architecture, because adversarial training improves accuracy and robustness
+by adjusting the weight connection affiliated to the architecture. In this
+work, we propose ARNAS to search for accurate and robust architectures for
+adversarial training. First we design an accurate and robust search space, in
+which the placement of the cells and the proportional relationship of the
+filter numbers are carefully determined. With the design, the architectures can
+obtain both accuracy and robustness by deploying accurate and robust structures
+to their sensitive positions, respectively. Then we propose a differentiable
+multi-objective search strategy, performing gradient descent towards directions
+that are beneficial for both natural loss and adversarial loss, thus the
+accuracy and robustness can be guaranteed at the same time. We conduct
+comprehensive experiments in terms of white-box attacks, black-box attacks, and
+transferability. Experimental results show that the searched architecture has
+the strongest robustness with the competitive accuracy, and breaks the
+traditional idea that NAS-based architectures cannot transfer well to complex
+tasks in robustness scenarios. By analyzing outstanding architectures searched,
+we also conclude that accurate and robust neural architectures tend to deploy
+different structures near the input and output, which has great practical
+significance on both hand-crafting and automatically designing of accurate and
+robust architectures.",cs.CV,"['cs.CV', 'cs.CR', 'cs.LG']"
+Fast Adaptation for Human Pose Estimation via Meta-Optimization,Shengxiang Hu · Huaijiang Sun · Bin Li · Dong Wei · Weiqing Li · Jianfeng Lu, ,https://arxiv.org/abs/2405.05216,,2405.05216.pdf,FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models,"The 3D Human Pose Estimation (3D HPE) task uses 2D images or videos to
+predict human joint coordinates in 3D space. Despite recent advancements in
+deep learning-based methods, they mostly ignore the capability of coupling
+accessible texts and naturally feasible knowledge of humans, missing out on
+valuable implicit supervision to guide the 3D HPE task. Moreover, previous
+efforts often study this task from the perspective of the whole human body,
+neglecting fine-grained guidance hidden in different body parts. To this end,
+we present a new Fine-Grained Prompt-Driven Denoiser based on a diffusion model
+for 3D HPE, named \textbf{FinePOSE}. It consists of three core blocks enhancing
+the reverse process of the diffusion model: (1) Fine-grained Part-aware Prompt
+learning (FPP) block constructs fine-grained part-aware prompts via coupling
+accessible texts and naturally feasible knowledge of body parts with learnable
+prompts to model implicit guidance. (2) Fine-grained Prompt-pose Communication
+(FPC) block establishes fine-grained communications between learned part-aware
+prompts and poses to improve the denoising quality. (3) Prompt-driven Timestamp
+Stylization (PTS) block integrates learned prompt embedding and temporal
+information related to the noise level to enable adaptive adjustment at each
+denoising step. Extensive experiments on public single-human pose estimation
+datasets show that FinePOSE outperforms state-of-the-art methods. We further
+extend FinePOSE to multi-human pose estimation. Achieving 34.3mm average MPJPE
+on the EgoHumans dataset demonstrates the potential of FinePOSE to deal with
+complex multi-human scenarios. Code is available at
+https://github.com/PKU-ICST-MIPL/FinePOSE_CVPR2024.",cs.CV,['cs.CV']
+PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation,Ruining Deng · Quan Liu · Can Cui · Tianyuan Yao · Jialin Yue · Juming Xiong · Lining yu · Yifei Wu · Mengmeng Yin · Yu Wang · Shilin Zhao · Yucheng Tang · Haichun Yang · Yuankai Huo, ,https://arxiv.org/abs/2402.19286,,2402.19286.pdf,PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation,"Understanding the anatomy of renal pathology is crucial for advancing disease
+diagnostics, treatment evaluation, and clinical research. The complex kidney
+system comprises various components across multiple levels, including regions
+(cortex, medulla), functional units (glomeruli, tubules), and cells (podocytes,
+mesangial cells in glomerulus). Prior studies have predominantly overlooked the
+intricate spatial interrelations among objects from clinical knowledge. In this
+research, we introduce a novel universal proposition learning approach, called
+panoramic renal pathology segmentation (PrPSeg), designed to segment
+comprehensively panoramic structures within kidney by integrating extensive
+knowledge of kidney anatomy.
+  In this paper, we propose (1) the design of a comprehensive universal
+proposition matrix for renal pathology, facilitating the incorporation of
+classification and spatial relationships into the segmentation process; (2) a
+token-based dynamic head single network architecture, with the improvement of
+the partial label image segmentation and capability for future data
+enlargement; and (3) an anatomy loss function, quantifying the inter-object
+relationships across the kidney.",eess.IV,"['eess.IV', 'cs.CV']"
+Analyzing and Improving the Training Dynamics of Diffusion Models,Tero Karras · Miika Aittala · Jaakko Lehtinen · Janne Hellsten · Timo Aila · Samuli Laine, ,https://arxiv.org/abs/2312.02696,,2312.02696.pdf,Analyzing and Improving the Training Dynamics of Diffusion Models,"Diffusion models currently dominate the field of data-driven image synthesis
+with their unparalleled scaling to large datasets. In this paper, we identify
+and rectify several causes for uneven and ineffective training in the popular
+ADM diffusion model architecture, without altering its high-level structure.
+Observing uncontrolled magnitude changes and imbalances in both the network
+activations and weights over the course of training, we redesign the network
+layers to preserve activation, weight, and update magnitudes on expectation. We
+find that systematic application of this philosophy eliminates the observed
+drifts and imbalances, resulting in considerably better networks at equal
+computational complexity. Our modifications improve the previous record FID of
+2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
+sampling.
+  As an independent contribution, we present a method for setting the
+exponential moving average (EMA) parameters post-hoc, i.e., after completing
+the training run. This allows precise tuning of EMA length without the cost of
+performing several training runs, and reveals its surprising interactions with
+network architecture, training time, and guidance.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.NE', 'stat.ML']"
+POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning,Jiayi Guan · Li Shen · Ao Zhou · Lusong Li · Han Hu · Xiaodong He · Guang Chen · Changjun Jiang, ,https://arxiv.org/abs/2401.14758,,2401.14758.pdf,Off-Policy Primal-Dual Safe Reinforcement Learning,"Primal-dual safe RL methods commonly perform iterations between the primal
+update of the policy and the dual update of the Lagrange Multiplier. Such a
+training paradigm is highly susceptible to the error in cumulative cost
+estimation since this estimation serves as the key bond connecting the primal
+and dual update processes. We show that this problem causes significant
+underestimation of cost when using off-policy methods, leading to the failure
+to satisfy the safety constraint. To address this issue, we propose
+conservative policy optimization, which learns a policy in a
+constraint-satisfying area by considering the uncertainty in cost estimation.
+This improves constraint satisfaction but also potentially hinders reward
+maximization. We then introduce local policy convexification to help eliminate
+such suboptimality by gradually reducing the estimation uncertainty. We provide
+theoretical interpretations of the joint coupling effect of these two
+ingredients and further verify them by extensive experiments. Results on
+benchmark tasks show that our method not only achieves an asymptotic
+performance comparable to state-of-the-art on-policy methods while using much
+fewer samples, but also significantly reduces constraint violation during
+training. Our code is available at https://github.com/ZifanWu/CAL.",cs.LG,['cs.LG']
+CAMEL: CAusal Motion Enhancement tailored for Lifting Text-driven Video Editing,Guiwei Zhang · Tianyu Zhang · Guanglin Niu · Zichang Tan · Zichang Tan · Yalong Bai · Qing Yang, ,,https://openreview.net/forum?id=5a79AqFr0c,,,,,nan
+VicTR: Video-conditioned Text Representations for Activity Recognition,Kumara Kahatapitiya · Anurag Arnab · Arsha Nagrani · Michael Ryoo, ,https://ar5iv.labs.arxiv.org/html/2309.00696,,2309.00696.pdf,AAN: Attributes-Aware Network for Temporal Action Detection,"The challenge of long-term video understanding remains constrained by the
+efficient extraction of object semantics and the modelling of their
+relationships for downstream tasks. Although the CLIP visual features exhibit
+discriminative properties for various vision tasks, particularly in object
+encoding, they are suboptimal for long-term video understanding. To address
+this issue, we present the Attributes-Aware Network (AAN), which consists of
+two key components: the Attributes Extractor and a Graph Reasoning block. These
+components facilitate the extraction of object-centric attributes and the
+modelling of their relationships within the video. By leveraging CLIP features,
+AAN outperforms state-of-the-art approaches on two popular action detection
+datasets: Charades and Toyota Smarthome Untrimmed datasets.",cs.CV,['cs.CV']
+Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain,Qunliang Xing · Mai Xu · Shengxi Li · Xin Deng · Meisong Zheng · huaida liu · Ying Chen, ,https://arxiv.org/abs/2402.17200,,2402.17200.pdf,Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain,"Existing quality enhancement methods for compressed images focus on aligning
+the enhancement domain with the raw domain to yield realistic images. However,
+these methods exhibit a pervasive enhancement bias towards the compression
+domain, inadvertently regarding it as more realistic than the raw domain. This
+bias makes enhanced images closely resemble their compressed counterparts, thus
+degrading their perceptual quality. In this paper, we propose a simple yet
+effective method to mitigate this bias and enhance the quality of compressed
+images. Our method employs a conditional discriminator with the compressed
+image as a key condition, and then incorporates a domain-divergence
+regularization to actively distance the enhancement domain from the compression
+domain. Through this dual strategy, our method enables the discrimination
+against the compression domain, and brings the enhancement domain closer to the
+raw domain. Comprehensive quality evaluations confirm the superiority of our
+method over other state-of-the-art methods without incurring inference
+overheads.",cs.CV,"['cs.CV', 'eess.IV']"
+SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction,Conghao Wong · Beihao Xia · Ziqian Zou · Yulong Wang · Xinge You,https://cocoon2wong.github.io/SocialCircle,https://arxiv.org/abs/2310.05370,,2310.05370.pdf,SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction,"Analyzing and forecasting trajectories of agents like pedestrians and cars in
+complex scenes has become more and more significant in many intelligent systems
+and applications. The diversity and uncertainty in socially interactive
+behaviors among a rich variety of agents make this task more challenging than
+other deterministic computer vision tasks. Researchers have made a lot of
+efforts to quantify the effects of these interactions on future trajectories
+through different mathematical models and network structures, but this problem
+has not been well solved. Inspired by marine animals that localize the
+positions of their companions underwater through echoes, we build a new
+anglebased trainable social interaction representation, named SocialCircle, for
+continuously reflecting the context of social interactions at different angular
+orientations relative to the target agent. We validate the effect of the
+proposed SocialCircle by training it along with several newly released
+trajectory prediction models, and experiments show that the SocialCircle not
+only quantitatively improves the prediction performance, but also qualitatively
+helps better simulate social interactions when forecasting pedestrian
+trajectories in a way that is consistent with human intuitions.",cs.CV,['cs.CV']
+Dr.Hair: Reconstructing Scalp-Connected Hair Strands without Pre-training via Differentiable Rendering of Line Segments,Yusuke Takimoto · Hikari Takehara · Hiroyuki Sato · Zihao Zhu · Bo Zheng,https://dr-hair.github.io/Dr-Hair/,https://arxiv.org/abs/2403.17496,,2403.17496.pdf,Dr.Hair: Reconstructing Scalp-Connected Hair Strands without Pre-training via Differentiable Rendering of Line Segments,"In the film and gaming industries, achieving a realistic hair appearance
+typically involves the use of strands originating from the scalp. However,
+reconstructing these strands from observed surface images of hair presents
+significant challenges. The difficulty in acquiring Ground Truth (GT) data has
+led state-of-the-art learning-based methods to rely on pre-training with
+manually prepared synthetic CG data. This process is not only labor-intensive
+and costly but also introduces complications due to the domain gap when
+compared to real-world data. In this study, we propose an optimization-based
+approach that eliminates the need for pre-training. Our method represents hair
+strands as line segments growing from the scalp and optimizes them using a
+novel differentiable rendering algorithm. To robustly optimize a substantial
+number of slender explicit geometries, we introduce 3D orientation estimation
+utilizing global optimization, strand initialization based on Laplace's
+equation, and reparameterization that leverages geometric connectivity and
+spatial proximity. Unlike existing optimization-based methods, our method is
+capable of reconstructing internal hair flow in an absolute direction. Our
+method exhibits robust and accurate inverse rendering, surpassing the quality
+of existing methods and significantly improving processing speed.",cs.CV,"['cs.CV', 'cs.GR']"
+SnAG: Scalable and Accurate Video Grounding,Fangzhou Mu · Sicheng Mo · Yin Li, ,https://arxiv.org/abs/2404.02257,,2404.02257.pdf,SnAG: Scalable and Accurate Video Grounding,"Temporal grounding of text descriptions in videos is a central problem in
+vision-language learning and video understanding. Existing methods often
+prioritize accuracy over scalability -- they have been optimized for grounding
+only a few text queries within short videos, and fail to scale up to long
+videos with hundreds of queries. In this paper, we study the effect of
+cross-modal fusion on the scalability of video grounding models. Our analysis
+establishes late fusion as a more cost-effective fusion scheme for long-form
+videos with many text queries. Moreover, it leads us to a novel, video-centric
+sampling scheme for efficient training. Based on these findings, we present
+SnAG, a simple baseline for scalable and accurate video grounding. Without
+bells and whistles, SnAG is 43% more accurate and 1.5x faster than CONE, a
+state of the art for long-form video grounding on the challenging MAD dataset,
+while achieving highly competitive results on short videos.",cs.CV,['cs.CV']
+Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Clouds,Tianrui Lou · Xiaojun Jia · Jindong Gu · Li Liu · Siyuan Liang · Bangyan He · Xiaochun Cao, ,https://arxiv.org/abs/2403.05247,,2403.05247.pdf,Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Clouds,"Adversarial attack methods based on point manipulation for 3D point cloud
+classification have revealed the fragility of 3D models, yet the adversarial
+examples they produce are easily perceived or defended against. The trade-off
+between the imperceptibility and adversarial strength leads most point attack
+methods to inevitably introduce easily detectable outlier points upon a
+successful attack. Another promising strategy, shape-based attack, can
+effectively eliminate outliers, but existing methods often suffer significant
+reductions in imperceptibility due to irrational deformations. We find that
+concealing deformation perturbations in areas insensitive to human eyes can
+achieve a better trade-off between imperceptibility and adversarial strength,
+specifically in parts of the object surface that are complex and exhibit
+drastic curvature changes. Therefore, we propose a novel shape-based
+adversarial attack method, HiT-ADV, which initially conducts a two-stage search
+for attack regions based on saliency and imperceptibility scores, and then adds
+deformation perturbations in each attack region using Gaussian kernel
+functions. Additionally, HiT-ADV is extendable to physical attack. We propose
+that by employing benign resampling and benign rigid transformations, we can
+further enhance physical adversarial strength with little sacrifice to
+imperceptibility. Extensive experiments have validated the superiority of our
+method in terms of adversarial and imperceptible properties in both digital and
+physical spaces. Our code is avaliable at: https://github.com/TRLou/HiT-ADV.",cs.CV,"['cs.CV', 'eess.IV']"
+PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization,Zining Chen · Weiqiu Wang · Zhicheng Zhao · Fei Su · Aidong Men · Hongying Meng, ,https://arxiv.org/abs/2404.09011,,2404.09011.pdf,PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization,"Domain Generalization (DG) aims to resolve distribution shifts between source
+and target domains, and current DG methods are default to the setting that data
+from source and target domains share identical categories. Nevertheless, there
+exists unseen classes from target domains in practical scenarios. To address
+this issue, Open Set Domain Generalization (OSDG) has emerged and several
+methods have been exclusively proposed. However, most existing methods adopt
+complex architectures with slight improvement compared with DG methods.
+Recently, vision-language models (VLMs) have been introduced in DG following
+the fine-tuning paradigm, but consume huge training overhead with large vision
+models. Therefore, in this paper, we innovate to transfer knowledge from VLMs
+to lightweight vision models and improve the robustness by introducing
+Perturbation Distillation (PD) from three perspectives, including Score, Class
+and Instance (SCI), named SCI-PD. Moreover, previous methods are oriented by
+the benchmarks with identical and fixed splits, ignoring the divergence between
+source domains. These methods are revealed to suffer from sharp performance
+decay with our proposed new benchmark Hybrid Domain Generalization (HDG) and a
+novel metric $H^{2}$-CV, which construct various splits to comprehensively
+assess the robustness of algorithms. Extensive experiments demonstrate that our
+method outperforms state-of-the-art algorithms on multiple datasets, especially
+improving the robustness when confronting data scarcity.",cs.CV,"['cs.CV', 'cs.LG']"
+Kernel Adaptive Convolution for Scene Text Detection via Distance Map Prediction,Jinzhi Zheng · Heng Fan · Libo Zhang, ,https://arxiv.org/html/2401.11704v1,,2401.11704v1.pdf,EK-Net:Real-time Scene Text Detection with Expand Kernel Distance,"Recently, scene text detection has received significant attention due to its
+wide application. However, accurate detection in complex scenes of multiple
+scales, orientations, and curvature remains a challenge. Numerous detection
+methods adopt the Vatti clipping (VC) algorithm for multiple-instance training
+to address the issue of arbitrary-shaped text. Yet we identify several bias
+results from these approaches called the ""shrinked kernel"". Specifically, it
+refers to a decrease in accuracy resulting from an output that overly favors
+the text kernel. In this paper, we propose a new approach named Expand Kernel
+Network (EK-Net) with expand kernel distance to compensate for the previous
+deficiency, which includes three-stages regression to complete instance
+detection. Moreover, EK-Net not only realize the precise positioning of
+arbitrary-shaped text, but also achieve a trade-off between performance and
+speed. Evaluation results demonstrate that EK-Net achieves state-of-the-art or
+competitive performance compared to other advanced methods, e.g., F-measure of
+85.72% at 35.42 FPS on ICDAR 2015, F-measure of 85.75% at 40.13 FPS on CTW1500.",cs.CV,['cs.CV']
+CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion,Xiaoyu Wu · Yang Hua · Chumeng Liang · Jiaru Zhang · Hao Wang · Tao Song · Haibing Guan,https://github.com/Nicholas0228/Revelio,https://arxiv.org/abs/2403.11162,,2403.11162.pdf,CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion,"Diffusion Models (DMs) have evolved into advanced image generation tools,
+especially for few-shot generation where a pretrained model is fine-tuned on a
+small set of images to capture a specific style or object. Despite their
+success, concerns exist about potential copyright violations stemming from the
+use of unauthorized data in this process. In response, we present Contrasting
+Gradient Inversion for Diffusion Models (CGI-DM), a novel method featuring
+vivid visual representations for digital copyright authentication. Our approach
+involves removing partial information of an image and recovering missing
+details by exploiting conceptual differences between the pretrained and
+fine-tuned models. We formulate the differences as KL divergence between latent
+variables of the two models when given the same input image, which can be
+maximized through Monte Carlo sampling and Projected Gradient Descent (PGD).
+The similarity between original and recovered images serves as a strong
+indicator of potential infringements. Extensive experiments on the WikiArt and
+Dreambooth datasets demonstrate the high accuracy of CGI-DM in digital
+copyright authentication, surpassing alternative validation techniques. Code
+implementation is available at https://github.com/Nicholas0228/Revelio.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CR', 'cs.CY', 'cs.LG']"
+Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration,Yuxi Wei · Zi Wang · Yifan Lu · Chenxin Xu · Changxing Liu · Hao Zhao · Siheng Chen · Yanfeng Wang,https://yifanlu0227.github.io/ChatSim/,https://arxiv.org/abs/2402.05746,,2402.05746.pdf,Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents,"Scene simulation in autonomous driving has gained significant attention
+because of its huge potential for generating customized data. However, existing
+editable scene simulation approaches face limitations in terms of user
+interaction efficiency, multi-camera photo-realistic rendering and external
+digital assets integration. To address these challenges, this paper introduces
+ChatSim, the first system that enables editable photo-realistic 3D driving
+scene simulations via natural language commands with external digital assets.
+To enable editing with high command flexibility,~ChatSim leverages a large
+language model (LLM) agent collaboration framework. To generate photo-realistic
+outcomes, ChatSim employs a novel multi-camera neural radiance field method.
+Furthermore, to unleash the potential of extensive high-quality digital assets,
+ChatSim employs a novel multi-camera lighting estimation method to achieve
+scene-consistent assets' rendering. Our experiments on Waymo Open Dataset
+demonstrate that ChatSim can handle complex language commands and generate
+corresponding photo-realistic scene videos.",cs.CV,['cs.CV']
+An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning,Jianqing Zhang · Yang Liu · Yang Hua · Jian Cao,https://github.com/TsingZ0/FedKTL,https://arxiv.org/abs/2403.15760,,2403.15760.pdf,An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning,"Heterogeneous Federated Learning (HtFL) enables collaborative learning on
+multiple clients with different model architectures while preserving privacy.
+Despite recent research progress, knowledge sharing in HtFL is still difficult
+due to data and model heterogeneity. To tackle this issue, we leverage the
+knowledge stored in pre-trained generators and propose a new upload-efficient
+knowledge transfer scheme called Federated Knowledge-Transfer Loop (FedKTL).
+Our FedKTL can produce client-task-related prototypical image-vector pairs via
+the generator's inference on the server. With these pairs, each client can
+transfer pre-existing knowledge from the generator to its local model through
+an additional supervised local task. We conduct extensive experiments on four
+datasets under two types of data heterogeneity with 14 kinds of models
+including CNNs and ViTs. Results show that our upload-efficient FedKTL
+surpasses seven state-of-the-art methods by up to 7.31% in accuracy. Moreover,
+our knowledge transfer scheme is applicable in scenarios with only one edge
+client. Code: https://github.com/TsingZ0/FedKTL",cs.AI,"['cs.AI', 'cs.DC']"
+Language-conditioned Detection Transformer,Jang Hyun Cho · Philipp Krähenbühl,https://janghyuncho.github.io/DECOLA/,,https://www.semanticscholar.org/paper/Language-conditioned-Detection-Transformer-Cho-Krähenbühl/d590b8cabee3630327fa72149a2b137b2c0892f9/figure/0,,,,,nan
+Audio-Visual Segmentation via Unlabeled Frame Exploitation,Jinxiang Liu · Yikun Liu · Ferenas · Chen Ju · Ya Zhang · Yanfeng Wang, ,https://arxiv.org/abs/2403.11074,,2403.11074.pdf,Audio-Visual Segmentation via Unlabeled Frame Exploitation,"Audio-visual segmentation (AVS) aims to segment the sounding objects in video
+frames. Although great progress has been witnessed, we experimentally reveal
+that current methods reach marginal performance gain within the use of the
+unlabeled frames, leading to the underutilization issue. To fully explore the
+potential of the unlabeled frames for AVS, we explicitly divide them into two
+categories based on their temporal characteristics, i.e., neighboring frame
+(NF) and distant frame (DF). NFs, temporally adjacent to the labeled frame,
+often contain rich motion information that assists in the accurate localization
+of sounding objects. Contrary to NFs, DFs have long temporal distances from the
+labeled frame, which share semantic-similar objects with appearance variations.
+Considering their unique characteristics, we propose a versatile framework that
+effectively leverages them to tackle AVS. Specifically, for NFs, we exploit the
+motion cues as the dynamic guidance to improve the objectness localization.
+Besides, we exploit the semantic cues in DFs by treating them as valid
+augmentations to the labeled frames, which are then used to enrich data
+diversity in a self-training manner. Extensive experimental results demonstrate
+the versatility and superiority of our method, unleashing the power of the
+abundant unlabeled frames.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM', 'cs.SD', 'eess.AS']"
+Distilling ODE Solvers of Diffusion Models into Smaller Steps,Sanghwan Kim · Hao Tang · Fisher Yu, ,https://arxiv.org/abs/2309.16421,,2309.16421.pdf,Distilling ODE Solvers of Diffusion Models into Smaller Steps,"Abstract Diffusion models have recently gained prominence as a novel category
+of generative models. Despite their success, these models face a notable
+drawback in terms of slow sampling speeds, requiring a high number of function
+evaluations (NFE) in the order of hundreds or thousands. In response, both
+learning-free and learning-based sampling strategies have been explored to
+expedite the sampling process. Learning-free sampling employs various ordinary
+differential equation (ODE) solvers based on the formulation of diffusion ODEs.
+However, it encounters challenges in faithfully tracking the true sampling
+trajectory, particularly for small NFE. Conversely, learning-based sampling
+methods, such as knowledge distillation, demand extensive additional training,
+limiting their practical applicability. To overcome these limitations, we
+introduce Distilled-ODE solvers (D-ODE solvers), a straightforward distillation
+approach grounded in ODE solver formulations. Our method seamlessly integrates
+the strengths of both learning-free and learning-based sampling. D-ODE solvers
+are constructed by introducing a single parameter adjustment to existing ODE
+solvers. Furthermore, we optimize D-ODE solvers with smaller steps using
+knowledge distillation from ODE solvers with larger steps across a batch of
+samples. Comprehensive experiments demonstrate the superior performance of
+D-ODE solvers compared to existing ODE solvers, including DDIM, PNDM,
+DPM-Solver, DEIS, and EDM, particularly in scenarios with fewer NFE. Notably,
+our method incurs negligible computational overhead compared to previous
+distillation techniques, facilitating straightforward and rapid integration
+with existing samplers. Qualitative analysis reveals that D-ODE solvers not
+only enhance image quality but also faithfully follow the target ODE
+trajectory.",cs.CV,['cs.CV']
+Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation,Keonhee Han · Dominik Muhle · Felix Wimbauer · Daniel Cremers,https://keonhee-han.github.io/publications/kdbts/,https://arxiv.org/abs/2404.07933,,2404.07933.pdf,Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation,"Inferring scene geometry from images via Structure from Motion is a
+long-standing and fundamental problem in computer vision. While classical
+approaches and, more recently, depth map predictions only focus on the visible
+parts of a scene, the task of scene completion aims to reason about geometry
+even in occluded regions. With the popularity of neural radiance fields
+(NeRFs), implicit representations also became popular for scene completion by
+predicting so-called density fields. Unlike explicit approaches. e.g.
+voxel-based methods, density fields also allow for accurate depth prediction
+and novel-view synthesis via image-based rendering. In this work, we propose to
+fuse the scene reconstruction from multiple images and distill this knowledge
+into a more accurate single-view scene reconstruction. To this end, we propose
+Multi-View Behind the Scenes (MVBTS) to fuse density fields from multiple posed
+images, trained fully self-supervised only from image data. Using knowledge
+distillation, we use MVBTS to train a single-view scene completion network via
+direct supervision called KDBTS. It achieves state-of-the-art performance on
+occupancy prediction, especially in occluded regions.",cs.CV,['cs.CV']
+Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion,Sofia Casarin · Cynthia Ugwu · Sergio Escalera · Oswald Lanz, ,https://arxiv.org/abs/2403.15194,,2403.15194.pdf,Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion,"The landscape of deep learning research is moving towards innovative
+strategies to harness the true potential of data. Traditionally, emphasis has
+been on scaling model architectures, resulting in large and complex neural
+networks, which can be difficult to train with limited computational resources.
+However, independently of the model size, data quality (i.e. amount and
+variability) is still a major factor that affects model generalization. In this
+work, we propose a novel technique to exploit available data through the use of
+automatic data augmentation for the tasks of image classification and semantic
+segmentation. We introduce the first Differentiable Augmentation Search method
+(DAS) to generate variations of images that can be processed as videos.
+Compared to previous approaches, DAS is extremely fast and flexible, allowing
+the search on very large search spaces in less than a GPU day. Our intuition is
+that the increased receptive field in the temporal dimension provided by DAS
+could lead to benefits also to the spatial receptive field. More specifically,
+we leverage DAS to guide the reshaping of the spatial receptive field by
+selecting task-dependant transformations. As a result, compared to standard
+augmentation alternatives, we improve in terms of accuracy on ImageNet,
+Cifar10, Cifar100, Tiny-ImageNet, Pascal-VOC-2012 and CityScapes datasets when
+plugging-in our DAS over different light-weight video backbones.",cs.CV,"['cs.CV', 'cs.LG']"
+A Dynamic Kernel Prior Model for Unsupervised Blind Image Super-Resolution,Zhixiong Yang · Jingyuan Xia · Shengxi Li · Xinghua Huang · Shuanghui Zhang · Zhen Liu · Yaowen Fu · Yongxiang Liu, ,https://arxiv.org/abs/2404.15620,,2404.15620.pdf,A Dynamic Kernel Prior Model for Unsupervised Blind Image Super-Resolution,"Deep learning-based methods have achieved significant successes on solving
+the blind super-resolution (BSR) problem. However, most of them request
+supervised pre-training on labelled datasets. This paper proposes an
+unsupervised kernel estimation model, named dynamic kernel prior (DKP), to
+realize an unsupervised and pre-training-free learning-based algorithm for
+solving the BSR problem. DKP can adaptively learn dynamic kernel priors to
+realize real-time kernel estimation, and thereby enables superior HR image
+restoration performances. This is achieved by a Markov chain Monte Carlo
+sampling process on random kernel distributions. The learned kernel prior is
+then assigned to optimize a blur kernel estimation network, which entails a
+network-based Langevin dynamic optimization strategy. These two techniques
+ensure the accuracy of the kernel estimation. DKP can be easily used to replace
+the kernel estimation models in the existing methods, such as Double-DIP and
+FKP-DIP, or be added to the off-the-shelf image restoration model, such as
+diffusion model. In this paper, we incorporate our DKP model with DIP and
+diffusion model, referring to DIP-DKP and Diff-DKP, for validations. Extensive
+simulations on Gaussian and motion kernel scenarios demonstrate that the
+proposed DKP model can significantly improve the kernel estimation with
+comparable runtime and memory usage, leading to state-of-the-art BSR results.
+The code is available at https://github.com/XYLGroup/DKP.",eess.IV,['eess.IV']
+DiffusionTrack:  Point Set Diffusion Model for Visual Object Tracking,Fei Xie · Zhongdao Wang · Chao Ma, ,https://arxiv.org/abs/2308.09905,,2308.09905.pdf,DiffusionTrack: Diffusion Model For Multi-Object Tracking,"Multi-object tracking (MOT) is a challenging vision task that aims to detect
+individual objects within a single frame and associate them across multiple
+frames. Recent MOT approaches can be categorized into two-stage
+tracking-by-detection (TBD) methods and one-stage joint detection and tracking
+(JDT) methods. Despite the success of these approaches, they also suffer from
+common problems, such as harmful global or local inconsistency, poor trade-off
+between robustness and model complexity, and lack of flexibility in different
+scenes within the same video. In this paper we propose a simple but robust
+framework that formulates object detection and association jointly as a
+consistent denoising diffusion process from paired noise boxes to paired
+ground-truth boxes. This novel progressive denoising diffusion strategy
+substantially augments the tracker's effectiveness, enabling it to discriminate
+between various objects. During the training stage, paired object boxes diffuse
+from paired ground-truth boxes to random distribution, and the model learns
+detection and tracking simultaneously by reversing this noising process. In
+inference, the model refines a set of paired randomly generated boxes to the
+detection and tracking results in a flexible one-step or multi-step denoising
+diffusion process. Extensive experiments on three widely used MOT benchmarks,
+including MOT17, MOT20, and Dancetrack, demonstrate that our approach achieves
+competitive performance compared to the current state-of-the-art methods.",cs.CV,['cs.CV']
+SurroundSDF: Implicit 3D Scene Understanding Based on Signed Distance Field,Lizhe Liu · Bohua Wang · Hongwei Xie · Daqi Liu · Li Liu · Kuiyuan Yang · Bing Wang · Zhiqiang Tian, ,https://arxiv.org/abs/2403.14366,,2403.14366.pdf,SurroundSDF: Implicit 3D Scene Understanding Based on Signed Distance Field,"Vision-centric 3D environment understanding is both vital and challenging for
+autonomous driving systems. Recently, object-free methods have attracted
+considerable attention. Such methods perceive the world by predicting the
+semantics of discrete voxel grids but fail to construct continuous and accurate
+obstacle surfaces. To this end, in this paper, we propose SurroundSDF to
+implicitly predict the signed distance field (SDF) and semantic field for the
+continuous perception from surround images. Specifically, we introduce a
+query-based approach and utilize SDF constrained by the Eikonal formulation to
+accurately describe the surfaces of obstacles. Furthermore, considering the
+absence of precise SDF ground truth, we propose a novel weakly supervised
+paradigm for SDF, referred to as the Sandwich Eikonal formulation, which
+emphasizes applying correct and dense constraints on both sides of the surface,
+thereby enhancing the perceptual accuracy of the surface. Experiments suggest
+that our method achieves SOTA for both occupancy prediction and 3D scene
+reconstruction tasks on the nuScenes dataset.",cs.CV,['cs.CV']
+DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior,Tianyu Huang · Yihan Zeng · Zhilu Zhang · Wan Xu · Hang Xu · Songcen Xu · Rynson W.H. Lau · Wangmeng Zuo,https://github.com/tyhuang0428/DreamControl,https://arxiv.org/abs/2312.06439,,2312.06439.pdf,DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior,"3D generation has raised great attention in recent years. With the success of
+text-to-image diffusion models, the 2D-lifting technique becomes a promising
+route to controllable 3D generation. However, these methods tend to present
+inconsistent geometry, which is also known as the Janus problem. We observe
+that the problem is caused mainly by two aspects, i.e., viewpoint bias in 2D
+diffusion models and overfitting of the optimization objective. To address it,
+we propose a two-stage 2D-lifting framework, namely DreamControl, which
+optimizes coarse NeRF scenes as 3D self-prior and then generates fine-grained
+objects with control-based score distillation. Specifically, adaptive viewpoint
+sampling and boundary integrity metric are proposed to ensure the consistency
+of generated priors. The priors are then regarded as input conditions to
+maintain reasonable geometries, in which conditional LoRA and weighted score
+are further proposed to optimize detailed textures. DreamControl can generate
+high-quality 3D content in terms of both geometry consistency and texture
+fidelity. Moreover, our control-based optimization guidance is applicable to
+more downstream tasks, including user-guided generation and 3D animation. The
+project page is available at https://github.com/tyhuang0428/DreamControl.",cs.CV,['cs.CV']
+Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID,Wentao Tan · Changxing Ding · Jiayu Jiang · Fei Wang · Yibing Zhan · Dapeng Tao, ,https://arxiv.org/abs/2405.04940,,,Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID,"Text-to-image person re-identification (ReID) retrieves pedestrian images
+according to textual descriptions. Manually annotating textual descriptions is
+time-consuming, restricting the scale of existing datasets and therefore the
+generalization ability of ReID models. As a result, we study the transferable
+text-to-image ReID problem, where we train a model on our proposed large-scale
+database and directly deploy it to various datasets for evaluation. We obtain
+substantial training data via Multi-modal Large Language Models (MLLMs).
+Moreover, we identify and address two key challenges in utilizing the obtained
+textual descriptions. First, an MLLM tends to generate descriptions with
+similar structures, causing the model to overfit specific sentence patterns.
+Thus, we propose a novel method that uses MLLMs to caption images according to
+various templates. These templates are obtained using a multi-turn dialogue
+with a Large Language Model (LLM). Therefore, we can build a large-scale
+dataset with diverse textual descriptions. Second, an MLLM may produce
+incorrect descriptions. Hence, we introduce a novel method that automatically
+identifies words in a description that do not correspond with the image. This
+method is based on the similarity between one text and all patch token
+embeddings in the image. Then, we mask these words with a larger probability in
+the subsequent training epoch, alleviating the impact of noisy textual
+descriptions. The experimental results demonstrate that our methods
+significantly boost the direct transfer text-to-image ReID performance.
+Benefiting from the pre-trained model weights, we also achieve state-of-the-art
+performance in the traditional evaluation settings.",cs.CV,['cs.CV']
+Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields,Leili Goli · Cody Reading · Silvia Sellán · Alec Jacobson · Andrea Tagliasacchi, ,https://arxiv.org/abs/2309.03185,,2309.03185.pdf,Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields,"Neural Radiance Fields (NeRFs) have shown promise in applications like view
+synthesis and depth estimation, but learning from multiview images faces
+inherent uncertainties. Current methods to quantify them are either heuristic
+or computationally demanding. We introduce BayesRays, a post-hoc framework to
+evaluate uncertainty in any pre-trained NeRF without modifying the training
+process. Our method establishes a volumetric uncertainty field using spatial
+perturbations and a Bayesian Laplace approximation. We derive our algorithm
+statistically and show its superior performance in key metrics and
+applications. Additional results available at: https://bayesrays.github.io.",cs.CV,['cs.CV']
+CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment,Sajid Javed · Arif Mahmood · IYYAKUTTI IYAPPAN GANAPATHI · Fayaz Ali · Naoufel Werghi · Mohammed Bennamoun, ,https://arxiv.org/abs/2306.07831,,2306.07831.pdf,Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images,"Contrastive visual language pretraining has emerged as a powerful method for
+either training new language-aware image encoders or augmenting existing
+pretrained models with zero-shot visual recognition capabilities. However,
+existing works typically train on large datasets of image-text pairs and have
+been designed to perform downstream tasks involving only small to medium
+sized-images, neither of which are applicable to the emerging field of
+computational pathology where there are limited publicly available paired
+image-text datasets and each image can span up to 100,000 x 100,000 pixels. In
+this paper we present MI-Zero, a simple and intuitive framework for unleashing
+the zero-shot transfer capabilities of contrastively aligned image and text
+models on gigapixel histopathology whole slide images, enabling multiple
+downstream diagnostic tasks to be carried out by pretrained encoders without
+requiring any additional labels. MI-Zero reformulates zero-shot transfer under
+the framework of multiple instance learning to overcome the computational
+challenge of inference on extremely large images. We used over 550k pathology
+reports and other available in-domain text corpora to pre-train our text
+encoder. By effectively leveraging strong pre-trained encoders, our best model
+pretrained on over 33k histopathology image-caption pairs achieves an average
+median zero-shot accuracy of 70.2% across three different real-world cancer
+subtyping tasks. Our code is available at:
+https://github.com/mahmoodlab/MI-Zero.",cs.CV,['cs.CV']
+UniMODE: Unified Monocular 3D Object Detection,Zhuoling Li · Xiaogang Xu · Ser-Nam Lim · Hengshuang Zhao, ,https://arxiv.org/abs/2402.18573,,2402.18573.pdf,UniMODE: Unified Monocular 3D Object Detection,"Realizing unified monocular 3D object detection, including both indoor and
+outdoor scenes, holds great importance in applications like robot navigation.
+However, involving various scenarios of data to train models poses challenges
+due to their significantly different characteristics, e.g., diverse geometry
+properties and heterogeneous domain distributions. To address these challenges,
+we build a detector based on the bird's-eye-view (BEV) detection paradigm,
+where the explicit feature projection is beneficial to addressing the geometry
+learning ambiguity when employing multiple scenarios of data to train
+detectors. Then, we split the classical BEV detection architecture into two
+stages and propose an uneven BEV grid design to handle the convergence
+instability caused by the aforementioned challenges. Moreover, we develop a
+sparse BEV feature projection strategy to reduce computational cost and a
+unified domain alignment method to handle heterogeneous domains. Combining
+these techniques, a unified detector UniMODE is derived, which surpasses the
+previous state-of-the-art on the challenging Omni3D dataset (a large-scale
+dataset including both indoor and outdoor scenes) by 4.9% AP_3D, revealing the
+first successful generalization of a BEV detector to unified 3D object
+detection.",cs.CV,['cs.CV']
+Perceptual Assessment and Optimization of HDR Image Rendering,Peibei Cao · Rafal Mantiuk · Kede Ma, ,https://arxiv.org/abs/2310.12877v4,,2310.12877v4.pdf,Perceptual Assessment and Optimization of High Dynamic Range Image Rendering,"High dynamic range (HDR) rendering has the ability to faithfully reproduce
+the wide luminance ranges in natural scenes, but how to accurately assess the
+rendering quality is relatively underexplored. Existing quality models are
+mostly designed for low dynamic range (LDR) images, and do not align well with
+human perception of HDR image quality. To fill this gap, we propose a family of
+HDR quality metrics, in which the key step is employing a simple inverse
+display model to decompose an HDR image into a stack of LDR images with varying
+exposures. Subsequently, these decomposed images are assessed through
+well-established LDR quality metrics. Our HDR quality models present three
+distinct benefits. First, they directly inherit the recent advancements of LDR
+quality metrics. Second, they do not rely on human perceptual data of HDR image
+quality for re-calibration. Third, they facilitate the alignment and
+prioritization of specific luminance ranges for more accurate and detailed
+quality assessment. Experimental results show that our HDR quality metrics
+consistently outperform existing models in terms of quality assessment on four
+HDR image quality datasets and perceptual optimization of HDR novel view
+synthesis.",eess.IV,"['eess.IV', 'cs.CV']"
+From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior,Jaeho Moon · Juan Luis Gonzalez Bello · Byeongjun Kwon · Munchurl Kim,https://kaist-viclab.github.io/From_Ground_To_Objects_site/,https://arxiv.org/abs/2312.10118,,2312.10118.pdf,From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior,"Self-supervised monocular depth estimation (DE) is an approach to learning
+depth without costly depth ground truths. However, it often struggles with
+moving objects that violate the static scene assumption during training. To
+address this issue, we introduce a coarse-to-fine training strategy leveraging
+the ground contacting prior based on the observation that most moving objects
+in outdoor scenes contact the ground. In the coarse training stage, we exclude
+the objects in dynamic classes from the reprojection loss calculation to avoid
+inaccurate depth learning. To provide precise supervision on the depth of the
+objects, we present a novel Ground-contacting-prior Disparity Smoothness Loss
+(GDS-Loss) that encourages a DE network to align the depth of the objects with
+their ground-contacting points. Subsequently, in the fine training stage, we
+refine the DE network to learn the detailed depth of the objects from the
+reprojection loss, while ensuring accurate DE on the moving object regions by
+employing our regularization loss with a cost-volume-based weighting factor.
+Our overall coarse-to-fine training strategy can easily be integrated with
+existing DE methods without any modifications, significantly enhancing DE
+performance on challenging Cityscapes and KITTI datasets, especially in the
+moving object regions.",cs.CV,['cs.CV']
+DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models,Khawar Islam · Muhammad Zaigham Zaheer · Arif Mahmood · Karthik Nandakumar,https://diffusemix.github.io/,https://arxiv.org/abs/2405.14881,,2405.14881.pdf,DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models,"Recently, a number of image-mixing-based augmentation techniques have been
+introduced to improve the generalization of deep neural networks. In these
+techniques, two or more randomly selected natural images are mixed together to
+generate an augmented image. Such methods may not only omit important portions
+of the input images but also introduce label ambiguities by mixing images
+across labels resulting in misleading supervisory signals. To address these
+limitations, we propose DiffuseMix, a novel data augmentation technique that
+leverages a diffusion model to reshape training images, supervised by our
+bespoke conditional prompts. First, concatenation of a partial natural image
+and its generated counterpart is obtained which helps in avoiding the
+generation of unrealistic images or label ambiguities. Then, to enhance
+resilience against adversarial attacks and improves safety measures, a randomly
+selected structural pattern from a set of fractal images is blended into the
+concatenated image to form the final augmented image for training. Our
+empirical results on seven different datasets reveal that DiffuseMix achieves
+superior performance compared to existing state-of the-art methods on tasks
+including general classification,fine-grained classification, fine-tuning, data
+scarcity, and adversarial robustness. Augmented datasets and codes are
+available here: https://diffusemix.github.io/",cs.CV,['cs.CV']
+Neural Exposure Fusion for High-Dynamic Range Object Detection,Emmanuel Onzon · Maximilian Bömer · Fahim Mannan · Felix Heide, ,https://arxiv.org/abs/2405.16038,,2405.16038.pdf,Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection,"Most recent multispectral object detectors employ a two-branch structure to
+extract features from RGB and thermal images. While the two-branch structure
+achieves better performance than a single-branch structure, it overlooks
+inference efficiency. This conflict is increasingly aggressive, as recent works
+solely pursue higher performance rather than both performance and efficiency.
+In this paper, we address this issue by improving the performance of efficient
+single-branch structures. We revisit the reasons causing the performance gap
+between these structures. For the first time, we reveal the information
+interference problem in the naive early-fusion strategy adopted by previous
+single-branch structures. Besides, we find that the domain gap between
+multispectral images, and weak feature representation of the single-branch
+structure are also key obstacles for performance. Focusing on these three
+problems, we propose corresponding solutions, including a novel shape-priority
+early-fusion strategy, a weakly supervised learning method, and a core
+knowledge distillation technique. Experiments demonstrate that single-branch
+networks equipped with these three contributions achieve significant
+performance enhancements while retaining high efficiency. Our code will be
+available at
+\url{https://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection}.",cs.CV,['cs.CV']
+Cross-view and Cross-pose Completion for 3D Human Understanding,Matthieu Armando · Salma Galaaoui · Fabien Baradel · Thomas Lucas · Vincent Leroy · Romain BRÉGIER · Philippe Weinzaepfel · Grégory Rogez, ,https://arxiv.org/abs/2311.09104,,2311.09104.pdf,Cross-view and Cross-pose Completion for 3D Human Understanding,"Human perception and understanding is a major domain of computer vision
+which, like many other vision subdomains recently, stands to gain from the use
+of large models pre-trained on large datasets. We hypothesize that the most
+common pre-training strategy of relying on general purpose, object-centric
+image datasets such as ImageNet, is limited by an important domain shift. On
+the other hand, collecting domain-specific ground truth such as 2D or 3D labels
+does not scale well. Therefore, we propose a pre-training approach based on
+self-supervised learning that works on human-centric data using only images.
+Our method uses pairs of images of humans: the first is partially masked and
+the model is trained to reconstruct the masked parts given the visible ones and
+a second image. It relies on both stereoscopic (cross-view) pairs, and temporal
+(cross-pose) pairs taken from videos, in order to learn priors about 3D as well
+as human motion. We pre-train a model for body-centric tasks and one for
+hand-centric tasks. With a generic transformer architecture, these models
+outperform existing self-supervised pre-training methods on a wide set of
+human-centric downstream tasks, and obtain state-of-the-art performance for
+instance when fine-tuning for model-based and model-free human mesh recovery.",cs.CV,['cs.CV']
+Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation,Zhiwu Qing · Shiwei Zhang · Jiayu Wang · Xiang Wang · Yujie Wei · Yingya Zhang · Changxin Gao · Nong Sang, ,https://arxiv.org/abs/2312.04483,,2312.04483.pdf,Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation,"Despite diffusion models having shown powerful abilities to generate
+photorealistic images, generating videos that are realistic and diverse still
+remains in its infancy. One of the key reasons is that current methods
+intertwine spatial content and temporal dynamics together, leading to a notably
+increased complexity of text-to-video generation (T2V). In this work, we
+propose HiGen, a diffusion model-based method that improves performance by
+decoupling the spatial and temporal factors of videos from two perspectives,
+i.e., structure level and content level. At the structure level, we decompose
+the T2V task into two steps, including spatial reasoning and temporal
+reasoning, using a unified denoiser. Specifically, we generate spatially
+coherent priors using text during spatial reasoning and then generate
+temporally coherent motions from these priors during temporal reasoning. At the
+content level, we extract two subtle cues from the content of the input video
+that can express motion and appearance changes, respectively. These two cues
+then guide the model's training for generating videos, enabling flexible
+content variations and enhancing temporal stability. Through the decoupled
+paradigm, HiGen can effectively reduce the complexity of this task and generate
+realistic videos with semantics accuracy and motion stability. Extensive
+experiments demonstrate the superior performance of HiGen over the
+state-of-the-art T2V methods.",cs.CV,['cs.CV']
+SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks,Xinyu Shi · Zecheng Hao · Zhaofei Yu, ,https://arxiv.org/abs/2403.14302,,2403.14302.pdf,SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks,"The remarkable success of Vision Transformers in Artificial Neural Networks
+(ANNs) has led to a growing interest in incorporating the self-attention
+mechanism and transformer-based architecture into Spiking Neural Networks
+(SNNs). While existing methods propose spiking self-attention mechanisms that
+are compatible with SNNs, they lack reasonable scaling methods, and the overall
+architectures proposed by these methods suffer from a bottleneck in effectively
+extracting local features. To address these challenges, we propose a novel
+spiking self-attention mechanism named Dual Spike Self-Attention (DSSA) with a
+reasonable scaling method. Based on DSSA, we propose a novel spiking Vision
+Transformer architecture called SpikingResformer, which combines the
+ResNet-based multi-stage architecture with our proposed DSSA to improve both
+performance and energy efficiency while reducing parameters. Experimental
+results show that SpikingResformer achieves higher accuracy with fewer
+parameters and lower energy consumption than other spiking Vision Transformer
+counterparts. Notably, our SpikingResformer-L achieves 79.40% top-1 accuracy on
+ImageNet with 4 time-steps, which is the state-of-the-art result in the SNN
+field.",cs.NE,"['cs.NE', 'cs.CV', 'cs.LG']"
+C$^2$KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation,Fushuo Huo · Wenchao Xu · Jingcai Guo · Haozhao Wang · Song Guo, ,https://arxiv.org/abs/2312.17648,,2312.17648.pdf,Bridging Modality Gap for Visual Grounding with Effecitve Cross-modal Distillation,"Visual grounding aims to align visual information of specific regions of
+images with corresponding natural language expressions. Current visual
+grounding methods leverage pre-trained visual and language backbones separately
+to obtain visual features and linguistic features. Although these two types of
+features are then fused via delicately designed networks, the heterogeneity of
+the features makes them inapplicable for multi-modal reasoning. This problem
+arises from the domain gap between the single-modal pre-training backbone used
+in current visual grounding methods, which can hardly be overcome by the
+traditional end-to-end training method. To alleviate this, our work proposes an
+Empowering pre-trained model for Visual Grounding (EpmVG) framework, which
+distills a multimodal pre-trained model to guide the visual grounding task.
+EpmVG is based on a novel cross-modal distillation mechanism, which can
+effectively introduce the consistency information of images and texts in the
+pre-trained model, to reduce the domain gap existing in the backbone networks,
+thereby improving the performance of the model in the visual grounding task.
+Extensive experiments are carried out on five conventionally used datasets, and
+results demonstrate that our method achieves better performance than
+state-of-the-art methods.",cs.CV,"['cs.CV', 'cs.AI']"
+Map-Relative Pose Regression for Visual Re-Localization,Shuai Chen · Tommaso Cavallari · Victor Adrian Prisacariu · Eric Brachmann, ,https://arxiv.org/abs/2404.09884,,2404.09884.pdf,Map-Relative Pose Regression for Visual Re-Localization,"Pose regression networks predict the camera pose of a query image relative to
+a known environment. Within this family of methods, absolute pose regression
+(APR) has recently shown promising accuracy in the range of a few centimeters
+in position error. APR networks encode the scene geometry implicitly in their
+weights. To achieve high accuracy, they require vast amounts of training data
+that, realistically, can only be created using novel view synthesis in a
+days-long process. This process has to be repeated for each new scene again and
+again. We present a new approach to pose regression, map-relative pose
+regression (marepo), that satisfies the data hunger of the pose regression
+network in a scene-agnostic fashion. We condition the pose regressor on a
+scene-specific map representation such that its pose predictions are relative
+to the scene map. This allows us to train the pose regressor across hundreds of
+scenes to learn the generic relation between a scene-specific map
+representation and the camera pose. Our map-relative pose regressor can be
+applied to new map representations immediately or after mere minutes of
+fine-tuning for the highest accuracy. Our approach outperforms previous pose
+regression methods by far on two public datasets, indoor and outdoor. Code is
+available: https://nianticlabs.github.io/marepo",cs.CV,"['cs.CV', 'cs.LG']"
+Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation,Zihan Wang · Xiangyang Li · Jiahao Yang · Yeqi Liu · Junjie Hu · Ming Jiang · Shuqiang Jiang,https://github.com/MrZihan/HNR-VLN,https://arxiv.org/abs/2404.01943,,2404.01943.pdf,Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation,"Vision-and-language navigation (VLN) enables the agent to navigate to a
+remote location following the natural language instruction in 3D environments.
+At each navigation step, the agent selects from possible candidate locations
+and then makes the move. For better navigation planning, the lookahead
+exploration strategy aims to effectively evaluate the agent's next action by
+accurately anticipating the future environment of candidate locations. To this
+end, some existing works predict RGB images for future environments, while this
+strategy suffers from image distortion and high computational cost. To address
+these issues, we propose the pre-trained hierarchical neural radiance
+representation model (HNR) to produce multi-level semantic features for future
+environments, which are more robust and efficient than pixel-wise RGB
+reconstruction. Furthermore, with the predicted future environmental
+representations, our lookahead VLN model is able to construct the navigable
+future path tree and select the optimal path via efficient parallel evaluation.
+Extensive experiments on the VLN-CE datasets confirm the effectiveness of our
+method.",cs.CV,"['cs.CV', 'cs.RO']"
+CAT-Seg: Cost Aggregation for Open-vocabulary Semantic Segmentation,Seokju Cho · Heeseong Shin · Sunghwan Hong · Anurag Arnab · Paul Hongsuck Seo · Seungryong Kim, ,,https://openreview.net/forum?id=ZWytHTcnTy,,,,,nan
+Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments,Liyuan Zhu · Shengyu Huang · Konrad Schindler · Iro Armeni,https://www.zhuliyuan.net/livingscenes,https://arxiv.org/abs/2312.09138,,2312.09138.pdf,Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments,"Research into dynamic 3D scene understanding has primarily focused on
+short-term change tracking from dense observations, while little attention has
+been paid to long-term changes with sparse observations. We address this gap
+with MoRE, a novel approach for multi-object relocalization and reconstruction
+in evolving environments. We view these environments as ""living scenes"" and
+consider the problem of transforming scans taken at different points in time
+into a 3D reconstruction of the object instances, whose accuracy and
+completeness increase over time. At the core of our method lies an
+SE(3)-equivariant representation in a single encoder-decoder network, trained
+on synthetic data. This representation enables us to seamlessly tackle instance
+matching, registration, and reconstruction. We also introduce a joint
+optimization algorithm that facilitates the accumulation of point clouds
+originating from the same instance across multiple scans taken at different
+points in time. We validate our method on synthetic and real-world data and
+demonstrate state-of-the-art performance in both end-to-end performance and
+individual subtasks.",cs.CV,['cs.CV']
+CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation,Kangfu Mei · Mauricio Delbracio · Hossein Talebi · Zhengzhong Tu · Vishal M. Patel · Peyman Milanfar,https://fast-codi.github.io/,https://arxiv.org/abs/2310.01407,,2310.01407.pdf,CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation,"Large generative diffusion models have revolutionized text-to-image
+generation and offer immense potential for conditional generation tasks such as
+image enhancement, restoration, editing, and compositing. However, their
+widespread adoption is hindered by the high computational cost, which limits
+their real-time application. To address this challenge, we introduce a novel
+method dubbed CoDi, that adapts a pre-trained latent diffusion model to accept
+additional image conditioning inputs while significantly reducing the sampling
+steps required to achieve high-quality results. Our method can leverage
+architectures such as ControlNet to incorporate conditioning inputs without
+compromising the model's prior knowledge gained during large scale
+pre-training. Additionally, a conditional consistency loss enforces consistent
+predictions across diffusion steps, effectively compelling the model to
+generate high-quality images with conditions in a few steps. Our
+conditional-task learning and distillation approach outperforms previous
+distillation methods, achieving a new state-of-the-art in producing
+high-quality images with very few steps (e.g., 1-4) across multiple tasks,
+including super-resolution, text-guided image editing, and depth-to-image
+generation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows,Zhenggang Tang · Jason Ren · Xiaoming Zhao · Bowen Wen · Jonathan Tremblay · Stan Birchfield · Alexander G. Schwing, ,https://arxiv.org/abs/2405.05010,,2405.05010.pdf,${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields,"Neural fields (NeRF) have emerged as a promising approach for representing
+continuous 3D scenes. Nevertheless, the lack of semantic encoding in NeRFs
+poses a significant challenge for scene decomposition. To address this
+challenge, we present a single model, Multi-Modal Decomposition NeRF
+(${M^2D}$NeRF), that is capable of both text-based and visual patch-based
+edits. Specifically, we use multi-modal feature distillation to integrate
+teacher features from pretrained visual and language models into 3D semantic
+feature volumes, thereby facilitating consistent 3D editing. To enforce
+consistency between the visual and language features in our 3D feature volumes,
+we introduce a multi-modal similarity constraint. We also introduce a
+patch-based joint contrastive loss that helps to encourage object-regions to
+coalesce in the 3D feature space, resulting in more precise boundaries.
+Experiments on various real-world scenes show superior performance in 3D scene
+decomposition tasks compared to prior NeRF-based methods.",cs.CV,['cs.CV']
+Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements,Niccolò Biondi · Federico Pernici · Simone Ricci · Alberto Del Bimbo,https://github.com/miccunifi/iamcl2r,https://arxiv.org/abs/2405.02581,,2405.02581.pdf,Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements,"Learning compatible representations enables the interchangeable use of
+semantic features as models are updated over time. This is particularly
+relevant in search and retrieval systems where it is crucial to avoid
+reprocessing of the gallery images with the updated model. While recent
+research has shown promising empirical evidence, there is still a lack of
+comprehensive theoretical understanding about learning compatible
+representations. In this paper, we demonstrate that the stationary
+representations learned by the $d$-Simplex fixed classifier optimally
+approximate compatibility representation according to the two inequality
+constraints of its formal definition. This not only establishes a solid
+foundation for future works in this line of research but also presents
+implications that can be exploited in practical learning scenarios. An
+exemplary application is the now-standard practice of downloading and
+fine-tuning new pre-trained models. Specifically, we show the strengths and
+critical issues of stationary representations in the case in which a model
+undergoing sequential fine-tuning is asynchronously replaced by downloading a
+better-performing model pre-trained elsewhere. Such a representation enables
+seamless delivery of retrieval service (i.e., no reprocessing of gallery
+images) and offers improved performance without operational disruptions during
+model replacement. Code available at: https://github.com/miccunifi/iamcl2r.",cs.CV,['cs.CV']
+Boosting Flow-based Generative Super-Resolution Models via Learned Prior,Li-Yuan Tsao · Yi-Chen Lo · Chia-Che Chang · Hao-Wei Chen · Roy Tseng · Chien Feng · Chun-Yi Lee,https://github.com/liyuantsao/FlowSR-LP,https://arxiv.org/abs/2403.10988,,2403.10988.pdf,Boosting Flow-based Generative Super-Resolution Models via Learned Prior,"Flow-based super-resolution (SR) models have demonstrated astonishing
+capabilities in generating high-quality images. However, these methods
+encounter several challenges during image generation, such as grid artifacts,
+exploding inverses, and suboptimal results due to a fixed sampling temperature.
+To overcome these issues, this work introduces a conditional learned prior to
+the inference phase of a flow-based SR model. This prior is a latent code
+predicted by our proposed latent module conditioned on the low-resolution
+image, which is then transformed by the flow model into an SR image. Our
+framework is designed to seamlessly integrate with any contemporary flow-based
+SR model without modifying its architecture or pre-trained weights. We evaluate
+the effectiveness of our proposed framework through extensive experiments and
+ablation analyses. The proposed framework successfully addresses all the
+inherent issues in flow-based SR models and enhances their performance in
+various SR scenarios. Our code is available at:
+https://github.com/liyuantsao/BFSR",cs.CV,"['cs.CV', 'cs.AI']"
+Video Frame Interpolation via Direct Synthesis with the Event-based Reference,Yuhan Liu · Yongjian Deng · Hao Chen · Zhen Yang, ,https://arxiv.org/abs/2404.18156,,2404.18156.pdf,Event-based Video Frame Interpolation with Edge Guided Motion Refinement,"Video frame interpolation, the process of synthesizing intermediate frames
+between sequential video frames, has made remarkable progress with the use of
+event cameras. These sensors, with microsecond-level temporal resolution, fill
+information gaps between frames by providing precise motion cues. However,
+contemporary Event-Based Video Frame Interpolation (E-VFI) techniques often
+neglect the fact that event data primarily supply high-confidence features at
+scene edges during multi-modal feature fusion, thereby diminishing the role of
+event signals in optical flow (OF) estimation and warping refinement. To
+address this overlooked aspect, we introduce an end-to-end E-VFI learning
+method (referred to as EGMR) to efficiently utilize edge features from event
+signals for motion flow and warping enhancement. Our method incorporates an
+Edge Guided Attentive (EGA) module, which rectifies estimated video motion
+through attentive aggregation based on the local correlation of multi-modal
+features in a coarse-to-fine strategy. Moreover, given that event data can
+provide accurate visual references at scene edges between consecutive frames,
+we introduce a learned visibility map derived from event data to adaptively
+mitigate the occlusion problem in the warping refinement process. Extensive
+experiments on both synthetic and real datasets show the effectiveness of the
+proposed approach, demonstrating its potential for higher quality video frame
+interpolation.",cs.CV,['cs.CV']
+Universal Robustness via Median Random Smoothing for Real-World Super-Resolution,Zakariya Chaouai · Mohamed Tamaazousti, ,https://arxiv.org/abs/2405.14934,,2405.14934.pdf,Universal Robustness via Median Randomized Smoothing for Real-World Super-Resolution,"Most of the recent literature on image Super-Resolution (SR) can be
+classified into two main approaches. The first one involves learning a
+corruption model tailored to a specific dataset, aiming to mimic the noise and
+corruption in low-resolution images, such as sensor noise. However, this
+approach is data-specific, tends to lack adaptability, and its accuracy
+diminishes when faced with unseen types of image corruptions. A second and more
+recent approach, referred to as Robust Super-Resolution (RSR), proposes to
+improve real-world SR by harnessing the generalization capabilities of a model
+by making it robust to adversarial attacks. To delve further into this second
+approach, our paper explores the universality of various methods for enhancing
+the robustness of deep learning SR models. In other words, we inquire: ""Which
+robustness method exhibits the highest degree of adaptability when dealing with
+a wide range of adversarial attacks ?"". Our extensive experimentation on both
+synthetic and real-world images empirically demonstrates that median randomized
+smoothing (MRS) is more general in terms of robustness compared to adversarial
+learning techniques, which tend to focus on specific types of attacks.
+Furthermore, as expected, we also illustrate that the proposed universal robust
+method enables the SR model to handle standard corruptions more effectively,
+such as blur and Gaussian noise, and notably, corruptions naturally present in
+real-world images. These results support the significance of shifting the
+paradigm in the development of real-world SR methods towards RSR, especially
+via MRS.",eess.IV,"['eess.IV', 'cs.CV']"
+AAMDM: Accelerated Auto-regressive Motion Diffusion Model,Tianyu Li · Calvin Zhuhan Qiao · Ren Guanqiao · KangKang Yin · Sehoon Ha, ,https://arxiv.org/abs/2401.06146,,2401.06146.pdf,AAMDM: Accelerated Auto-regressive Motion Diffusion Model,"Interactive motion synthesis is essential in creating immersive experiences
+in entertainment applications, such as video games and virtual reality.
+However, generating animations that are both high-quality and contextually
+responsive remains a challenge. Traditional techniques in the game industry can
+produce high-fidelity animations but suffer from high computational costs and
+poor scalability. Trained neural network models alleviate the memory and speed
+issues, yet fall short on generating diverse motions. Diffusion models offer
+diverse motion synthesis with low memory usage, but require expensive reverse
+diffusion processes. This paper introduces the Accelerated Auto-regressive
+Motion Diffusion Model (AAMDM), a novel motion synthesis framework designed to
+achieve quality, diversity, and efficiency all together. AAMDM integrates
+Denoising Diffusion GANs as a fast Generation Module, and an Auto-regressive
+Diffusion Model as a Polishing Module. Furthermore, AAMDM operates in a
+lower-dimensional embedded space rather than the full-dimensional pose space,
+which reduces the training complexity as well as further improves the
+performance. We show that AAMDM outperforms existing methods in motion quality,
+diversity, and runtime efficiency, through comprehensive quantitative analyses
+and visual comparisons. We also demonstrate the effectiveness of each
+algorithmic component through ablation studies.",cs.CV,"['cs.CV', 'cs.GR']"
+SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion,Hsuan-I Ho · Jie Song · Otmar Hilliges,https://ait.ethz.ch/sith,https://arxiv.org/abs/2311.15855,,2311.15855.pdf,SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion,"A long-standing goal of 3D human reconstruction is to create lifelike and
+fully detailed 3D humans from single-view images. The main challenge lies in
+inferring unknown body shapes, appearances, and clothing details in areas not
+visible in the images. To address this, we propose SiTH, a novel pipeline that
+uniquely integrates an image-conditioned diffusion model into a 3D mesh
+reconstruction workflow. At the core of our method lies the decomposition of
+the challenging single-view reconstruction problem into generative
+hallucination and reconstruction subproblems. For the former, we employ a
+powerful generative diffusion model to hallucinate unseen back-view appearance
+based on the input images. For the latter, we leverage skinned body meshes as
+guidance to recover full-body texture meshes from the input and back-view
+images. SiTH requires as few as 500 3D human scans for training while
+maintaining its generality and robustness to diverse images. Extensive
+evaluations on two 3D human benchmarks, including our newly created one,
+highlighted our method's superior accuracy and perceptual quality in 3D
+textured human reconstruction. Our code and evaluation benchmark are available
+at https://ait.ethz.ch/sith",cs.CV,['cs.CV']
+HUGS: Human Gaussian Splatting,Muhammed Kocabas · Jen-Hao Rick Chang · James Gabriel · Oncel Tuzel · Anurag Ranjan,https://machinelearning.apple.com/research/hugs,https://arxiv.org/abs/2311.17910v1,,2311.17910v1.pdf,HUGS: Human Gaussian Splats,"Recent advances in neural rendering have improved both training and rendering
+times by orders of magnitude. While these methods demonstrate state-of-the-art
+quality and speed, they are designed for photogrammetry of static scenes and do
+not generalize well to freely moving humans in the environment. In this work,
+we introduce Human Gaussian Splats (HUGS) that represents an animatable human
+together with the scene using 3D Gaussian Splatting (3DGS). Our method takes
+only a monocular video with a small number of (50-100) frames, and it
+automatically learns to disentangle the static scene and a fully animatable
+human avatar within 30 minutes. We utilize the SMPL body model to initialize
+the human Gaussians. To capture details that are not modeled by SMPL (e.g.
+cloth, hairs), we allow the 3D Gaussians to deviate from the human body model.
+Utilizing 3D Gaussians for animated humans brings new challenges, including the
+artifacts created when articulating the Gaussians. We propose to jointly
+optimize the linear blend skinning weights to coordinate the movements of
+individual Gaussians during animation. Our approach enables novel-pose
+synthesis of human and novel view synthesis of both the human and the scene. We
+achieve state-of-the-art rendering quality with a rendering speed of 60 FPS
+while being ~100x faster to train over previous work. Our code will be
+announced here: https://github.com/apple/ml-hugs",cs.CV,"['cs.CV', 'cs.GR']"
+SpiderMatch: 3D Shape Matching with Global Optimality and Geometric Consistency,Paul Roetzer · Florian Bernard, ,https://arxiv.org/abs/2310.08230,,2310.08230.pdf,Fast Discrete Optimisation for Geometrically Consistent 3D Shape Matching,"In this work we propose to combine the advantages of learning-based and
+combinatorial formalisms for 3D shape matching. While learning-based shape
+matching solutions lead to state-of-the-art matching performance, they do not
+ensure geometric consistency, so that obtained matchings are locally unsmooth.
+On the contrary, axiomatic methods allow to take geometric consistency into
+account by explicitly constraining the space of valid matchings. However,
+existing axiomatic formalisms are impractical since they do not scale to
+practically relevant problem sizes, or they require user input for the
+initialisation of non-convex optimisation problems. In this work we aim to
+close this gap by proposing a novel combinatorial solver that combines a unique
+set of favourable properties: our approach is (i) initialisation free, (ii)
+massively parallelisable powered by a quasi-Newton method, (iii) provides
+optimality gaps, and (iv) delivers decreased runtime and globally optimal
+results for many instances.",cs.CV,['cs.CV']
+Building Optimal Neural Architectures using Interpretable Knowledge,Keith Mills · Fred Han · Mohammad Salameh · Shengyao Lu · CHUNHUA ZHOU · Jiao He · Fengyu Sun · Di Niu,https://github.com/Ascend-Research/AutoBuild,https://arxiv.org/abs/2403.13293,,2403.13293.pdf,Building Optimal Neural Architectures using Interpretable Knowledge,"Neural Architecture Search is a costly practice. The fact that a search space
+can span a vast number of design choices with each architecture evaluation
+taking nontrivial overhead makes it hard for an algorithm to sufficiently
+explore candidate networks. In this paper, we propose AutoBuild, a scheme which
+learns to align the latent embeddings of operations and architecture modules
+with the ground-truth performance of the architectures they appear in. By doing
+so, AutoBuild is capable of assigning interpretable importance scores to
+architecture modules, such as individual operation features and larger macro
+operation sequences such that high-performance neural networks can be
+constructed without any need for search. Through experiments performed on
+state-of-the-art image classification, segmentation, and Stable Diffusion
+models, we show that by mining a relatively small set of evaluated
+architectures, AutoBuild can learn to build high-quality architectures directly
+or help to reduce search space to focus on relevant areas, finding better
+architectures that outperform both the original labeled ones and ones found by
+search baselines. Code available at
+https://github.com/Ascend-Research/AutoBuild",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Using Human Feedback to Fine-tune Diffusion Models  without Any Reward Model,Kai Yang · Jian Tao · Jiafei Lyu · Chunjiang Ge · Jiaxin Chen · Weihan Shen · Xiaolong Zhu · Xiu Li,https://github.com/yk7333/d3po/,https://arxiv.org/abs/2311.13231,,2311.13231.pdf,Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model,"Using reinforcement learning with human feedback (RLHF) has shown significant
+promise in fine-tuning diffusion models. Previous methods start by training a
+reward model that aligns with human preferences, then leverage RL techniques to
+fine-tune the underlying models. However, crafting an efficient reward model
+demands extensive datasets, optimal architecture, and manual hyperparameter
+tuning, making the process both time and cost-intensive. The direct preference
+optimization (DPO) method, effective in fine-tuning large language models,
+eliminates the necessity for a reward model. However, the extensive GPU memory
+requirement of the diffusion model's denoising process hinders the direct
+application of the DPO method. To address this issue, we introduce the Direct
+Preference for Denoising Diffusion Policy Optimization (D3PO) method to
+directly fine-tune diffusion models. The theoretical analysis demonstrates that
+although D3PO omits training a reward model, it effectively functions as the
+optimal reward model trained using human feedback data to guide the learning
+process. This approach requires no training of a reward model, proving to be
+more direct, cost-effective, and minimizing computational overhead. In
+experiments, our method uses the relative scale of objectives as a proxy for
+human preference, delivering comparable results to methods using ground-truth
+rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
+rates and generate safer images, overcoming challenges lacking robust reward
+models. Our code is publicly available at https://github.com/yk7333/D3PO.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization,Anna Kukleva · Fadime Sener · Edoardo Remelli · Bugra Tekin · Eric Sauser · Bernt Schiele · Shugao Ma, ,https://arxiv.org/abs/2403.19811,,2403.19811.pdf,X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization,"Lately, there has been growing interest in adapting vision-language models
+(VLMs) to image and third-person video classification due to their success in
+zero-shot recognition. However, the adaptation of these models to egocentric
+videos has been largely unexplored. To address this gap, we propose a simple
+yet effective cross-modal adaptation framework, which we call X-MIC. Using a
+video adapter, our pipeline learns to align frozen text embeddings to each
+egocentric video directly in the shared embedding space. Our novel adapter
+architecture retains and improves generalization of the pre-trained VLMs by
+disentangling learnable temporal modeling and frozen visual encoder. This
+results in an enhanced alignment of text embeddings to each egocentric video,
+leading to a significant improvement in cross-dataset generalization. We
+evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for
+fine-grained cross-dataset action generalization, demonstrating the
+effectiveness of our method. Code is available at
+https://github.com/annusha/xmic",cs.CV,['cs.CV']
+MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation,Zhicheng Zhang · Pancheng Zhao · Eunil Park · Jufeng Yang, ,https://arxiv.org/abs/2306.15876,,2306.15876.pdf,Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners,"Representation learning has been evolving from traditional supervised
+training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous
+works have demonstrated their pros and cons in specific scenarios, i.e., CL and
+supervised pre-training excel at capturing longer-range global patterns and
+enabling better feature discrimination, while MIM can introduce more local and
+diverse attention across all transformer layers. In this paper, we explore how
+to obtain a model that combines their strengths. We start by examining previous
+feature distillation and mask feature reconstruction methods and identify their
+limitations. We find that their increasing diversity mainly derives from the
+asymmetric designs, but these designs may in turn compromise the discrimination
+ability. In order to better obtain both discrimination and diversity, we
+propose a simple but effective Hybrid Distillation strategy, which utilizes
+both the supervised/CL teacher and the MIM teacher to jointly guide the student
+model. Hybrid Distill imitates the token relations of the MIM teacher to
+alleviate attention collapse, as well as distills the feature maps of the
+supervised/CL teacher to enable discrimination. Furthermore, a progressive
+redundant token masking strategy is also utilized to reduce the distilling
+costs and avoid falling into local optima. Experiment results prove that Hybrid
+Distill can achieve superior performance on different benchmarks.",cs.CV,['cs.CV']
+Monocular Identity-Conditioned Facial Reflectance Reconstruction,Xingyu Ren · Jiankang Deng · Yuhao Cheng · Jia Guo · Chao Ma · Yichao Yan · Wenhan Zhu · Xiaokang Yang,https://xingyuren.github.io/id2reflectance/,https://arxiv.org/abs/2404.00301,,2404.00301.pdf,Monocular Identity-Conditioned Facial Reflectance Reconstruction,"Recent 3D face reconstruction methods have made remarkable advancements, yet
+there remain huge challenges in monocular high-quality facial reflectance
+reconstruction. Existing methods rely on a large amount of light-stage captured
+data to learn facial reflectance models. However, the lack of subject diversity
+poses challenges in achieving good generalization and widespread applicability.
+In this paper, we learn the reflectance prior in image space rather than UV
+space and present a framework named ID2Reflectance. Our framework can directly
+estimate the reflectance maps of a single image while using limited reflectance
+data for training. Our key insight is that reflectance data shares facial
+structures with RGB faces, which enables obtaining expressive facial prior from
+inexpensive RGB data thus reducing the dependency on reflectance data. We first
+learn a high-quality prior for facial reflectance. Specifically, we pretrain
+multi-domain facial feature codebooks and design a codebook fusion method to
+align the reflectance and RGB domains. Then, we propose an identity-conditioned
+swapping module that injects facial identity from the target image into the
+pre-trained autoencoder to modify the identity of the source reflectance image.
+Finally, we stitch multi-view swapped reflectance images to obtain renderable
+assets. Extensive experiments demonstrate that our method exhibits excellent
+generalization capability and achieves state-of-the-art facial reflectance
+reconstruction results for in-the-wild faces. Our project page is
+https://xingyuren.github.io/id2reflectance/.",cs.CV,['cs.CV']
+CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing,Ajian Liu · Shuai Xue · Gan Jianwen · Jun Wan · Yanyan Liang · Jiankang Deng · Sergio Escalera · Zhen Lei, ,https://arxiv.org/abs/2403.14333,,2403.14333.pdf,CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing,"Domain generalization (DG) based Face Anti-Spoofing (FAS) aims to improve the
+model's performance on unseen domains. Existing methods either rely on domain
+labels to align domain-invariant feature spaces, or disentangle generalizable
+features from the whole sample, which inevitably lead to the distortion of
+semantic feature structures and achieve limited generalization. In this work,
+we make use of large-scale VLMs like CLIP and leverage the textual feature to
+dynamically adjust the classifier's weights for exploring generalizable visual
+features. Specifically, we propose a novel Class Free Prompt Learning (CFPL)
+paradigm for DG FAS, which utilizes two lightweight transformers, namely
+Content Q-Former (CQF) and Style Q-Former (SQF), to learn the different
+semantic prompts conditioned on content and style features by using a set of
+learnable query vectors, respectively. Thus, the generalizable prompt can be
+learned by two improvements: (1) A Prompt-Text Matched (PTM) supervision is
+introduced to ensure CQF learns visual representation that is most informative
+of the content description. (2) A Diversified Style Prompt (DSP) technology is
+proposed to diversify the learning of style prompts by mixing feature
+statistics between instance-specific styles. Finally, the learned text features
+modulate visual features to generalization through the designed Prompt
+Modulation (PM). Extensive experiments show that the CFPL is effective and
+outperforms the state-of-the-art methods on several cross-domain datasets.",cs.CV,['cs.CV']
+BEM: Balanced and Entropy-based Mix for Long-Tailed Semi-Supervised Learning,Hongwei Zheng · Linyuan Zhou · Han Li · Jinming Su · Xiaoming Wei · Xu Xiaoming, ,https://arxiv.org/abs/2404.01179,,2404.01179.pdf,BEM: Balanced and Entropy-based Mix for Long-Tailed Semi-Supervised Learning,"Data mixing methods play a crucial role in semi-supervised learning (SSL),
+but their application is unexplored in long-tailed semi-supervised learning
+(LTSSL). The primary reason is that the in-batch mixing manner fails to address
+class imbalance. Furthermore, existing LTSSL methods mainly focus on
+re-balancing data quantity but ignore class-wise uncertainty, which is also
+vital for class balance. For instance, some classes with sufficient samples
+might still exhibit high uncertainty due to indistinguishable features. To this
+end, this paper introduces the Balanced and Entropy-based Mix (BEM), a
+pioneering mixing approach to re-balance the class distribution of both data
+quantity and uncertainty. Specifically, we first propose a class balanced mix
+bank to store data of each class for mixing. This bank samples data based on
+the estimated quantity distribution, thus re-balancing data quantity. Then, we
+present an entropy-based learning approach to re-balance class-wise
+uncertainty, including entropy-based sampling strategy, entropy-based selection
+module, and entropy-based class balanced loss. Our BEM first leverages data
+mixing for improving LTSSL, and it can also serve as a complement to the
+existing re-balancing methods. Experimental results show that BEM significantly
+enhances various LTSSL frameworks and achieves state-of-the-art performances
+across multiple benchmarks.",cs.CV,"['cs.CV', 'cs.LG']"
+Relightable Gaussian Codec Avatars,Shunsuke Saito · Gabriel Schwartz · Tomas Simon · Junxuan Li · Giljoo Nam, ,https://arxiv.org/abs/2312.03704,,2312.03704.pdf,Relightable Gaussian Codec Avatars,"The fidelity of relighting is bounded by both geometry and appearance
+representations. For geometry, both mesh and volumetric approaches have
+difficulty modeling intricate structures like 3D hair geometry. For appearance,
+existing relighting models are limited in fidelity and often too slow to render
+in real-time with high-resolution continuous environments. In this work, we
+present Relightable Gaussian Codec Avatars, a method to build high-fidelity
+relightable head avatars that can be animated to generate novel expressions.
+Our geometry model based on 3D Gaussians can capture 3D-consistent
+sub-millimeter details such as hair strands and pores on dynamic face
+sequences. To support diverse materials of human heads such as the eyes, skin,
+and hair in a unified manner, we present a novel relightable appearance model
+based on learnable radiance transfer. Together with global illumination-aware
+spherical harmonics for the diffuse components, we achieve real-time relighting
+with all-frequency reflections using spherical Gaussians. This appearance model
+can be efficiently relit under both point light and continuous illumination. We
+further improve the fidelity of eye reflections and enable explicit gaze
+control by introducing relightable explicit eye models. Our method outperforms
+existing approaches without compromising real-time performance. We also
+demonstrate real-time relighting of avatars on a tethered consumer VR headset,
+showcasing the efficiency and fidelity of our avatars.",cs.GR,"['cs.GR', 'cs.CV']"
+4D-DRESS: A 4D Dataset of Real-World Human Clothing With Semantic Annotations,Wenbo Wang · Hsuan-I Ho · Chen Guo · Boxiang Rong · Artur Grigorev · Jie Song · Juan Jose Zarate · Otmar Hilliges,https://ait.ethz.ch/4d-dress,https://arxiv.org/abs/2404.18630,,2404.18630.pdf,4D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotations,"The studies of human clothing for digital avatars have predominantly relied
+on synthetic datasets. While easy to collect, synthetic data often fall short
+in realism and fail to capture authentic clothing dynamics. Addressing this
+gap, we introduce 4D-DRESS, the first real-world 4D dataset advancing human
+clothing research with its high-quality 4D textured scans and garment meshes.
+4D-DRESS captures 64 outfits in 520 human motion sequences, amounting to 78k
+textured scans. Creating a real-world clothing dataset is challenging,
+particularly in annotating and segmenting the extensive and complex 4D human
+scans. To address this, we develop a semi-automatic 4D human parsing pipeline.
+We efficiently combine a human-in-the-loop process with automation to
+accurately label 4D scans in diverse garments and body movements. Leveraging
+precise annotations and high-quality garment meshes, we establish several
+benchmarks for clothing simulation and reconstruction. 4D-DRESS offers
+realistic and challenging data that complements synthetic sources, paving the
+way for advancements in research of lifelike human clothing. Website:
+https://ait.ethz.ch/4d-dress.",cs.CV,['cs.CV']
+DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback,Yangyi Chen · Karan Sikka · Michael Cogswell · Heng Ji · Ajay Divakaran, ,https://arxiv.org/abs/2311.10081,,2311.10081.pdf,DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback,"We present DRESS, a large vision language model (LVLM) that innovatively
+exploits Natural Language feedback (NLF) from Large Language Models to enhance
+its alignment and interactions by addressing two key limitations in the
+state-of-the-art LVLMs. First, prior LVLMs generally rely only on the
+instruction finetuning stage to enhance alignment with human preferences.
+Without incorporating extra feedback, they are still prone to generate
+unhelpful, hallucinated, or harmful responses. Second, while the visual
+instruction tuning data is generally structured in a multi-turn dialogue
+format, the connections and dependencies among consecutive conversational turns
+are weak. This reduces the capacity for effective multi-turn interactions. To
+tackle these, we propose a novel categorization of the NLF into two key types:
+critique and refinement. The critique NLF identifies the strengths and
+weaknesses of the responses and is used to align the LVLMs with human
+preferences. The refinement NLF offers concrete suggestions for improvement and
+is adopted to improve the interaction ability of the LVLMs-- which focuses on
+LVLMs' ability to refine responses by incorporating feedback in multi-turn
+interactions. To address the non-differentiable nature of NLF, we generalize
+conditional reinforcement learning for training. Our experimental results
+demonstrate that DRESS can generate more helpful (9.76%), honest (11.52%), and
+harmless (21.03%) responses, and more effectively learn from feedback during
+multi-turn interactions compared to SOTA LVMLs.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Gaussian Shadow Casting for Neural Characters,Luis Bolanos · Shih-Yang Su · Helge Rhodin, ,https://arxiv.org/abs/2401.06116v1,,2401.06116v1.pdf,Gaussian Shadow Casting for Neural Characters,"Neural character models can now reconstruct detailed geometry and texture
+from video, but they lack explicit shadows and shading, leading to artifacts
+when generating novel views and poses or during relighting. It is particularly
+difficult to include shadows as they are a global effect and the required
+casting of secondary rays is costly. We propose a new shadow model using a
+Gaussian density proxy that replaces sampling with a simple analytic formula.
+It supports dynamic motion and is tailored for shadow computation, thereby
+avoiding the affine projection approximation and sorting required by the
+closely related Gaussian splatting. Combined with a deferred neural rendering
+model, our Gaussian shadows enable Lambertian shading and shadow casting with
+minimal overhead. We demonstrate improved reconstructions, with better
+separation of albedo, shading, and shadows in challenging outdoor scenes with
+direct sun light and hard shadows. Our method is able to optimize the light
+direction without any input from the user. As a result, novel poses have fewer
+shadow artifacts and relighting in novel scenes is more realistic compared to
+the state-of-the-art methods, providing new ways to pose neural characters in
+novel environments, increasing their applicability.",cs.CV,['cs.CV']
+CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update,Zhi Gao · Yuntao Du. · Xintong Zhang · Xiaojian Ma · Wenjuan Han · Song-Chun Zhu · Qing Li, ,https://arxiv.org/abs/2312.10908,,2312.10908.pdf,CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update,"Utilizing large language models (LLMs) to compose off-the-shelf visual tools
+represents a promising avenue of research for developing robust visual
+assistants capable of addressing diverse visual tasks. However, these methods
+often overlook the potential for continual learning, typically by freezing the
+utilized tools, thus limiting their adaptation to environments requiring new
+knowledge. To tackle this challenge, we propose CLOVA, a Closed-Loop Visual
+Assistant, which operates within a framework encompassing inference,
+reflection, and learning phases. During the inference phase, LLMs generate
+programs and execute corresponding tools to complete assigned tasks. In the
+reflection phase, a multimodal global-local reflection scheme analyzes human
+feedback to determine which tools require updating. Lastly, the learning phase
+employs three flexible approaches to automatically gather training data and
+introduces a novel prompt tuning scheme to update the tools, allowing CLOVA to
+efficiently acquire new knowledge. Experimental findings demonstrate that CLOVA
+surpasses existing tool-usage methods by 5% in visual question answering and
+multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image
+editing. These results underscore the significance of the continual learning
+capability in general visual assistants.",cs.CV,['cs.CV']
+Enhancing the Power of OOD Detection via Sample-Aware Model Selection,Feng Xue · Zi He · Yuan Zhang · Chuanlong Xie · Zhenguo Li · Falong Tan, ,,https://www.youtube.com/watch?v=XNso9qsWxHo,,,,,nan
+Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation,Zhekai Du · Xinyao Li · Fengling Li · Ke Lu · Lei Zhu · Jingjing Li,https://github.com/TL-UESTC/DAMP,https://arxiv.org/abs/2403.02899,,2403.02899.pdf,Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation,"Conventional Unsupervised Domain Adaptation (UDA) strives to minimize
+distribution discrepancy between domains, which neglects to harness rich
+semantics from data and struggles to handle complex domain shifts. A promising
+technique is to leverage the knowledge of large-scale pre-trained
+vision-language models for more guided adaptation. Despite some endeavors,
+current methods often learn textual prompts to embed domain semantics for
+source and target domains separately and perform classification within each
+domain, limiting cross-domain knowledge transfer. Moreover, prompting only the
+language branch lacks flexibility to adapt both modalities dynamically. To
+bridge this gap, we propose Domain-Agnostic Mutual Prompting (DAMP) to exploit
+domain-invariant semantics by mutually aligning visual and textual embeddings.
+Specifically, the image contextual information is utilized to prompt the
+language branch in a domain-agnostic and instance-conditioned way. Meanwhile,
+visual prompts are imposed based on the domain-agnostic textual prompt to
+elicit domain-invariant visual embeddings. These two branches of prompts are
+learned mutually with a cross-attention module and regularized with a
+semantic-consistency loss and an instance-discrimination contrastive loss.
+Experiments on three UDA benchmarks demonstrate the superiority of DAMP over
+state-of-the-art approaches.",cs.AI,['cs.AI']
+DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction,Weiyi Lv · Yuhang Huang · NING Zhang · Ruei-Sung Lin · Mei Han · Dan Zeng,https://diffmot.github.io/,https://arxiv.org/abs/2403.02075,,2403.02075.pdf,DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction,"In Multiple Object Tracking, objects often exhibit non-linear motion of
+acceleration and deceleration, with irregular direction changes.
+Tacking-by-detection (TBD) trackers with Kalman Filter motion prediction work
+well in pedestrian-dominant scenarios but fall short in complex situations when
+multiple objects perform non-linear and diverse motion simultaneously. To
+tackle the complex non-linear motion, we propose a real-time diffusion-based
+MOT approach named DiffMOT. Specifically, for the motion predictor component,
+we propose a novel Decoupled Diffusion-based Motion Predictor (D$^2$MP). It
+models the entire distribution of various motion presented by the data as a
+whole. It also predicts an individual object's motion conditioning on an
+individual's historical motion information. Furthermore, it optimizes the
+diffusion process with much fewer sampling steps. As a MOT tracker, the DiffMOT
+is real-time at 22.7FPS, and also outperforms the state-of-the-art on
+DanceTrack and SportsMOT datasets with $62.3\%$ and $76.2\%$ in HOTA metrics,
+respectively. To the best of our knowledge, DiffMOT is the first to introduce a
+diffusion probabilistic model into the MOT to tackle non-linear motion
+prediction.",cs.CV,['cs.CV']
+Dynamic Support Information Mining for Category-Agnostic Pose Estimation,Pengfei Ren · Yuanyuan Gao · Haifeng Sun · Qi Qi · Jingyu Wang · Jianxin Liao, ,https://arxiv.org/abs/2403.13647,,,Meta-Point Learning and Refining for Category-Agnostic Pose Estimation,"Category-agnostic pose estimation (CAPE) aims to predict keypoints for
+arbitrary classes given a few support images annotated with keypoints. Existing
+methods only rely on the features extracted at support keypoints to predict or
+refine the keypoints on query image, but a few support feature vectors are
+local and inadequate for CAPE. Considering that human can quickly perceive
+potential keypoints of arbitrary objects, we propose a novel framework for CAPE
+based on such potential keypoints (named as meta-points). Specifically, we
+maintain learnable embeddings to capture inherent information of various
+keypoints, which interact with image feature maps to produce meta-points
+without any support. The produced meta-points could serve as meaningful
+potential keypoints for CAPE. Due to the inevitable gap between inherency and
+annotation, we finally utilize the identities and details offered by support
+keypoints to assign and refine meta-points to desired keypoints in query image.
+In addition, we propose a progressive deformable point decoder and a slacked
+regression loss for better prediction and supervision. Our novel framework not
+only reveals the inherency of keypoints but also outperforms existing methods
+of CAPE. Comprehensive experiments and in-depth studies on large-scale MP-100
+dataset demonstrate the effectiveness of our framework.",cs.CV,['cs.CV']
+Implicit Event-RGBD Neural SLAM,Delin Qu · Chi Yan · Dong Wang · Jie Yin · Qizhi Chen · Dan Xu · Yiting Zhang · Bin Zhao · Xuelong Li,https://delinqu.github.io/EN-SLAM,https://arxiv.org/abs/2311.11013,,2311.11013.pdf,Implicit Event-RGBD Neural SLAM,"Implicit neural SLAM has achieved remarkable progress recently. Nevertheless,
+existing methods face significant challenges in non-ideal scenarios, such as
+motion blur or lighting variation, which often leads to issues like convergence
+failures, localization drifts, and distorted mapping. To address these
+challenges, we propose EN-SLAM, the first event-RGBD implicit neural SLAM
+framework, which effectively leverages the high rate and high dynamic range
+advantages of event data for tracking and mapping. Specifically, EN-SLAM
+proposes a differentiable CRF (Camera Response Function) rendering technique to
+generate distinct RGB and event camera data via a shared radiance field, which
+is optimized by learning a unified implicit representation with the captured
+event and RGBD supervision. Moreover, based on the temporal difference property
+of events, we propose a temporal aggregating optimization strategy for the
+event joint tracking and global bundle adjustment, capitalizing on the
+consecutive difference constraints of events, significantly enhancing tracking
+accuracy and robustness. Finally, we construct the simulated dataset
+DEV-Indoors and real captured dataset DEV-Reals containing 6 scenes, 17
+sequences with practical motion blur and lighting changes for evaluations.
+Experimental results show that our method outperforms the SOTA methods in both
+tracking ATE and mapping ACC with a real-time 17 FPS in various challenging
+environments. Project page: https://delinqu.github.io/EN-SLAM.",cs.CV,['cs.CV']
+DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting,Demin Yu · Xutao Li · Yunming Ye · Baoquan Zhang · Luo Chuyao · Kuai Dai · wangrui · Chenxunlai, ,https://arxiv.org/abs/2312.06734,,2312.06734.pdf,DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting,"Precipitation nowcasting is an important spatio-temporal prediction task to
+predict the radar echoes sequences based on current observations, which can
+serve both meteorological science and smart city applications. Due to the
+chaotic evolution nature of the precipitation systems, it is a very challenging
+problem. Previous studies address the problem either from the perspectives of
+deterministic modeling or probabilistic modeling. However, their predictions
+suffer from the blurry, high-value echoes fading away and position inaccurate
+issues. The root reason of these issues is that the chaotic evolutionary
+precipitation systems are not appropriately modeled. Inspired by the nature of
+the systems, we propose to decompose and model them from the perspective of
+global deterministic motion and local stochastic variations with residual
+mechanism. A unified and flexible framework that can equip any type of
+spatio-temporal models is proposed based on residual diffusion, which
+effectively tackles the shortcomings of previous methods. Extensive
+experimental results on four publicly available radar datasets demonstrate the
+effectiveness and superiority of the proposed framework, compared to
+state-of-the-art techniques. Our code is publicly available at
+https://github.com/DeminYu98/DiffCast.",cs.CV,['cs.CV']
+Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions,Zeyu Han · Fangrui Zhu · Qianru Lao · Huaizu Jiang, ,https://arxiv.org/abs/2311.17048,,2311.17048.pdf,Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions,"Zero-shot referring expression comprehension aims at localizing bounding
+boxes in an image corresponding to provided textual prompts, which requires:
+(i) a fine-grained disentanglement of complex visual scene and textual context,
+and (ii) a capacity to understand relationships among disentangled entities.
+Unfortunately, existing large vision-language alignment (VLA) models, e.g.,
+CLIP, struggle with both aspects so cannot be directly used for this task. To
+mitigate this gap, we leverage large foundation models to disentangle both
+images and texts into triplets in the format of (subject, predicate, object).
+After that, grounding is accomplished by calculating the structural similarity
+matrix between visual and textual triplets with a VLA model, and subsequently
+propagate it to an instance-level similarity matrix. Furthermore, to equip VLA
+models with the ability of relationship understanding, we design a
+triplet-matching objective to fine-tune the VLA models on a collection of
+curated dataset containing abundant entity relationships. Experiments
+demonstrate that our visual grounding performance increase of up to 19.5% over
+the SOTA zero-shot model on RefCOCO/+/g. On the more challenging Who's Waldo
+dataset, our zero-shot approach achieves comparable accuracy to the fully
+supervised model. Code is available at
+https://github.com/Show-han/Zeroshot_REC.",cs.CV,['cs.CV']
+Self-Supervised Multi-Object Tracking with Path Consistency,Zijia Lu · Bing Shuai · Yanbei Chen · Zhenlin Xu · Davide Modolo, ,https://arxiv.org/abs/2404.05136,,2404.05136.pdf,Self-Supervised Multi-Object Tracking with Path Consistency,"In this paper, we propose a novel concept of path consistency to learn robust
+object matching without using manual object identity supervision. Our key idea
+is that, to track a object through frames, we can obtain multiple different
+association results from a model by varying the frames it can observe, i.e.,
+skipping frames in observation. As the differences in observations do not alter
+the identities of objects, the obtained association results should be
+consistent. Based on this rationale, we generate multiple observation paths,
+each specifying a different set of frames to be skipped, and formulate the Path
+Consistency Loss that enforces the association results are consistent across
+different observation paths. We use the proposed loss to train our object
+matching model with only self-supervision. By extensive experiments on three
+tracking datasets (MOT17, PersonPath22, KITTI), we demonstrate that our method
+outperforms existing unsupervised methods with consistent margins on various
+evaluation metrics, and even achieves performance close to supervised methods.",cs.CV,"['cs.CV', 'cs.AI']"
+Correcting Diffusion Generation through Resampling,Yujian Liu · Yang Zhang · Tommi Jaakkola · Shiyu Chang, ,https://arxiv.org/abs/2312.06038,,2312.06038.pdf,Correcting Diffusion Generation through Resampling,"Despite diffusion models' superior capabilities in modeling complex
+distributions, there are still non-trivial distributional discrepancies between
+generated and ground-truth images, which has resulted in several notable
+problems in image generation, including missing object errors in text-to-image
+generation and low image quality. Existing methods that attempt to address
+these problems mostly do not tend to address the fundamental cause behind these
+problems, which is the distributional discrepancies, and hence achieve
+sub-optimal results. In this paper, we propose a particle filtering framework
+that can effectively address both problems by explicitly reducing the
+distributional discrepancies. Specifically, our method relies on a set of
+external guidance, including a small set of real images and a pre-trained
+object detector, to gauge the distribution gap, and then design the resampling
+weight accordingly to correct the gap. Experiments show that our methods can
+effectively correct missing object errors and improve image quality in various
+image generation tasks. Notably, our method outperforms the existing strongest
+baseline by 5% in object occurrence and 1.0 in FID on MS-COCO. Our code is
+publicly available at
+https://github.com/UCSB-NLP-Chang/diffusion_resampling.git.",cs.CV,"['cs.CV', 'cs.LG']"
+Exploring Orthogonality in Open World Object Detection,Zhicheng Sun · Jinghan Li · Yadong Mu,https://github.com/feifeiobama/OrthogonalDet,,https://www.youtube.com/watch?v=fNDF2pIWbmM,,,,,nan
+YolOOD: Utilizing Object Detection Concepts for Multi-Label Out-of-Distribution Detection,Alon Zolfi · Guy AmiT · Amit Baras · Satoru Koda · Ikuya Morikawa · Yuval Elovici · Asaf Shabtai, ,https://arxiv.org/abs/2402.18162,,2402.18162.pdf,Out-of-Distribution Detection using Neural Activation Prior,"Out-of-distribution detection (OOD) is a crucial technique for deploying
+machine learning models in the real world to handle the unseen scenarios. In
+this paper, we first propose a simple yet effective Neural Activation Prior
+(NAP) for OOD detection. Our neural activation prior is based on a key
+observation that, for a channel before the global pooling layer of a fully
+trained neural network, the probability of a few neurons being activated with a
+large response by an in-distribution (ID) sample is significantly higher than
+that by an OOD sample. An intuitive explanation is that for a model fully
+trained on ID dataset, each channel would play a role in detecting a certain
+pattern in the ID dataset, and a few neurons can be activated with a large
+response when the pattern is detected in an input sample. Then, a new scoring
+function based on this prior is proposed to highlight the role of these
+strongly activated neurons in OOD detection. Our approach is plug-and-play and
+does not lead to any performance degradation on ID data classification and
+requires no extra training or statistics from training or external datasets.
+Notice that previous methods primarily rely on post-global-pooling features of
+the neural networks, while the within-channel distribution information we
+leverage would be discarded by the global pooling operator. Consequently, our
+method is orthogonal to existing approaches and can be effectively combined
+with them in various applications. Experimental results show that our method
+achieves the state-of-the-art performance on CIFAR benchmark and ImageNet
+dataset, which demonstrates the power of the proposed prior. Finally, we extend
+our method to Transformers and the experimental findings indicate that NAP can
+also significantly enhance the performance of OOD detection on Transformers,
+thereby demonstrating the broad applicability of this prior knowledge.",cs.CV,['cs.CV']
+3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling,Chaokang Jiang · Guangming Wang · Jiuming Liu · Hesheng Wang · Zhuang Ma · Zhenqiang Liu · LIANG · Yi Shan · Dalong Du,https://jiangchaokang.github.io/3DSFLabelling-Page/,https://arxiv.org/abs/2402.18146,,2402.18146.pdf,3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling,"Learning 3D scene flow from LiDAR point clouds presents significant
+difficulties, including poor generalization from synthetic datasets to real
+scenes, scarcity of real-world 3D labels, and poor performance on real sparse
+LiDAR point clouds. We present a novel approach from the perspective of
+auto-labelling, aiming to generate a large number of 3D scene flow pseudo
+labels for real-world LiDAR point clouds. Specifically, we employ the
+assumption of rigid body motion to simulate potential object-level rigid
+movements in autonomous driving scenarios. By updating different motion
+attributes for multiple anchor boxes, the rigid motion decomposition is
+obtained for the whole scene. Furthermore, we developed a novel 3D scene flow
+data augmentation method for global and local motion. By perfectly synthesizing
+target point clouds based on augmented motion parameters, we easily obtain lots
+of 3D scene flow labels in point clouds highly consistent with real scenarios.
+On multiple real-world datasets including LiDAR KITTI, nuScenes, and Argoverse,
+our method outperforms all previous supervised and unsupervised methods without
+requiring manual labelling. Impressively, our method achieves a tenfold
+reduction in EPE3D metric on the LiDAR KITTI dataset, reducing it from $0.190m$
+to a mere $0.008m$ error.",cs.CV,['cs.CV']
+Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling,Leon Sick · Dominik Engel · Pedro Hermosilla · Timo Ropinski,https://leonsick.github.io/depthg/,https://arxiv.org/abs/2309.12378,,2309.12378.pdf,Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling,"Traditionally, training neural networks to perform semantic segmentation
+required expensive human-made annotations. But more recently, advances in the
+field of unsupervised learning have made significant progress on this issue and
+towards closing the gap to supervised algorithms. To achieve this, semantic
+knowledge is distilled by learning to correlate randomly sampled features from
+images across an entire dataset. In this work, we build upon these advances by
+incorporating information about the structure of the scene into the training
+process through the use of depth information. We achieve this by (1) learning
+depth-feature correlation by spatially correlate the feature maps with the
+depth maps to induce knowledge about the structure of the scene and (2)
+implementing farthest-point sampling to more effectively select relevant
+features by utilizing 3D sampling techniques on depth information of the scene.
+Finally, we demonstrate the effectiveness of our technical contributions
+through extensive experimentation and present significant improvements in
+performance across multiple benchmark datasets.",cs.CV,['cs.CV']
+Super-Resolution Reconstruction from Bayer-Pattern Spike Streams,Yanchen Dong · Ruiqin Xiong · Jian Zhang · Zhaofei Yu · Xiaopeng Fan · Shuyuan Zhu · Tiejun Huang,https://github.com/csycdong/CSCSR,,https://ojs.aaai.org/index.php/AAAI/article/view/27924,,,,,nan
+Random Entangled Tokens for Adversarially Robust Vision Transformer,Huihui Gong · Minjing Dong · Siqi Ma · Seyit Camtepe · Surya Nepal · Chang Xu, ,https://arxiv.org/abs/2402.07183,,2402.07183.pdf,A Random Ensemble of Encrypted Vision Transformers for Adversarially Robust Defense,"Deep neural networks (DNNs) are well known to be vulnerable to adversarial
+examples (AEs). In previous studies, the use of models encrypted with a secret
+key was demonstrated to be robust against white-box attacks, but not against
+black-box ones. In this paper, we propose a novel method using the vision
+transformer (ViT) that is a random ensemble of encrypted models for enhancing
+robustness against both white-box and black-box attacks. In addition, a
+benchmark attack method, called AutoAttack, is applied to models to test
+adversarial robustness objectively. In experiments, the method was demonstrated
+to be robust against not only white-box attacks but also black-box ones in an
+image classification task on the CIFAR-10 and ImageNet datasets. The method was
+also compared with the state-of-the-art in a standardized benchmark for
+adversarial robustness, RobustBench, and it was verified to outperform
+conventional defenses in terms of clean accuracy and robust accuracy.",cs.AI,['cs.AI']
+PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding,Zhen Li · Mingdeng Cao · Xintao Wang · Zhongang Qi · Ming-Ming Cheng · Ying Shan,https://github.com/TencentARC/PhotoMaker,https://arxiv.org/abs/2312.04461,,2312.04461.pdf,PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding,"Recent advances in text-to-image generation have made remarkable progress in
+synthesizing realistic human photos conditioned on given text prompts. However,
+existing personalized generation methods cannot simultaneously satisfy the
+requirements of high efficiency, promising identity (ID) fidelity, and flexible
+text controllability. In this work, we introduce PhotoMaker, an efficient
+personalized text-to-image generation method, which mainly encodes an arbitrary
+number of input ID images into a stack ID embedding for preserving ID
+information. Such an embedding, serving as a unified ID representation, can not
+only encapsulate the characteristics of the same input ID comprehensively, but
+also accommodate the characteristics of different IDs for subsequent
+integration. This paves the way for more intriguing and practically valuable
+applications. Besides, to drive the training of our PhotoMaker, we propose an
+ID-oriented data construction pipeline to assemble the training data. Under the
+nourishment of the dataset constructed through the proposed pipeline, our
+PhotoMaker demonstrates better ID preservation ability than test-time
+fine-tuning based methods, yet provides significant speed improvements,
+high-quality generation results, strong generalization capabilities, and a wide
+range of applications. Our project page is available at
+https://photo-maker.github.io/",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+Hierarchical Correlation Clustering and Tree Preserving Embedding,Morteza Haghir Chehreghani · Mostafa Haghir Chehreghani, ,https://arxiv.org/abs/2402.03587,,2402.03587.pdf,Information-Theoretic Active Correlation Clustering,"We study correlation clustering where the pairwise similarities are not known
+in advance. For this purpose, we employ active learning to query pairwise
+similarities in a cost-efficient way. We propose a number of effective
+information-theoretic acquisition functions based on entropy and information
+gain. We extensively investigate the performance of our methods in different
+settings and demonstrate their superior performance compared to the
+alternatives.",cs.LG,"['cs.LG', 'stat.ML']"
+Referring Image Editing: Object-level Image Editing via Referring Expressions,Chang Liu · Xiangtai Li · Henghui Ding, ,,https://link.springer.com/article/10.1007/s11063-024-11487-2,,,,,nan
+CORE-MPI: Consistency Object Removal with Embedding MultiPlane Image,Donggeun Yoon · Donghyeon Cho, ,https://arxiv.org/abs/2310.08092,,2310.08092.pdf,Consistent123: Improve Consistency for One Image to 3D Object Synthesis,"Large image diffusion models enable novel view synthesis with high quality
+and excellent zero-shot capability. However, such models based on
+image-to-image translation have no guarantee of view consistency, limiting the
+performance for downstream tasks like 3D reconstruction and image-to-3D
+generation. To empower consistency, we propose Consistent123 to synthesize
+novel views simultaneously by incorporating additional cross-view attention
+layers and the shared self-attention mechanism. The proposed attention
+mechanism improves the interaction across all synthesized views, as well as the
+alignment between the condition view and novel views. In the sampling stage,
+such architecture supports simultaneously generating an arbitrary number of
+views while training at a fixed length. We also introduce a progressive
+classifier-free guidance strategy to achieve the trade-off between texture and
+geometry for synthesized object views. Qualitative and quantitative experiments
+show that Consistent123 outperforms baselines in view consistency by a large
+margin. Furthermore, we demonstrate a significant improvement of Consistent123
+on varying downstream tasks, showing its great potential in the 3D generation
+field. The project page is available at consistent-123.github.io.",cs.CV,['cs.CV']
+WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models,Changhoon Kim · Kyle Min · Maitreya Patel · Sheng Cheng · 'YZ' Yezhou Yang, ,https://arxiv.org/abs/2306.04744,,2306.04744.pdf,WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models,"The rapid advancement of generative models, facilitating the creation of
+hyper-realistic images from textual descriptions, has concurrently escalated
+critical societal concerns such as misinformation. Although providing some
+mitigation, traditional fingerprinting mechanisms fall short in attributing
+responsibility for the malicious use of synthetic images. This paper introduces
+a novel approach to model fingerprinting that assigns responsibility for the
+generated images, thereby serving as a potential countermeasure to model
+misuse. Our method modifies generative models based on each user's unique
+digital fingerprint, imprinting a unique identifier onto the resultant content
+that can be traced back to the user. This approach, incorporating fine-tuning
+into Text-to-Image (T2I) tasks using the Stable Diffusion Model, demonstrates
+near-perfect attribution accuracy with a minimal impact on output quality.
+Through extensive evaluation, we show that our method outperforms baseline
+methods with an average improvement of 11\% in handling image post-processes.
+Our method presents a promising and novel avenue for accountable model
+distribution and responsible use. Our code is available in
+\url{https://github.com/kylemin/WOUAF}.",cs.CV,['cs.CV']
+Improving Unsupervised Hierarchical Representation with Reinforcement Learning,Ruyi An · Yewen Li · Xu He · Pengjie Gu · Mengchen Zhao · Dong Li · Jianye Hao · Bo An · Chaojie Wang · Mingyuan Zhou, ,,https://www2.scut.edu.cn/sse/2024/0226/c16789a534834/page.htm,,,,,nan
+Joint-Task Regularization for Partially Labeled Multi-Task Learning,Kento Nishi · Junsik Kim · Wanhua Li · Hanspeter Pfister,https://kentonishi.com/JTR-CVPR-2024/,https://arxiv.org/abs/2404.01976,,2404.01976.pdf,Joint-Task Regularization for Partially Labeled Multi-Task Learning,"Multi-task learning has become increasingly popular in the machine learning
+field, but its practicality is hindered by the need for large, labeled
+datasets. Most multi-task learning methods depend on fully labeled datasets
+wherein each input example is accompanied by ground-truth labels for all target
+tasks. Unfortunately, curating such datasets can be prohibitively expensive and
+impractical, especially for dense prediction tasks which require per-pixel
+labels for each image. With this in mind, we propose Joint-Task Regularization
+(JTR), an intuitive technique which leverages cross-task relations to
+simultaneously regularize all tasks in a single joint-task latent space to
+improve learning when data is not fully labeled for all tasks. JTR stands out
+from existing approaches in that it regularizes all tasks jointly rather than
+separately in pairs -- therefore, it achieves linear complexity relative to the
+number of tasks while previous methods scale quadratically. To demonstrate the
+validity of our approach, we extensively benchmark our method across a wide
+variety of partially labeled scenarios based on NYU-v2, Cityscapes, and
+Taskonomy.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations,Peng Dai · Yang Zhang · Tao Liu · ZhenFan · Tianyuan Du · Zhuo Su · Xiaozheng Zheng · Zeming Li,https://pico-ai-team.github.io/hmd-poser,https://arxiv.org/abs/2403.03561,,2403.03561.pdf,HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations,"It is especially challenging to achieve real-time human motion tracking on a
+standalone VR Head-Mounted Display (HMD) such as Meta Quest and PICO. In this
+paper, we propose HMD-Poser, the first unified approach to recover full-body
+motions using scalable sparse observations from HMD and body-worn IMUs. In
+particular, it can support a variety of input scenarios, such as HMD,
+HMD+2IMUs, HMD+3IMUs, etc. The scalability of inputs may accommodate users'
+choices for both high tracking accuracy and easy-to-wear. A lightweight
+temporal-spatial feature learning network is proposed in HMD-Poser to guarantee
+that the model runs in real-time on HMDs. Furthermore, HMD-Poser presents
+online body shape estimation to improve the position accuracy of body joints.
+Extensive experimental results on the challenging AMASS dataset show that
+HMD-Poser achieves new state-of-the-art results in both accuracy and real-time
+performance. We also build a new free-dancing motion dataset to evaluate
+HMD-Poser's on-device performance and investigate the performance gap between
+synthetic data and real-captured sensor data. Finally, we demonstrate our
+HMD-Poser with a real-time Avatar-driving application on a commercial HMD. Our
+code and free-dancing motion dataset are available
+https://pico-ai-team.github.io/hmd-poser",cs.CV,['cs.CV']
+BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model,song yiran · Qianyu Zhou · Xiangtai Li · Deng-Ping Fan · Xuequan Lu · Lizhuang Ma, ,https://arxiv.org/abs/2401.02317,,2401.02317.pdf,BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model,"In this paper, we address the challenge of image resolution variation for the
+Segment Anything Model (SAM). SAM, known for its zero-shot generalizability,
+exhibits a performance degradation when faced with datasets with varying image
+sizes. Previous approaches tend to resize the image to a fixed size or adopt
+structure modifications, hindering the preservation of SAM's rich prior
+knowledge. Besides, such task-specific tuning necessitates a complete
+retraining of the model, which is cost-expensive and unacceptable for
+deployment in the downstream tasks. In this paper, we reformulate this issue as
+a length extrapolation problem, where token sequence length varies while
+maintaining a consistent patch size for images of different sizes. To this end,
+we propose Scalable Bias-Mode Attention Mask (BA-SAM) to enhance SAM's
+adaptability to varying image resolutions while eliminating the need for
+structure modifications. Firstly, we introduce a new scaling factor to ensure
+consistent magnitude in the attention layer's dot product values when the token
+sequence length changes. Secondly, we present a bias-mode attention mask that
+allows each token to prioritize neighboring information, mitigating the impact
+of untrained distant information. Our BA-SAM demonstrates efficacy in two
+scenarios: zero-shot and fine-tuning. Extensive evaluation on diverse datasets,
+including DIS5K, DUTS, ISIC, COD10K, and COCO, reveals its ability to
+significantly mitigate performance degradation in the zero-shot setting and
+achieve state-of-the-art performance with minimal fine-tuning. Furthermore, we
+propose a generalized model and benchmark, showcasing BA-SAM's generalizability
+across all four datasets simultaneously. Code is available at
+https://github.com/zongzi13545329/BA-SAM",cs.CV,['cs.CV']
+CosalPure: Learning Concept from Group Images for Robust Co-Saliency Detection,Jiayi Zhu · Qing Guo · Felix Juefei Xu · Yihao Huang · Yang Liu · Geguang Pu, ,https://arxiv.org/abs/2403.18554,,2403.18554.pdf,CosalPure: Learning Concept from Group Images for Robust Co-Saliency Detection,"Co-salient object detection (CoSOD) aims to identify the common and salient
+(usually in the foreground) regions across a given group of images. Although
+achieving significant progress, state-of-the-art CoSODs could be easily
+affected by some adversarial perturbations, leading to substantial accuracy
+reduction. The adversarial perturbations can mislead CoSODs but do not change
+the high-level semantic information (e.g., concept) of the co-salient objects.
+In this paper, we propose a novel robustness enhancement framework by first
+learning the concept of the co-salient objects based on the input group images
+and then leveraging this concept to purify adversarial perturbations, which are
+subsequently fed to CoSODs for robustness enhancement. Specifically, we propose
+CosalPure containing two modules, i.e., group-image concept learning and
+concept-guided diffusion purification. For the first module, we adopt a
+pre-trained text-to-image diffusion model to learn the concept of co-salient
+objects within group images where the learned concept is robust to adversarial
+examples. For the second module, we map the adversarial image to the latent
+space and then perform diffusion generation by embedding the learned concept
+into the noise prediction function as an extra condition. Our method can
+effectively alleviate the influence of the SOTA adversarial attack containing
+different adversarial patterns, including exposure and noise. The extensive
+results demonstrate that our method could enhance the robustness of CoSODs
+significantly.",cs.CV,['cs.CV']
+Adaptive Bidirectional Displacement for Semi-Supervised Medical Image Segmentation,Hanyang Chi · Jian Pang · Bingfeng Zhang · Weifeng Liu, ,https://arxiv.org/abs/2405.00378,,2405.00378.pdf,Adaptive Bidirectional Displacement for Semi-Supervised Medical Image Segmentation,"Consistency learning is a central strategy to tackle unlabeled data in
+semi-supervised medical image segmentation (SSMIS), which enforces the model to
+produce consistent predictions under the perturbation. However, most current
+approaches solely focus on utilizing a specific single perturbation, which can
+only cope with limited cases, while employing multiple perturbations
+simultaneously is hard to guarantee the quality of consistency learning. In
+this paper, we propose an Adaptive Bidirectional Displacement (ABD) approach to
+solve the above challenge. Specifically, we first design a bidirectional patch
+displacement based on reliable prediction confidence for unlabeled data to
+generate new samples, which can effectively suppress uncontrollable regions and
+still retain the influence of input perturbations. Meanwhile, to enforce the
+model to learn the potentially uncontrollable content, a bidirectional
+displacement operation with inverse confidence is proposed for the labeled
+images, which generates samples with more unreliable information to facilitate
+model learning. Extensive experiments show that ABD achieves new
+state-of-the-art performances for SSMIS, significantly improving different
+baselines. Source code is available at https://github.com/chy-upc/ABD.",cs.CV,['cs.CV']
+UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather,Haimei Zhao · Jing Zhang · Zhuo Chen · Shanshan Zhao · Dacheng Tao, ,https://arxiv.org/abs/2404.05145,,2404.05145.pdf,UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather,"LiDAR semantic segmentation (LSS) is a critical task in autonomous driving
+and has achieved promising progress. However, prior LSS methods are
+conventionally investigated and evaluated on datasets within the same domain in
+clear weather. The robustness of LSS models in unseen scenes and all weather
+conditions is crucial for ensuring safety and reliability in real applications.
+To this end, we propose UniMix, a universal method that enhances the
+adaptability and generalizability of LSS models. UniMix first leverages
+physically valid adverse weather simulation to construct a Bridge Domain, which
+serves to bridge the domain gap between the clear weather scenes and the
+adverse weather scenes. Then, a Universal Mixing operator is defined regarding
+spatial, intensity, and semantic distributions to create the intermediate
+domain with mixed samples from given domains. Integrating the proposed two
+techniques into a teacher-student framework, UniMix efficiently mitigates the
+domain gap and enables LSS models to learn weather-robust and domain-invariant
+representations. We devote UniMix to two main setups: 1) unsupervised domain
+adaption, adapting the model from the clear weather source domain to the
+adverse weather target domain; 2) domain generalization, learning a model that
+generalizes well to unseen scenes in adverse weather. Extensive experiments
+validate the effectiveness of UniMix across different tasks and datasets, all
+achieving superior performance over state-of-the-art methods. The code will be
+released.",cs.CV,['cs.CV']
+Estimating Extreme 3D Image Rotations using Cascaded Attention,Shay Dekel · Yosi Keller · Martin Čadík, ,,https://www.youtube.com/watch?v=LzUPefef_8Q,,,,,nan
+PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics,Tianyi Xie · Zeshun Zong · Yuxing Qiu · Xuan Li · Yutao Feng · Yin Yang · Chenfanfu Jiang, ,https://arxiv.org/abs/2311.12198,,2311.12198.pdf,PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics,"We introduce PhysGaussian, a new method that seamlessly integrates physically
+grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel
+motion synthesis. Employing a custom Material Point Method (MPM), our approach
+enriches 3D Gaussian kernels with physically meaningful kinematic deformation
+and mechanical stress attributes, all evolved in line with continuum mechanics
+principles. A defining characteristic of our method is the seamless integration
+between physical simulation and visual rendering: both components utilize the
+same 3D Gaussian kernels as their discrete representations. This negates the
+necessity for triangle/tetrahedron meshing, marching cubes, ""cage meshes,"" or
+any other geometry embedding, highlighting the principle of ""what you see is
+what you simulate (WS$^2$)."" Our method demonstrates exceptional versatility
+across a wide variety of materials--including elastic entities, metals,
+non-Newtonian fluids, and granular materials--showcasing its strong
+capabilities in creating diverse visual content with novel viewpoints and
+movements. Our project page is at: https://xpandora.github.io/PhysGaussian/",cs.GR,"['cs.GR', 'cs.AI', 'cs.CV', 'cs.LG']"
+RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding,Jihan Yang · Runyu Ding · Weipeng DENG · Zhe Wang · Xiaojuan Qi, ,https://arxiv.org/abs/2308.00353,,2308.00353.pdf,Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding,"Open-world instance-level scene understanding aims to locate and recognize
+unseen object categories that are not present in the annotated dataset. This
+task is challenging because the model needs to both localize novel 3D objects
+and infer their semantic categories. A key factor for the recent progress in 2D
+open-world perception is the availability of large-scale image-text pairs from
+the Internet, which cover a wide range of vocabulary concepts. However, this
+success is hard to replicate in 3D scenarios due to the scarcity of 3D-text
+pairs. To address this challenge, we propose to harness pre-trained
+vision-language (VL) foundation models that encode extensive knowledge from
+image-text pairs to generate captions for multi-view images of 3D scenes. This
+allows us to establish explicit associations between 3D shapes and
+semantic-rich captions. Moreover, to enhance the fine-grained visual-semantic
+representation learning from captions for object-level categorization, we
+design hierarchical point-caption association methods to learn semantic-aware
+embeddings that exploit the 3D geometry between 3D points and multi-view
+images. In addition, to tackle the localization challenge for novel classes in
+the open-world setting, we develop debiased instance localization, which
+involves training object grouping modules on unlabeled data using
+instance-level pseudo supervision. This significantly improves the
+generalization capabilities of instance grouping and thus the ability to
+accurately locate novel objects. We conduct extensive experiments on 3D
+semantic, instance, and panoptic segmentation tasks, covering indoor and
+outdoor scenes across three datasets. Our method outperforms baseline methods
+by a significant margin in semantic segmentation (e.g. 34.5%$\sim$65.3%),
+instance segmentation (e.g. 21.8%$\sim$54.0%) and panoptic segmentation (e.g.
+14.7%$\sim$43.3%). Code will be available.",cs.CV,['cs.CV']
+Modality-Collaborative Test-Time Adaptation for Action Recognition,Baochen Xiong · Xiaoshan Yang · Yaguang Song · Yaowei Wang · Changsheng Xu, ,,https://dl.acm.org/doi/pdf/10.1145/3581783.3611757,,,,,nan
+3D Human Pose Perception from Egocentric Stereo Videos,Hiroyasu Akada · Jian Wang · Vladislav Golyanik · Christian Theobalt, ,https://arxiv.org/abs/2401.00889,,2401.00889.pdf,3D Human Pose Perception from Egocentric Stereo Videos,"While head-mounted devices are becoming more compact, they provide egocentric
+views with significant self-occlusions of the device user. Hence, existing
+methods often fail to accurately estimate complex 3D poses from egocentric
+views. In this work, we propose a new transformer-based framework to improve
+egocentric stereo 3D human pose estimation, which leverages the scene
+information and temporal context of egocentric stereo videos. Specifically, we
+utilize 1) depth features from our 3D scene reconstruction module with
+uniformly sampled windows of egocentric stereo frames, and 2) human joint
+queries enhanced by temporal features of the video inputs. Our method is able
+to accurately estimate human poses even in challenging scenarios, such as
+crouching and sitting. Furthermore, we introduce two new benchmark datasets,
+i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a
+much larger number of egocentric stereo views with a wider variety of human
+motions than the existing datasets, allowing comprehensive evaluation of
+existing and upcoming methods. Our extensive experiments show that the proposed
+approach significantly outperforms previous methods. We will release
+UnrealEgo2, UnrealEgo-RW, and trained models on our project page.",cs.CV,['cs.CV']
+Deep Generative Model based Rate-Distortion for Image Downscaling Assessment,yuanbang liang · Bhavesh Garg · Paul L. Rosin · Yipeng Qin, ,https://arxiv.org/abs/2403.15139,,2403.15139.pdf,Deep Generative Model based Rate-Distortion for Image Downscaling Assessment,"In this paper, we propose Image Downscaling Assessment by Rate-Distortion
+(IDA-RD), a novel measure to quantitatively evaluate image downscaling
+algorithms. In contrast to image-based methods that measure the quality of
+downscaled images, ours is process-based that draws ideas from rate-distortion
+theory to measure the distortion incurred during downscaling. Our main idea is
+that downscaling and super-resolution (SR) can be viewed as the encoding and
+decoding processes in the rate-distortion model, respectively, and that a
+downscaling algorithm that preserves more details in the resulting
+low-resolution (LR) images should lead to less distorted high-resolution (HR)
+images in SR. In other words, the distortion should increase as the downscaling
+algorithm deteriorates. However, it is non-trivial to measure this distortion
+as it requires the SR algorithm to be blind and stochastic. Our key insight is
+that such requirements can be met by recent SR algorithms based on deep
+generative models that can find all matching HR images for a given LR image on
+their learned image manifolds. Extensive experimental results show the
+effectiveness of our IDA-RD measure.",cs.CV,"['cs.CV', 'eess.IV']"
+Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?,Zhiqi Li · Zhiding Yu · Shiyi Lan · Jiahan Li · Jan Kautz · Tong Lu · Jose M. Alvarez, ,https://arxiv.org/abs/2312.03031,,2312.03031.pdf,Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?,"End-to-end autonomous driving recently emerged as a promising research
+direction to target autonomy from a full-stack perspective. Along this line,
+many of the latest works follow an open-loop evaluation setting on nuScenes to
+study the planning behavior. In this paper, we delve deeper into the problem by
+conducting thorough analyses and demystifying more devils in the details. We
+initially observed that the nuScenes dataset, characterized by relatively
+simple driving scenarios, leads to an under-utilization of perception
+information in end-to-end models incorporating ego status, such as the ego
+vehicle's velocity. These models tend to rely predominantly on the ego
+vehicle's status for future path planning. Beyond the limitations of the
+dataset, we also note that current metrics do not comprehensively assess the
+planning quality, leading to potentially biased conclusions drawn from existing
+benchmarks. To address this issue, we introduce a new metric to evaluate
+whether the predicted trajectories adhere to the road. We further propose a
+simple baseline able to achieve competitive results without relying on
+perception annotations. Given the current limitations on the benchmark and
+metrics, we suggest the community reassess relevant prevailing research and be
+cautious whether the continued pursuit of state-of-the-art would yield
+convincing and universal conclusions. Code and models are available at
+\url{https://github.com/NVlabs/BEV-Planner}",cs.CV,['cs.CV']
+FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding,Jun Xiang · Xuan Gao · Yudong Guo · Juyong Zhang, ,https://arxiv.org/abs/2312.02214,,2312.02214.pdf,FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding,"We propose FlashAvatar, a novel and lightweight 3D animatable avatar
+representation that could reconstruct a digital avatar from a short monocular
+video sequence in minutes and render high-fidelity photo-realistic images at
+300FPS on a consumer-grade GPU. To achieve this, we maintain a uniform 3D
+Gaussian field embedded in the surface of a parametric face model and learn
+extra spatial offset to model non-surface regions and subtle facial details.
+While full use of geometric priors can capture high-frequency facial details
+and preserve exaggerated expressions, proper initialization can help reduce the
+number of Gaussians, thus enabling super-fast rendering speed. Extensive
+experimental results demonstrate that FlashAvatar outperforms existing works
+regarding visual quality and personalized details and is almost an order of
+magnitude faster in rendering speed. Project page:
+https://ustc3dv.github.io/FlashAvatar/",cs.CV,"['cs.CV', 'cs.GR']"
+The Manga Whisperer: Automatically Generating Transcriptions for Comics,Ragav Sachdeva · Andrew Zisserman,https://github.com/ragavsachdeva/magi,https://arxiv.org/abs/2401.10224,,2401.10224.pdf,The Manga Whisperer: Automatically Generating Transcriptions for Comics,"In the past few decades, Japanese comics, commonly referred to as Manga, have
+transcended both cultural and linguistic boundaries to become a true worldwide
+sensation. Yet, the inherent reliance on visual cues and illustration within
+manga renders it largely inaccessible to individuals with visual impairments.
+In this work, we seek to address this substantial barrier, with the aim of
+ensuring that manga can be appreciated and actively engaged by everyone.
+Specifically, we tackle the problem of diarisation i.e. generating a
+transcription of who said what and when, in a fully automatic way.
+  To this end, we make the following contributions: (1) we present a unified
+model, Magi, that is able to (a) detect panels, text boxes and character boxes,
+(b) cluster characters by identity (without knowing the number of clusters
+apriori), and (c) associate dialogues to their speakers; (2) we propose a novel
+approach that is able to sort the detected text boxes in their reading order
+and generate a dialogue transcript; (3) we annotate an evaluation benchmark for
+this task using publicly available [English] manga pages. The code, evaluation
+datasets and the pre-trained model can be found at:
+https://github.com/ragavsachdeva/magi.",cs.CV,['cs.CV']
+SNIDA: Unlocking Few-Shot Object Detection with Non-linear Semantic Decoupling Augmentation,Yanjie Wang · Xu Zou · Luxin Yan · Sheng Zhong · Jiahuan Zhou, ,https://arxiv.org/abs/2401.11140,,2401.11140.pdf,Stability Plasticity Decoupled Fine-tuning For Few-shot end-to-end Object Detection,"Few-shot object detection(FSOD) aims to design methods to adapt object
+detectors efficiently with only few annotated samples. Fine-tuning has been
+shown to be an effective and practical approach. However, previous works often
+take the classical base-novel two stage fine-tuning procedure but ignore the
+implicit stability-plasticity contradiction among different modules.
+Specifically, the random re-initialized classifiers need more plasticity to
+adapt to novel samples. The other modules inheriting pre-trained weights demand
+more stability to reserve their class-agnostic knowledge. Regular fine-tuning
+which couples the optimization of these two parts hurts the model
+generalization in FSOD scenarios. In this paper, we find that this problem is
+prominent in the end-to-end object detector Sparse R-CNN for its
+multi-classifier cascaded architecture. We propose to mitigate this
+contradiction by a new three-stage fine-tuning procedure by introducing an
+addtional plasticity classifier fine-tuning(PCF) stage. We further design the
+multi-source ensemble(ME) technique to enhance the generalization of the model
+in the final fine-tuning stage. Extensive experiments verify that our method is
+effective in regularizing Sparse R-CNN, outperforming previous methods in the
+FSOD benchmark.",cs.CV,"['cs.CV', 'cs.AI']"
+Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement,Xiuquan Hou · Meiqin Liu · Senlin Zhang · Ping Wei · Badong Chen,https://github.com/xiuqhou/Salience-DETR,https://arxiv.org/abs/2403.16131,,2403.16131.pdf,Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement,"DETR-like methods have significantly increased detection performance in an
+end-to-end manner. The mainstream two-stage frameworks of them perform dense
+self-attention and select a fraction of queries for sparse cross-attention,
+which is proven effective for improving performance but also introduces a heavy
+computational burden and high dependence on stable query selection. This paper
+demonstrates that suboptimal two-stage selection strategies result in scale
+bias and redundancy due to the mismatch between selected queries and objects in
+two-stage initialization. To address these issues, we propose hierarchical
+salience filtering refinement, which performs transformer encoding only on
+filtered discriminative queries, for a better trade-off between computational
+efficiency and precision. The filtering process overcomes scale bias through a
+novel scale-independent salience supervision. To compensate for the semantic
+misalignment among queries, we introduce elaborate query refinement modules for
+stable two-stage initialization. Based on above improvements, the proposed
+Salience DETR achieves significant improvements of +4.0% AP, +0.2% AP, +4.4% AP
+on three challenging task-specific detection datasets, as well as 49.2% AP on
+COCO 2017 with less FLOPs. The code is available at
+https://github.com/xiuqhou/Salience-DETR.",cs.CV,['cs.CV']
+One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion,Minghua Liu · Ruoxi Shi · Linghao Chen · Zhuoyang Zhang · Chao Xu · Xinyue Wei · Hansheng Chen · Chong Zeng · Jiayuan Gu · Hao Su,https://sudo-ai-3d.github.io/One2345plus_page/,,https://github.com/SUDO-AI-3D/One2345plus,,,,,nan
+Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Driving,Junhao Zheng · Chenhao Lin · Jiahao Sun · Zhengyu Zhao · Qian Li · Chao Shen,https://github.com/gandolfczjh/3d2fool,https://arxiv.org/abs/2403.17301,,2403.17301.pdf,Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Driving,"Deep learning-based monocular depth estimation (MDE), extensively applied in
+autonomous driving, is known to be vulnerable to adversarial attacks. Previous
+physical attacks against MDE models rely on 2D adversarial patches, so they
+only affect a small, localized region in the MDE map but fail under various
+viewpoints. To address these limitations, we propose 3D Depth Fool
+(3D$^2$Fool), the first 3D texture-based adversarial attack against MDE models.
+3D$^2$Fool is specifically optimized to generate 3D adversarial textures
+agnostic to model types of vehicles and to have improved robustness in bad
+weather conditions, such as rain and fog. Experimental results validate the
+superior performance of our 3D$^2$Fool across various scenarios, including
+vehicles, MDE models, weather conditions, and viewpoints. Real-world
+experiments with printed 3D textures on physical vehicle models further
+demonstrate that our 3D$^2$Fool can cause an MDE error of over 10 meters.",cs.CV,"['cs.CV', 'cs.CR']"
+VecFusion: Vector Font Generation with Diffusion,Vikas Thamizharasan · Difan Liu · Shantanu Agarwal · Matthew Fisher · Michaël Gharbi · Oliver Wang · Alec Jacobson · Evangelos Kalogerakis, ,https://arxiv.org/abs/2312.10540,,2312.10540.pdf,VecFusion: Vector Font Generation with Diffusion,"We present VecFusion, a new neural architecture that can generate vector
+fonts with varying topological structures and precise control point positions.
+Our approach is a cascaded diffusion model which consists of a raster diffusion
+model followed by a vector diffusion model. The raster model generates
+low-resolution, rasterized fonts with auxiliary control point information,
+capturing the global style and shape of the font, while the vector model
+synthesizes vector fonts conditioned on the low-resolution raster fonts from
+the first stage. To synthesize long and complex curves, our vector diffusion
+model uses a transformer architecture and a novel vector representation that
+enables the modeling of diverse vector geometry and the precise prediction of
+control points. Our experiments show that, in contrast to previous generative
+models for vector graphics, our new cascaded vector diffusion model generates
+higher quality vector fonts, with complex structures and diverse styles.",cs.CV,"['cs.CV', 'cs.GR']"
+LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection,Dat NGUYEN · Nesryne Mejri · Inder Pal Singh · Polina Kuleshova · Marcella Astrid · Anis Kacem · Enjie Ghorbel · Djamila Aouada,https://github.com/10Ring/LAA-Net,https://arxiv.org/abs/2401.13856,,2401.13856.pdf,LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection,"This paper introduces a novel approach for high-quality deepfake detection
+called Localized Artifact Attention Network (LAA-Net). Existing methods for
+high-quality deepfake detection are mainly based on a supervised binary
+classifier coupled with an implicit attention mechanism. As a result, they do
+not generalize well to unseen manipulations. To handle this issue, two main
+contributions are made. First, an explicit attention mechanism within a
+multi-task learning framework is proposed. By combining heatmap-based and
+self-consistency attention strategies, LAA-Net is forced to focus on a few
+small artifact-prone vulnerable regions. Second, an Enhanced Feature Pyramid
+Network (E-FPN) is proposed as a simple and effective mechanism for spreading
+discriminative low-level features into the final feature output, with the
+advantage of limiting redundancy. Experiments performed on several benchmarks
+show the superiority of our approach in terms of Area Under the Curve (AUC) and
+Average Precision (AP). The code is available at
+https://github.com/10Ring/LAA-Net.",cs.CV,['cs.CV']
+SAI3D: Segment Any Instance in 3D Scenes,Yingda Yin · Yuzheng Liu · Yang Xiao · Daniel Cohen-Or · Jingwei Huang · Baoquan Chen,https://yd-yin.github.io/SAI3D/,https://arxiv.org/abs/2312.11557,,,SAI3D: Segment Any Instance in 3D Scenes,"Advancements in 3D instance segmentation have traditionally been tethered to
+the availability of annotated datasets, limiting their application to a narrow
+spectrum of object categories. Recent efforts have sought to harness
+vision-language models like CLIP for open-set semantic reasoning, yet these
+methods struggle to distinguish between objects of the same categories and rely
+on specific prompts that are not universally applicable. In this paper, we
+introduce SAI3D, a novel zero-shot 3D instance segmentation approach that
+synergistically leverages geometric priors and semantic cues derived from
+Segment Anything Model (SAM). Our method partitions a 3D scene into geometric
+primitives, which are then progressively merged into 3D instance segmentations
+that are consistent with the multi-view SAM masks. Moreover, we design a
+hierarchical region-growing algorithm with a dynamic thresholding mechanism,
+which largely improves the robustness of finegrained 3D scene parsing.Empirical
+evaluations on ScanNet, Matterport3D and the more challenging ScanNet++
+datasets demonstrate the superiority of our approach. Notably, SAI3D
+outperforms existing open-vocabulary baselines and even surpasses
+fully-supervised methods in class-agnostic segmentation on ScanNet++. Our
+project page is at https://yd-yin.github.io/SAI3D.",cs.CV,['cs.CV']
+InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models,Jiun Tian Hoe · Xudong Jiang · Chee Seng Chan · Yap-peng Tan · Weipeng Hu,https://jiuntian.github.io/interactdiffusion/,https://arxiv.org/abs/2312.05849,,2312.05849.pdf,InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models,"Large-scale text-to-image (T2I) diffusion models have showcased incredible
+capabilities in generating coherent images based on textual descriptions,
+enabling vast applications in content generation. While recent advancements
+have introduced control over factors such as object localization, posture, and
+image contours, a crucial gap remains in our ability to control the
+interactions between objects in the generated content. Well-controlling
+interactions in generated images could yield meaningful applications, such as
+creating realistic scenes with interacting characters. In this work, we study
+the problems of conditioning T2I diffusion models with Human-Object Interaction
+(HOI) information, consisting of a triplet label (person, action, object) and
+corresponding bounding boxes. We propose a pluggable interaction control model,
+called InteractDiffusion that extends existing pre-trained T2I diffusion models
+to enable them being better conditioned on interactions. Specifically, we
+tokenize the HOI information and learn their relationships via interaction
+embeddings. A conditioning self-attention layer is trained to map HOI tokens to
+visual tokens, thereby conditioning the visual tokens better in existing T2I
+diffusion models. Our model attains the ability to control the interaction and
+location on existing T2I diffusion models, which outperforms existing baselines
+by a large margin in HOI detection score, as well as fidelity in FID and KID.
+Project page: https://jiuntian.github.io/interactdiffusion.",cs.CV,"['cs.CV', 'cs.GR', 'cs.MM']"
+G3DR: Generative 3D Reconstruction in ImageNet,Pradyumna Reddy · Ismail Elezi · Jiankang Deng,https://preddy5.github.io/g3dr_website/,https://arxiv.org/abs/2403.00939,,2403.00939.pdf,G3DR: Generative 3D Reconstruction in ImageNet,"We introduce a novel 3D generative method, Generative 3D Reconstruction
+(G3DR) in ImageNet, capable of generating diverse and high-quality 3D objects
+from single images, addressing the limitations of existing methods. At the
+heart of our framework is a novel depth regularization technique that enables
+the generation of scenes with high-geometric fidelity. G3DR also leverages a
+pretrained language-vision model, such as CLIP, to enable reconstruction in
+novel views and improve the visual realism of generations. Additionally, G3DR
+designs a simple but effective sampling procedure to further improve the
+quality of generations. G3DR offers diverse and efficient 3D asset generation
+based on class or text conditioning. Despite its simplicity, G3DR is able to
+beat state-of-theart methods, improving over them by up to 22% in perceptual
+metrics and 90% in geometry scores, while needing only half of the training
+time. Code is available at https://github.com/preddy5/G3DR",cs.CV,"['cs.CV', 'cs.GR']"
+ZeroRF: Fast Sparse View 360° Reconstruction with Zero Pretraining,Ruoxi Shi · Xinyue Wei · Cheng Wang · Hao Su, ,https://arxiv.org/abs/2312.09249,,2312.09249.pdf,ZeroRF: Fast Sparse View 360° Reconstruction with Zero Pretraining,"We present ZeroRF, a novel per-scene optimization method addressing the
+challenge of sparse view 360{\deg} reconstruction in neural field
+representations. Current breakthroughs like Neural Radiance Fields (NeRF) have
+demonstrated high-fidelity image synthesis but struggle with sparse input
+views. Existing methods, such as Generalizable NeRFs and per-scene optimization
+approaches, face limitations in data dependency, computational cost, and
+generalization across diverse scenarios. To overcome these challenges, we
+propose ZeroRF, whose key idea is to integrate a tailored Deep Image Prior into
+a factorized NeRF representation. Unlike traditional methods, ZeroRF
+parametrizes feature grids with a neural network generator, enabling efficient
+sparse view 360{\deg} reconstruction without any pretraining or additional
+regularization. Extensive experiments showcase ZeroRF's versatility and
+superiority in terms of both quality and speed, achieving state-of-the-art
+results on benchmark datasets. ZeroRF's significance extends to applications in
+3D content generation and editing. Project page:
+https://sarahweiii.github.io/zerorf/",cs.CV,"['cs.CV', 'cs.GR']"
+HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data,Qifan Yu · Juncheng Li · Longhui Wei · Liang Pang · Wentao Ye · Bosheng Qin · Siliang Tang · Qi Tian · Yueting Zhuang, ,https://arxiv.org/abs/2311.13614,,2311.13614.pdf,HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data,"Multi-modal Large Language Models (MLLMs) tuned on machine-generated
+instruction-following data have demonstrated remarkable performance in various
+multi-modal understanding and generation tasks. However, the hallucinations
+inherent in machine-generated data, which could lead to hallucinatory outputs
+in MLLMs, remain under-explored. This work aims to investigate various
+hallucinations (i.e., object, relation, attribute hallucinations) and mitigate
+those hallucinatory toxicities in large-scale machine-generated visual
+instruction datasets. Drawing on the human ability to identify factual errors,
+we present a novel hallucination detection and elimination framework,
+HalluciDoctor, based on the cross-checking paradigm. We use our framework to
+identify and eliminate hallucinations in the training data automatically.
+Interestingly, HalluciDoctor also indicates that spurious correlations arising
+from long-tail object co-occurrences contribute to hallucinations. Based on
+that, we execute counterfactual visual instruction expansion to balance data
+distribution, thereby enhancing MLLMs' resistance to hallucinations.
+Comprehensive experiments on hallucination evaluation benchmarks show that our
+method successfully mitigates 44.6% hallucinations relatively and maintains
+competitive performance compared to LLaVA. The data and code for this paper are
+publicly available. \url{https://github.com/Yuqifan1117/HalluciDoctor}.",cs.CV,"['cs.CV', 'cs.AI']"
+Mudslide: A Universal Nuclear Instance Segmentation Method,Jun Wang, ,https://arxiv.org/abs/2311.15939,,2311.15939.pdf,Unleashing the Power of Prompt-driven Nucleus Instance Segmentation,"Nucleus instance segmentation in histology images is crucial for a broad
+spectrum of clinical applications. Current dominant algorithms rely on
+regression of nuclear proxy maps. Distinguishing nucleus instances from the
+estimated maps requires carefully curated post-processing, which is error-prone
+and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned
+huge attention in medical image segmentation, owing to its impressive
+generalization ability and promptable property. Nevertheless, its potential on
+nucleus instance segmentation remains largely underexplored. In this paper, we
+present a novel prompt-driven framework that consists of a nucleus prompter and
+SAM for automatic nucleus instance segmentation. Specifically, the prompter
+learns to generate a unique point prompt for each nucleus while the SAM is
+fine-tuned to output the corresponding mask for the prompted nucleus.
+Furthermore, we propose the inclusion of adjacent nuclei as negative prompts to
+enhance the model's capability to identify overlapping nuclei. Without
+complicated post-processing, our proposed method sets a new state-of-the-art
+performance on three challenging benchmarks. Code is available at
+\url{github.com/windygoo/PromptNucSeg}",cs.CV,['cs.CV']
+MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models,Yanting Wang · Hongye Fu · Wei Zou · Jinyuan Jia, ,https://arxiv.org/abs/2403.19080,,2403.19080.pdf,MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models,"Different from a unimodal model whose input is from a single modality, the
+input (called multi-modal input) of a multi-modal model is from multiple
+modalities such as image, 3D points, audio, text, etc. Similar to unimodal
+models, many existing studies show that a multi-modal model is also vulnerable
+to adversarial perturbation, where an attacker could add small perturbation to
+all modalities of a multi-modal input such that the multi-modal model makes
+incorrect predictions for it. Existing certified defenses are mostly designed
+for unimodal models, which achieve sub-optimal certified robustness guarantees
+when extended to multi-modal models as shown in our experimental results. In
+our work, we propose MMCert, the first certified defense against adversarial
+attacks to a multi-modal model. We derive a lower bound on the performance of
+our MMCert under arbitrary adversarial attacks with bounded perturbations to
+both modalities (e.g., in the context of auto-driving, we bound the number of
+changed pixels in both RGB image and depth image). We evaluate our MMCert using
+two benchmark datasets: one for the multi-modal road segmentation task and the
+other for the multi-modal emotion recognition task. Moreover, we compare our
+MMCert with a state-of-the-art certified defense extended from unimodal models.
+Our experimental results show that our MMCert outperforms the baseline.",cs.CV,"['cs.CV', 'cs.CR']"
+NTO3D: Neural Target Object 3D Reconstruction with Segment Anything,Xiaobao Wei · Renrui Zhang · Jiarui Wu · Jiaming Liu · Ming Lu · Yandong Guo · Shanghang Zhang, ,https://arxiv.org/abs/2309.12790,,2309.12790.pdf,NTO3D: Neural Target Object 3D Reconstruction with Segment Anything,"Neural 3D reconstruction from multi-view images has recently attracted
+increasing attention from the community. Existing methods normally learn a
+neural field for the whole scene, while it is still under-explored how to
+reconstruct a target object indicated by users. Considering the Segment
+Anything Model (SAM) has shown effectiveness in segmenting any 2D images, in
+this paper, we propose NTO3D, a novel high-quality Neural Target Object 3D
+(NTO3D) reconstruction method, which leverages the benefits of both neural
+field and SAM. We first propose a novel strategy to lift the multi-view 2D
+segmentation masks of SAM into a unified 3D occupancy field. The 3D occupancy
+field is then projected into 2D space and generates the new prompts for SAM.
+This process is iterative until convergence to separate the target object from
+the scene. After this, we then lift the 2D features of the SAM encoder into a
+3D feature field in order to improve the reconstruction quality of the target
+object. NTO3D lifts the 2D masks and features of SAM into the 3D neural field
+for high-quality neural target object 3D reconstruction. We conduct detailed
+experiments on several benchmark datasets to demonstrate the advantages of our
+method. The code will be available at: https://github.com/ucwxb/NTO3D.",cs.CV,['cs.CV']
+A Bayesian Approach to OOD Robustness in Image Classification,Prakhar Kaushik · Adam Kortylewski · Alan L. Yuille, ,https://arxiv.org/abs/2403.07277v1,,2403.07277v1.pdf,A Bayesian Approach to OOD Robustness in Image Classification,"An important and unsolved problem in computer vision is to ensure that the
+algorithms are robust to changes in image domains. We address this problem in
+the scenario where we have access to images from the target domains but no
+annotations. Motivated by the challenges of the OOD-CV benchmark where we
+encounter real world Out-of-Domain (OOD) nuisances and occlusion, we introduce
+a novel Bayesian approach to OOD robustness for object classification. Our work
+extends Compositional Neural Networks (CompNets), which have been shown to be
+robust to occlusion but degrade badly when tested on OOD data. We exploit the
+fact that CompNets contain a generative head defined over feature vectors
+represented by von Mises-Fisher (vMF) kernels, which correspond roughly to
+object parts, and can be learned without supervision. We obverse that some vMF
+kernels are similar between different domains, while others are not. This
+enables us to learn a transitional dictionary of vMF kernels that are
+intermediate between the source and target domains and train the generative
+model on this dictionary using the annotations on the source domain, followed
+by iterative refinement. This approach, termed Unsupervised Generative
+Transition (UGT), performs very well in OOD scenarios even when occlusion is
+present. UGT is evaluated on different OOD benchmarks including the OOD-CV
+dataset, several popular datasets (e.g., ImageNet-C [9]), artificial image
+corruptions (including adding occluders), and synthetic-to-real domain
+transfer, and does well in all scenarios outperforming SOTA alternatives (e.g.
+up to 10% top-1 accuracy on Occluded OOD-CV dataset).",cs.CV,"['cs.CV', 'cs.AI']"
+SNI-SLAM: Semantic Neural Implicit SLAM,Siting Zhu · Guangming Wang · Hermann Blum · Jiuming Liu · LiangSong · Marc Pollefeys · Hesheng Wang, ,https://arxiv.org/abs/2311.11016,,2311.11016.pdf,SNI-SLAM: Semantic Neural Implicit SLAM,"We propose SNI-SLAM, a semantic SLAM system utilizing neural implicit
+representation, that simultaneously performs accurate semantic mapping,
+high-quality surface reconstruction, and robust camera tracking. In this
+system, we introduce hierarchical semantic representation to allow multi-level
+semantic comprehension for top-down structured semantic mapping of the scene.
+In addition, to fully utilize the correlation between multiple attributes of
+the environment, we integrate appearance, geometry and semantic features
+through cross-attention for feature collaboration. This strategy enables a more
+multifaceted understanding of the environment, thereby allowing SNI-SLAM to
+remain robust even when single attribute is defective. Then, we design an
+internal fusion-based decoder to obtain semantic, RGB, Truncated Signed
+Distance Field (TSDF) values from multi-level features for accurate decoding.
+Furthermore, we propose a feature loss to update the scene representation at
+the feature level. Compared with low-level losses such as RGB loss and depth
+loss, our feature loss is capable of guiding the network optimization on a
+higher-level. Our SNI-SLAM method demonstrates superior performance over all
+recent NeRF-based SLAM methods in terms of mapping and tracking accuracy on
+Replica and ScanNet datasets, while also showing excellent capabilities in
+accurate semantic segmentation and real-time semantic mapping.",cs.RO,['cs.RO']
+PBWR: Parametric Building Wireframe Reconstruction from Aerial LiDAR Point Clouds,Shangfeng Huang · Ruisheng Wang · Bo Guo · Hongxin Yang, ,https://arxiv.org/abs/2311.12062,,2311.12062.pdf,PBWR: Parametric Building Wireframe Reconstruction from Aerial LiDAR Point Clouds,"In this paper, we present an end-to-end 3D building wireframe reconstruction
+method to regress edges directly from aerial LiDAR point clouds.Our method,
+named Parametric Building Wireframe Reconstruction (PBWR), takes aerial LiDAR
+point clouds and initial edge entities as input, and fully uses self-attention
+mechanism of transformers to regress edge parameters without any intermediate
+steps such as corner prediction. We propose an edge non-maximum suppression
+(E-NMS) module based on edge similarityto remove redundant edges. Additionally,
+a dedicated edge loss function is utilized to guide the PBWR in regressing
+edges parameters, where simple use of edge distance loss isn't suitable. In our
+experiments, we demonstrate state-of-the-art results on the Building3D dataset,
+achieving an improvement of approximately 36% in entry-level dataset edge
+accuracy and around 42% improvement in the Tallinn dataset.",cs.CV,"['cs.CV', 'cs.AI']"
+Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling,Zhe Li · Zerong Zheng · Lizhen Wang · Yebin Liu,https://animatable-gaussians.github.io/,https://arxiv.org/abs/2311.16096,,2311.16096.pdf,Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling,"Modeling animatable human avatars from RGB videos is a long-standing and
+challenging problem. Recent works usually adopt MLP-based neural radiance
+fields (NeRF) to represent 3D humans, but it remains difficult for pure MLPs to
+regress pose-dependent garment details. To this end, we introduce Animatable
+Gaussians, a new avatar representation that leverages powerful 2D CNNs and 3D
+Gaussian splatting to create high-fidelity avatars. To associate 3D Gaussians
+with the animatable avatar, we learn a parametric template from the input
+videos, and then parameterize the template on two front & back canonical
+Gaussian maps where each pixel represents a 3D Gaussian. The learned template
+is adaptive to the wearing garments for modeling looser clothes like dresses.
+Such template-guided 2D parameterization enables us to employ a powerful
+StyleGAN-based CNN to learn the pose-dependent Gaussian maps for modeling
+detailed dynamic appearances. Furthermore, we introduce a pose projection
+strategy for better generalization given novel poses. To tackle the realistic
+relighting of animatable avatars, we introduce physically-based rendering into
+the avatar representation for decomposing avatar materials and environment
+illumination. Overall, our method can create lifelike avatars with dynamic,
+realistic, generalized and relightable appearances. Experiments show that our
+method outperforms other state-of-the-art approaches.",cs.CV,"['cs.CV', 'cs.GR']"
+Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal,Yijun Yang · Hongtao Wu · Angelica I. Aviles-Rivero · Yulun Zhang · Jing Qin · Lei Zhu, ,https://arxiv.org/abs/2403.07684,,2403.07684.pdf,Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal,"Real-world vision tasks frequently suffer from the appearance of unexpected
+adverse weather conditions, including rain, haze, snow, and raindrops. In the
+last decade, convolutional neural networks and vision transformers have yielded
+outstanding results in single-weather video removal. However, due to the
+absence of appropriate adaptation, most of them fail to generalize to other
+weather conditions. Although ViWS-Net is proposed to remove adverse weather
+conditions in videos with a single set of pre-trained weights, it is seriously
+blinded by seen weather at train-time and degenerates when coming to unseen
+weather during test-time. In this work, we introduce test-time adaptation into
+adverse weather removal in videos, and propose the first framework that
+integrates test-time adaptation into the iterative diffusion reverse process.
+Specifically, we devise a diffusion-based network with a novel temporal noise
+model to efficiently explore frame-correlated information in degraded video
+clips at training stage. During inference stage, we introduce a proxy task
+named Diffusion Tubelet Self-Calibration to learn the primer distribution of
+test video stream and optimize the model by approximating the temporal noise
+model for online adaptation. Experimental results, on benchmark datasets,
+demonstrate that our Test-Time Adaptation method with Diffusion-based
+network(Diff-TTA) outperforms state-of-the-art methods in terms of restoring
+videos degraded by seen weather conditions. Its generalizable capability is
+also validated with unseen weather conditions in both synthesized and
+real-world videos.",cs.CV,['cs.CV']
+Generalizable Novel-View Synthesis using a Stereo Camera,Haechan Lee · Wonjoon Jin · Seung-Hwan Baek · Sunghyun Cho,https://jinwonjoon.github.io/stereonerf/,https://arxiv.org/abs/2404.13541,,2404.13541.pdf,Generalizable Novel-View Synthesis using a Stereo Camera,"In this paper, we propose the first generalizable view synthesis approach
+that specifically targets multi-view stereo-camera images. Since recent stereo
+matching has demonstrated accurate geometry prediction, we introduce stereo
+matching into novel-view synthesis for high-quality geometry reconstruction. To
+this end, this paper proposes a novel framework, dubbed StereoNeRF, which
+integrates stereo matching into a NeRF-based generalizable view synthesis
+approach. StereoNeRF is equipped with three key components to effectively
+exploit stereo matching in novel-view synthesis: a stereo feature extractor, a
+depth-guided plane-sweeping, and a stereo depth loss. Moreover, we propose the
+StereoNVS dataset, the first multi-view dataset of stereo-camera images,
+encompassing a wide variety of both real and synthetic scenes. Our experimental
+results demonstrate that StereoNeRF surpasses previous approaches in
+generalizable view synthesis.",cs.CV,['cs.CV']
+PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos,Yufei Zhang · Jeffrey Kephart · Zijun Cui · Qiang Ji, ,https://arxiv.org/abs/2404.04430,,2404.04430.pdf,PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos,"While current methods have shown promising progress on estimating 3D human
+motion from monocular videos, their motion estimates are often physically
+unrealistic because they mainly consider kinematics. In this paper, we
+introduce Physics-aware Pretrained Transformer (PhysPT), which improves
+kinematics-based motion estimates and infers motion forces. PhysPT exploits a
+Transformer encoder-decoder backbone to effectively learn human dynamics in a
+self-supervised manner. Moreover, it incorporates physics principles governing
+human motion. Specifically, we build a physics-based body representation and
+contact force model. We leverage them to impose novel physics-inspired training
+losses (i.e., force loss, contact loss, and Euler-Lagrange loss), enabling
+PhysPT to capture physical properties of the human body and the forces it
+experiences. Experiments demonstrate that, once trained, PhysPT can be directly
+applied to kinematics-based estimates to significantly enhance their physical
+plausibility and generate favourable motion forces. Furthermore, we show that
+these physically meaningful quantities translate into improved accuracy of an
+important downstream task: human action recognition.",cs.CV,['cs.CV']
+Self-Distilled Masked Auto-Encoders are Efficient  Video Anomaly Detectors,Nicolae Ristea · Florinel Croitoru · Radu Tudor Ionescu · Marius Popescu · Fahad Shahbaz Khan · Mubarak Shah,https://github.com/ristea/aed-mae/tree/main,https://arxiv.org/abs/2306.12041v2,,2306.12041v2.pdf,Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors,"We propose an efficient abnormal event detection model based on a lightweight
+masked auto-encoder (AE) applied at the video frame level. The novelty of the
+proposed model is threefold. First, we introduce an approach to weight tokens
+based on motion gradients, thus shifting the focus from the static background
+scene to the foreground objects. Second, we integrate a teacher decoder and a
+student decoder into our architecture, leveraging the discrepancy between the
+outputs given by the two decoders to improve anomaly detection. Third, we
+generate synthetic abnormal events to augment the training videos, and task the
+masked AE model to jointly reconstruct the original frames (without anomalies)
+and the corresponding pixel-level anomaly maps. Our design leads to an
+efficient and effective model, as demonstrated by the extensive experiments
+carried out on four benchmarks: Avenue, ShanghaiTech, UBnormal and UCSD Ped2.
+The empirical results show that our model achieves an excellent trade-off
+between speed and accuracy, obtaining competitive AUC scores, while processing
+1655 FPS. Hence, our model is between 8 and 70 times faster than competing
+methods. We also conduct an ablation study to justify our design. Our code is
+freely available at: https://github.com/ristea/aed-mae.",cs.CV,"['cs.CV', 'cs.LG']"
+Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization,Deng Li · Aming Wu · Yaowei Wang · Yahong Han, ,https://arxiv.org/abs/2402.18447,,2402.18447.pdf,Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization,"Single-domain generalization aims to learn a model from single source domain
+data to achieve generalized performance on other unseen target domains.
+Existing works primarily focus on improving the generalization ability of
+static networks. However, static networks are unable to dynamically adapt to
+the diverse variations in different image scenes, leading to limited
+generalization capability. Different scenes exhibit varying levels of
+complexity, and the complexity of images further varies significantly in
+cross-domain scenarios. In this paper, we propose a dynamic object-centric
+perception network based on prompt learning, aiming to adapt to the variations
+in image complexity. Specifically, we propose an object-centric gating module
+based on prompt learning to focus attention on the object-centric features
+guided by the various scene prompts. Then, with the object-centric gating
+masks, the dynamic selective module dynamically selects highly correlated
+feature regions in both spatial and channel dimensions enabling the model to
+adaptively perceive object-centric relevant features, thereby enhancing the
+generalization capability. Extensive experiments were conducted on
+single-domain generalization tasks in image classification and object
+detection. The experimental results demonstrate that our approach outperforms
+state-of-the-art methods, which validates the effectiveness and generally of
+our proposed method.",cs.CV,['cs.CV']
+PairAug: What Can Augmented Image-Text Pairs Do for Radiology?,Yutong Xie · Qi Chen · Sinuo Wang · Minh-Son To · Iris Lee · Ee Win Khoo · Kerolos Hendy · Daniel Koh · Yong Xia · Qi Wu, ,https://arxiv.org/abs/2404.04960,,2404.04960.pdf,PairAug: What Can Augmented Image-Text Pairs Do for Radiology?,"Current vision-language pre-training (VLP) methodologies predominantly depend
+on paired image-text datasets, a resource that is challenging to acquire in
+radiology due to privacy considerations and labelling complexities. Data
+augmentation provides a practical solution to overcome the issue of data
+scarcity, however, most augmentation methods exhibit a limited focus,
+prioritising either image or text augmentation exclusively. Acknowledging this
+limitation, our objective is to devise a framework capable of concurrently
+augmenting medical image and text data. We design a Pairwise Augmentation
+(PairAug) approach that contains an Inter-patient Augmentation (InterAug)
+branch and an Intra-patient Augmentation (IntraAug) branch. Specifically, the
+InterAug branch of our approach generates radiology images using synthesised
+yet plausible reports derived from a Large Language Model (LLM). The generated
+pairs can be considered a collection of new patient cases since they are
+artificially created and may not exist in the original dataset. In contrast,
+the IntraAug branch uses newly generated reports to manipulate images. This
+process allows us to create new paired data for each individual with diverse
+medical conditions. Our extensive experiments on various downstream tasks
+covering medical image classification zero-shot and fine-tuning analysis
+demonstrate that our PairAug, concurrently expanding both image and text data,
+substantially outperforms image-/text-only expansion baselines and advanced
+medical VLP baselines. Our code is released at
+\url{https://github.com/YtongXie/PairAug}.",cs.CV,['cs.CV']
+CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning,Lianggangxu Chen · Xuejiao Wang · Jiale Lu · Shaohui Lin · Changbo Wang · Gaoqi He, ,https://arxiv.org/abs/2309.16650,,2309.16650.pdf,ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning,"For robots to perform a wide variety of tasks, they require a 3D
+representation of the world that is semantically rich, yet compact and
+efficient for task-driven perception and planning. Recent approaches have
+attempted to leverage features from large vision-language models to encode
+semantics in 3D representations. However, these approaches tend to produce maps
+with per-point feature vectors, which do not scale well in larger environments,
+nor do they contain semantic spatial relationships between entities in the
+environment, which are useful for downstream planning. In this work, we propose
+ConceptGraphs, an open-vocabulary graph-structured representation for 3D
+scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
+their output to 3D by multi-view association. The resulting representations
+generalize to novel semantic classes, without the need to collect large 3D
+datasets or finetune models. We demonstrate the utility of this representation
+through a number of downstream planning tasks that are specified through
+abstract (language) prompts and require complex reasoning over spatial and
+semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
+video: https://youtu.be/mRhNkQwRYnc )",cs.RO,"['cs.RO', 'cs.CV']"
+Initialization Matters for Adversarial Transfer Learning,Andong Hua · Jindong Gu · Zhiyu Xue · Nicholas Carlini · Eric Wong · Yao Qin, ,https://arxiv.org/abs/2312.05716,,2312.05716.pdf,Initialization Matters for Adversarial Transfer Learning,"With the prevalence of the Pretraining-Finetuning paradigm in transfer
+learning, the robustness of downstream tasks has become a critical concern. In
+this work, we delve into adversarial robustness in transfer learning and reveal
+the critical role of initialization, including both the pretrained model and
+the linear head. First, we discover the necessity of an adversarially robust
+pretrained model. Specifically, we reveal that with a standard pretrained
+model, Parameter-Efficient Finetuning (PEFT) methods either fail to be
+adversarially robust or continue to exhibit significantly degraded adversarial
+robustness on downstream tasks, even with adversarial training during
+finetuning. Leveraging a robust pretrained model, surprisingly, we observe that
+a simple linear probing can outperform full finetuning and other PEFT methods
+with random initialization on certain datasets. We further identify that linear
+probing excels in preserving robustness from the robust pretraining. Based on
+this, we propose Robust Linear Initialization (RoLI) for adversarial
+finetuning, which initializes the linear head with the weights obtained by
+adversarial linear probing to maximally inherit the robustness from
+pretraining. Across five different image classification datasets, we
+demonstrate the effectiveness of RoLI and achieve new state-of-the-art results.
+Our code is available at \url{https://github.com/DongXzz/RoLI}.",cs.CV,['cs.CV']
+PEGASUS: Personalized Generative 3D Avatars with Composable Attributes,Hyunsoo Cha · Byungjun Kim · Hanbyul Joo, ,https://arxiv.org/abs/2402.10636,,2402.10636.pdf,PEGASUS: Personalized Generative 3D Avatars with Composable Attributes,"We present PEGASUS, a method for constructing a personalized generative 3D
+face avatar from monocular video sources. Our generative 3D avatar enables
+disentangled controls to selectively alter the facial attributes (e.g., hair or
+nose) while preserving the identity. Our approach consists of two stages:
+synthetic database generation and constructing a personalized generative
+avatar. We generate a synthetic video collection of the target identity with
+varying facial attributes, where the videos are synthesized by borrowing the
+attributes from monocular videos of diverse identities. Then, we build a
+person-specific generative 3D avatar that can modify its attributes
+continuously while preserving its identity. Through extensive experiments, we
+demonstrate that our method of generating a synthetic database and creating a
+3D generative avatar is the most effective in preserving identity while
+achieving high realism. Subsequently, we introduce a zero-shot approach to
+achieve the same goal of generative modeling more efficiently by leveraging a
+previously constructed personalized generative model.",cs.CV,['cs.CV']
+FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning,Yuxiang Lu · Suizhi Huang · Yuwen Yang · Shalayiding Sirejiding · Yue Ding · Hongtao Lu,https://github.com/innovator-zero/FedHCA2,https://arxiv.org/abs/2311.13250v2,,2311.13250v2.pdf,FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning,"Federated Learning (FL) enables joint training across distributed clients
+using their local data privately. Federated Multi-Task Learning (FMTL) builds
+on FL to handle multiple tasks, assuming model congruity that identical model
+architecture is deployed in each client. To relax this assumption and thus
+extend real-world applicability, we introduce a novel problem setting,
+Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse
+task setups. The main challenge of HC-FMTL is the model incongruity issue that
+invalidates conventional aggregation methods. It also escalates the
+difficulties in accurate model aggregation to deal with data and task
+heterogeneity inherent in FMTL. To address these challenges, we propose the
+FedHCA$^2$ framework, which allows for federated training of personalized
+models by modeling relationships among heterogeneous clients. Drawing on our
+theoretical insights into the difference between multi-task and federated
+optimization, we propose the Hyper Conflict-Averse Aggregation scheme to
+mitigate conflicts during encoder updates. Additionally, inspired by task
+interaction in MTL, the Hyper Cross Attention Aggregation scheme uses
+layer-wise cross attention to enhance decoder interactions while alleviating
+model incongruity. Moreover, we employ learnable Hyper Aggregation Weights for
+each client to customize personalized parameter updates. Extensive experiments
+demonstrate the superior performance of FedHCA$^2$ in various HC-FMTL scenarios
+compared to representative methods. Our code will be made publicly available.",cs.CV,"['cs.CV', 'cs.LG']"
+Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation,Jonas Herzog, ,https://arxiv.org/abs/2402.17614,,2402.17614.pdf,Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation,"Few-shot segmentation performance declines substantially when facing images
+from a domain different than the training domain, effectively limiting
+real-world use cases. To alleviate this, recently cross-domain few-shot
+segmentation (CD-FSS) has emerged. Works that address this task mainly
+attempted to learn segmentation on a source domain in a manner that generalizes
+across domains. Surprisingly, we can outperform these approaches while
+eliminating the training stage and removing their main segmentation network. We
+show test-time task-adaption is the key for successful CD-FSS instead.
+Task-adaption is achieved by appending small networks to the feature pyramid of
+a conventionally classification-pretrained backbone. To avoid overfitting to
+the few labeled samples in supervised fine-tuning, consistency across augmented
+views of input images serves as guidance while learning the parameters of the
+attached layers. Despite our self-restriction not to use any images other than
+the few labeled samples at test time, we achieve new state-of-the-art
+performance in CD-FSS, evidencing the need to rethink approaches for the task.",cs.CV,['cs.CV']
+TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models,Haomiao Ni · Bernhard Egger · Suhas Lohit · Anoop Cherian · Ye Wang · Toshiaki Koike-Akino · Sharon X. Huang · Tim Marks, ,https://arxiv.org/abs/2404.16306,,2404.16306.pdf,TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models,"Text-conditioned image-to-video generation (TI2V) aims to synthesize a
+realistic video starting from a given image (e.g., a woman's photo) and a text
+description (e.g., ""a woman is drinking water.""). Existing TI2V frameworks
+often require costly training on video-text datasets and specific model designs
+for text and image conditioning. In this paper, we propose TI2V-Zero, a
+zero-shot, tuning-free method that empowers a pretrained text-to-video (T2V)
+diffusion model to be conditioned on a provided image, enabling TI2V generation
+without any optimization, fine-tuning, or introducing external modules. Our
+approach leverages a pretrained T2V diffusion foundation model as the
+generative prior. To guide video generation with the additional image input, we
+propose a ""repeat-and-slide"" strategy that modulates the reverse denoising
+process, allowing the frozen diffusion model to synthesize a video
+frame-by-frame starting from the provided image. To ensure temporal continuity,
+we employ a DDPM inversion strategy to initialize Gaussian noise for each newly
+synthesized frame and a resampling technique to help preserve visual details.
+We conduct comprehensive experiments on both domain-specific and open-domain
+datasets, where TI2V-Zero consistently outperforms a recent open-domain TI2V
+model. Furthermore, we show that TI2V-Zero can seamlessly extend to other tasks
+such as video infilling and prediction when provided with more images. Its
+autoregressive design also supports long video generation.",cs.CV,['cs.CV']
+Atom-Level Optical Chemical Structure Recognition with Limited Supervision,Martijn Oldenhof · Edward De Brouwer · Adam Arany · Yves Moreau,https://github.com/molden/atomlenz,https://arxiv.org/abs/2404.01743,,2404.01743.pdf,Atom-Level Optical Chemical Structure Recognition with Limited Supervision,"Identifying the chemical structure from a graphical representation, or image,
+of a molecule is a challenging pattern recognition task that would greatly
+benefit drug development. Yet, existing methods for chemical structure
+recognition do not typically generalize well, and show diminished effectiveness
+when confronted with domains where data is sparse, or costly to generate, such
+as hand-drawn molecule images. To address this limitation, we propose a new
+chemical structure recognition tool that delivers state-of-the-art performance
+and can adapt to new domains with a limited number of data samples and
+supervision. Unlike previous approaches, our method provides atom-level
+localization, and can therefore segment the image into the different atoms and
+bonds. Our model is the first model to perform OCSR with atom-level entity
+detection with only SMILES supervision. Through rigorous and extensive
+benchmarking, we demonstrate the preeminence of our chemical structure
+recognition approach in terms of data efficiency, accuracy, and atom-level
+entity prediction.",cs.CV,['cs.CV']
+SubT-MRS Datasets: Pushing SLAM Towards All-weather Environments,Shibo Zhao · Yuanjun Gao · Tianhao Wu · Damanpreet Singh · Rushan Jiang · Haoxiang Sun · Mansi Sarawata · Warren Whittaker · Ian Higgins · Shaoshu Su · Yi Du · Can Xu · John Keller · Jay Karhade · Lucas Nogueira · Sourojit Saha · Yuheng Qiu · Ji Zhang · Wenshan Wang · Chen Wang · Sebastian Scherer,https://superodometry.com/datasets,https://arxiv.org/abs/2307.07607,,2307.07607.pdf,SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments,"Simultaneous localization and mapping (SLAM) is a fundamental task for
+numerous applications such as autonomous navigation and exploration. Despite
+many SLAM datasets have been released, current SLAM solutions still struggle to
+have sustained and resilient performance. One major issue is the absence of
+high-quality datasets including diverse all-weather conditions and a reliable
+metric for assessing robustness. This limitation significantly restricts the
+scalability and generalizability of SLAM technologies, impacting their
+development, validation, and deployment. To address this problem, we present
+SubT-MRS, an extremely challenging real-world dataset designed to push SLAM
+towards all-weather environments to pursue the most robust SLAM performance. It
+contains multi-degraded environments including over 30 diverse scenes such as
+structureless corridors, varying lighting conditions, and perceptual obscurants
+like smoke and dust; multimodal sensors such as LiDAR, fisheye camera, IMU, and
+thermal camera; and multiple locomotions like aerial, legged, and wheeled
+robots. We develop accuracy and robustness evaluation tracks for SLAM and
+introduced novel robustness metrics. Comprehensive studies are performed,
+revealing new observations, challenges, and opportunities for future research.",cs.RO,['cs.RO']
+Class Incremental Learning with Multi-Teacher Distillation,Haitao Wen · Lili Pan · Yu Dai · Heqian Qiu · Lanxiao Wang · Qingbo Wu · Hongliang Li, ,https://arxiv.org/abs/2306.17560,,2306.17560.pdf,Class-Incremental Learning using Diffusion Model for Distillation and Replay,"Class-incremental learning aims to learn new classes in an incremental
+fashion without forgetting the previously learned ones. Several research works
+have shown how additional data can be used by incremental models to help
+mitigate catastrophic forgetting. In this work, following the recent
+breakthrough in text-to-image generative models and their wide distribution, we
+propose the use of a pretrained Stable Diffusion model as a source of
+additional data for class-incremental learning. Compared to competitive methods
+that rely on external, often unlabeled, datasets of real images, our approach
+can generate synthetic samples belonging to the same classes as the previously
+encountered images. This allows us to use those additional data samples not
+only in the distillation loss but also for replay in the classification loss.
+Experiments on the competitive benchmarks CIFAR100, ImageNet-Subset, and
+ImageNet demonstrate how this new approach can be used to further improve the
+performance of state-of-the-art methods for class-incremental learning on large
+scale datasets.",cs.LG,"['cs.LG', 'cs.CV']"
+MPOD123: One Image to 3D Content Generation Using Mask-enhanced Progressive Outline-to-Detail Optimization,Jimin Xu · Tianbao Wang · Tao Jin · Shengyu Zhang · Dongjie Fu · Zhe Wang · Jiangjing Lyu · Chengfei Lv · Chaoyue Niu · Zhou Yu · Zhou Zhao · Fei Wu,https://mpod-123.github.io/,https://arxiv.org/abs/2306.17843,,2306.17843.pdf,Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors,"We present Magic123, a two-stage coarse-to-fine approach for high-quality,
+textured 3D meshes generation from a single unposed image in the wild using
+both2D and 3D priors. In the first stage, we optimize a neural radiance field
+to produce a coarse geometry. In the second stage, we adopt a memory-efficient
+differentiable mesh representation to yield a high-resolution mesh with a
+visually appealing texture. In both stages, the 3D content is learned through
+reference view supervision and novel views guided by a combination of 2D and 3D
+diffusion priors. We introduce a single trade-off parameter between the 2D and
+3D priors to control exploration (more imaginative) and exploitation (more
+precise) of the generated geometry. Additionally, we employ textual inversion
+and monocular depth regularization to encourage consistent appearances across
+views and to prevent degenerate solutions, respectively. Magic123 demonstrates
+a significant improvement over previous image-to-3D techniques, as validated
+through extensive experiments on synthetic benchmarks and diverse real-world
+images. Our code, models, and generated 3D assets are available at
+https://github.com/guochengqian/Magic123.",cs.CV,['cs.CV']
+RadSimReal: Bridging the Gap Between Synthetic and Real Data in Radar Object Detection With Simulation,Oded Bialer · Yuval Haitman,https://yuvalhg.github.io/RadSimReal/,https://arxiv.org/abs/2404.18150,,2404.18150.pdf,RadSimReal: Bridging the Gap Between Synthetic and Real Data in Radar Object Detection With Simulation,"Object detection in radar imagery with neural networks shows great potential
+for improving autonomous driving. However, obtaining annotated datasets from
+real radar images, crucial for training these networks, is challenging,
+especially in scenarios with long-range detection and adverse weather and
+lighting conditions where radar performance excels. To address this challenge,
+we present RadSimReal, an innovative physical radar simulation capable of
+generating synthetic radar images with accompanying annotations for various
+radar types and environmental conditions, all without the need for real data
+collection. Remarkably, our findings demonstrate that training object detection
+models on RadSimReal data and subsequently evaluating them on real-world data
+produce performance levels comparable to models trained and tested on real data
+from the same dataset, and even achieves better performance when testing across
+different real datasets. RadSimReal offers advantages over other physical radar
+simulations that it does not necessitate knowledge of the radar design details,
+which are often not disclosed by radar suppliers, and has faster run-time. This
+innovative tool has the potential to advance the development of computer vision
+algorithms for radar-based autonomous driving applications.",cs.CV,['cs.CV']
+AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error,Jonas Ricker · Denis Lukovnikov · Asja Fischer, ,https://arxiv.org/abs/2401.17879,,2401.17879.pdf,AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error,"With recent text-to-image models, anyone can generate deceptively realistic
+images with arbitrary contents, fueling the growing threat of visual
+disinformation. A key enabler for generating high-resolution images with low
+computational cost has been the development of latent diffusion models (LDMs).
+In contrast to conventional diffusion models, LDMs perform the denoising
+process in the low-dimensional latent space of a pre-trained autoencoder (AE)
+instead of the high-dimensional image space. Despite their relevance, the
+forensic analysis of LDMs is still in its infancy. In this work we propose
+AEROBLADE, a novel detection method which exploits an inherent component of
+LDMs: the AE used to transform images between image and latent space. We find
+that generated images can be more accurately reconstructed by the AE than real
+images, allowing for a simple detection approach based on the reconstruction
+error. Most importantly, our method is easy to implement and does not require
+any training, yet nearly matches the performance of detectors that rely on
+extensive training. We empirically demonstrate that AEROBLADE is effective
+against state-of-the-art LDMs, including Stable Diffusion and Midjourney.
+Beyond detection, our approach allows for the qualitative analysis of images,
+which can be leveraged for identifying inpainted regions. We release our code
+and data at https://github.com/jonasricker/aeroblade .",cs.CV,['cs.CV']
+"Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance",Zan Wang · Yixin Chen · Baoxiong Jia · Puhao Li · Jinlu Zhang · Jingze Zhang · Tengyu Liu · Yixin Zhu · Wei Liang · Siyuan Huang,https://afford-motion.github.io/,https://arxiv.org/abs/2403.18036,,2403.18036.pdf,"Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance","Despite significant advancements in text-to-motion synthesis, generating
+language-guided human motion within 3D environments poses substantial
+challenges. These challenges stem primarily from (i) the absence of powerful
+generative models capable of jointly modeling natural language, 3D scenes, and
+human motion, and (ii) the generative models' intensive data requirements
+contrasted with the scarcity of comprehensive, high-quality,
+language-scene-motion datasets. To tackle these issues, we introduce a novel
+two-stage framework that employs scene affordance as an intermediate
+representation, effectively linking 3D scene grounding and conditional motion
+generation. Our framework comprises an Affordance Diffusion Model (ADM) for
+predicting explicit affordance map and an Affordance-to-Motion Diffusion Model
+(AMDM) for generating plausible human motions. By leveraging scene affordance
+maps, our method overcomes the difficulty in generating human motion under
+multimodal condition signals, especially when training with limited data
+lacking extensive language-scene-motion pairs. Our extensive experiments
+demonstrate that our approach consistently outperforms all baselines on
+established benchmarks, including HumanML3D and HUMANISE. Additionally, we
+validate our model's exceptional generalization capabilities on a specially
+curated evaluation set featuring previously unseen descriptions and scenes.",cs.CV,['cs.CV']
+SignGraph: A Sign Sequence is Worth Graphs of Nodes,Shiwei Gan · Yafeng Yin · Zhiwei Jiang · Hongkai Wen · Lei Xie · Sanglu Lu,https://github.com/gswycf/SignGraph,,https://www.semanticscholar.org/paper/Towards-Real-Time-Sign-Language-Recognition-and-on-Gan-Yin/dba462bcf68db62a4722c7f220f38461ff981f15,,,,,nan
+Animating General Image with Large Visual Motion Model,Dengsheng Chen · Xiaoming Wei · Xiaolin Wei, ,https://arxiv.org/abs/2311.12886,,2311.12886.pdf,AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance,"Image animation is a key task in computer vision which aims to generate
+dynamic visual content from static image. Recent image animation methods employ
+neural based rendering technique to generate realistic animations. Despite
+these advancements, achieving fine-grained and controllable image animation
+guided by text remains challenging, particularly for open-domain images
+captured in diverse real environments. In this paper, we introduce an open
+domain image animation method that leverages the motion prior of video
+diffusion model. Our approach introduces targeted motion area guidance and
+motion strength guidance, enabling precise control the movable area and its
+motion speed. This results in enhanced alignment between the animated visual
+elements and the prompting text, thereby facilitating a fine-grained and
+interactive animation generation process for intricate motion sequences. We
+validate the effectiveness of our method through rigorous experiments on an
+open-domain dataset, with the results showcasing its superior performance.
+Project page can be found at https://animationai.github.io/AnimateAnything.",cs.CV,['cs.CV']
+DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video,Huiqiang Sun · Xingyi Li · Liao Shen · Xinyi Ye · Ke Xian · Zhiguo Cao, ,https://arxiv.org/abs/2403.10103,,2403.10103.pdf,DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video,"Recent advancements in dynamic neural radiance field methods have yielded
+remarkable outcomes. However, these approaches rely on the assumption of sharp
+input images. When faced with motion blur, existing dynamic NeRF methods often
+struggle to generate high-quality novel views. In this paper, we propose
+DyBluRF, a dynamic radiance field approach that synthesizes sharp novel views
+from a monocular video affected by motion blur. To account for motion blur in
+input images, we simultaneously capture the camera trajectory and object
+Discrete Cosine Transform (DCT) trajectories within the scene. Additionally, we
+employ a global cross-time rendering approach to ensure consistent temporal
+coherence across the entire scene. We curate a dataset comprising diverse
+dynamic scenes that are specifically tailored for our task. Experimental
+results on our dataset demonstrate that our method outperforms existing
+approaches in generating sharp novel views from motion-blurred inputs while
+maintaining spatial-temporal consistency of the scene.",cs.CV,['cs.CV']
+Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification,Tingting Zheng · Kui Jiang · Hongxun Yao,https://vilab.hit.edu.cn/projects/pamil,https://arxiv.org/abs/2403.07939,,2403.07939.pdf,Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification,"Multi-Instance Learning (MIL) has shown impressive performance for
+histopathology whole slide image (WSI) analysis using bags or pseudo-bags. It
+involves instance sampling, feature representation, and decision-making.
+However, existing MIL-based technologies at least suffer from one or more of
+the following problems: 1) requiring high storage and intensive pre-processing
+for numerous instances (sampling); 2) potential over-fitting with limited
+knowledge to predict bag labels (feature representation); 3) pseudo-bag counts
+and prior biases affect model robustness and generalizability
+(decision-making). Inspired by clinical diagnostics, using the past sampling
+instances can facilitate the final WSI analysis, but it is barely explored in
+prior technologies. To break free these limitations, we integrate the dynamic
+instance sampling and reinforcement learning into a unified framework to
+improve the instance selection and feature aggregation, forming a novel Dynamic
+Policy Instance Selection (DPIS) scheme for better and more credible
+decision-making. Specifically, the measurement of feature distance and reward
+function are employed to boost continuous instance sampling. To alleviate the
+over-fitting, we explore the latent global relations among instances for more
+robust and discriminative feature representation while establishing reward and
+punishment mechanisms to correct biases in pseudo-bags using contrastive
+learning. These strategies form the final Dynamic Policy-Driven Adaptive
+Multi-Instance Learning (PAMIL) method for WSI tasks. Extensive experiments
+reveal that our PAMIL method outperforms the state-of-the-art by 3.8\% on
+CAMELYON16 and 4.4\% on TCGA lung cancer datasets.",cs.CV,['cs.CV']
+OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos,Dongyoung Choi · Hyeonjoong Jang · Min H. Kim,https://vclab.kaist.ac.kr/cvpr2024p1,https://arxiv.org/abs/2404.00676,,2404.00676.pdf,OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos,"Omnidirectional cameras are extensively used in various applications to
+provide a wide field of vision. However, they face a challenge in synthesizing
+novel views due to the inevitable presence of dynamic objects, including the
+photographer, in their wide field of view. In this paper, we introduce a new
+approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can
+render static-only scene views, removing and inpainting dynamic objects
+simultaneously. Our approach combines the principles of local radiance fields
+with the bidirectional optimization of omnidirectional rays. Our input is an
+omnidirectional video, and we evaluate the mutual observations of the entire
+angle between the previous and current frames. To reduce ghosting artifacts of
+dynamic objects and inpaint occlusions, we devise a multi-resolution motion
+mask prediction module. Unlike existing methods that primarily separate dynamic
+components through the temporal domain, our method uses multi-resolution neural
+feature planes for precise segmentation, which is more suitable for long
+360-degree videos. Our experiments validate that OmniLocalRF outperforms
+existing methods in both qualitative and quantitative metrics, especially in
+scenarios with complex real-world scenes. In particular, our approach
+eliminates the need for manual interaction, such as drawing motion masks by
+hand and additional pose estimation, making it a highly effective and efficient
+solution.",cs.CV,"['cs.CV', 'cs.GR']"
+VBench: Comprehensive Benchmark Suite for Video Generative Models,Ziqi Huang · Yinan He · Jiashuo Yu · Fan Zhang · Chenyang Si · Yuming Jiang · Yuanhan Zhang · Tianxing Wu · Jin Qingyang · Nattapol Chanpaisit · Yaohui Wang · Xinyuan Chen · Limin Wang · Dahua Lin · Yu Qiao · Ziwei Liu,https://vchitect.github.io/VBench-project/,https://arxiv.org/abs/2311.17982,,2311.17982.pdf,VBench: Comprehensive Benchmark Suite for Video Generative Models,"Video generation has witnessed significant advancements, yet evaluating these
+models remains a challenge. A comprehensive evaluation benchmark for video
+generation is indispensable for two reasons: 1) Existing metrics do not fully
+align with human perceptions; 2) An ideal evaluation system should provide
+insights to inform future developments of video generation. To this end, we
+present VBench, a comprehensive benchmark suite that dissects ""video generation
+quality"" into specific, hierarchical, and disentangled dimensions, each with
+tailored prompts and evaluation methods. VBench has three appealing properties:
+1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation
+(e.g., subject identity inconsistency, motion smoothness, temporal flickering,
+and spatial relationship, etc). The evaluation metrics with fine-grained levels
+reveal individual models' strengths and weaknesses. 2) Human Alignment: We also
+provide a dataset of human preference annotations to validate our benchmarks'
+alignment with human perception, for each evaluation dimension respectively. 3)
+Valuable Insights: We look into current models' ability across various
+evaluation dimensions, and various content types. We also investigate the gaps
+between video and image generation models. We will open-source VBench,
+including all prompts, evaluation methods, generated videos, and human
+preference annotations, and also include more video generation models in VBench
+to drive forward the field of video generation.",cs.CV,['cs.CV']
+Privacy-preserving Optics for Enhancing Protection in Face De-identification,Jhon Lopez · Carlos Hinojosa · Henry Arguello · Bernard Ghanem,https://carloshinojosa.me/project/privacy-face-deid/,https://arxiv.org/abs/2404.00777,,2404.00777.pdf,Privacy-preserving Optics for Enhancing Protection in Face De-identification,"The modern surge in camera usage alongside widespread computer vision
+technology applications poses significant privacy and security concerns.
+Current artificial intelligence (AI) technologies aid in recognizing relevant
+events and assisting in daily tasks in homes, offices, hospitals, etc. The need
+to access or process personal information for these purposes raises privacy
+concerns. While software-level solutions like face de-identification provide a
+good privacy/utility trade-off, they present vulnerabilities to sniffing
+attacks. In this paper, we propose a hardware-level face de-identification
+method to solve this vulnerability. Specifically, our approach first learns an
+optical encoder along with a regression model to obtain a face heatmap while
+hiding the face identity from the source image. We also propose an
+anonymization framework that generates a new face using the privacy-preserving
+image, face heatmap, and a reference face image from a public dataset as input.
+We validate our approach with extensive simulations and hardware experiments.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CR', 'cs.LG', 'eess.IV']"
+Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection,Yicheng Xiao · Zhuoyan Luo · Yong Liu · Yue Ma · Hengwei Bian · Yatai Ji · Yujiu Yang · Xiu Li, ,https://arxiv.org/abs/2311.16464,,2311.16464.pdf,Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection,"Video Moment Retrieval (MR) and Highlight Detection (HD) have attracted
+significant attention due to the growing demand for video analysis. Recent
+approaches treat MR and HD as similar video grounding problems and address them
+together with transformer-based architecture. However, we observe that the
+emphasis of MR and HD differs, with one necessitating the perception of local
+relationships and the other prioritizing the understanding of global contexts.
+Consequently, the lack of task-specific design will inevitably lead to
+limitations in associating the intrinsic specialty of two tasks. To tackle the
+issue, we propose a Unified Video COMprehension framework (UVCOM) to bridge the
+gap and jointly solve MR and HD effectively. By performing progressive
+integration on intra and inter-modality across multi-granularity, UVCOM
+achieves the comprehensive understanding in processing a video. Moreover, we
+present multi-aspect contrastive learning to consolidate the local relation
+modeling and global knowledge accumulation via well aligned multi-modal space.
+Extensive experiments on QVHighlights, Charades-STA, TACoS , YouTube Highlights
+and TVSum datasets demonstrate the effectiveness and rationality of UVCOM which
+outperforms the state-of-the-art methods by a remarkable margin.",cs.CV,"['cs.CV', 'cs.AI']"
+Hyperbolic Learning with Synthetic Captions for Open-World Detection,Fanjie Kong · Yanbei Chen · Jiarui Cai · Davide Modolo, ,https://arxiv.org/abs/2404.05016,,2404.05016.pdf,Hyperbolic Learning with Synthetic Captions for Open-World Detection,"Open-world detection poses significant challenges, as it requires the
+detection of any object using either object class labels or free-form texts.
+Existing related works often use large-scale manual annotated caption datasets
+for training, which are extremely expensive to collect. Instead, we propose to
+transfer knowledge from vision-language models (VLMs) to enrich the
+open-vocabulary descriptions automatically. Specifically, we bootstrap dense
+synthetic captions using pre-trained VLMs to provide rich descriptions on
+different regions in images, and incorporate these captions to train a novel
+detector that generalizes to novel concepts. To mitigate the noise caused by
+hallucination in synthetic captions, we also propose a novel hyperbolic
+vision-language learning approach to impose a hierarchy between visual and
+caption embeddings. We call our detector ``HyperLearner''. We conduct extensive
+experiments on a wide variety of open-world detection benchmarks (COCO, LVIS,
+Object Detection in the Wild, RefCOCO) and our results show that our model
+consistently outperforms existing state-of-the-art methods, such as GLIP,
+GLIPv2 and Grounding DINO, when using the same backbone.",cs.CV,['cs.CV']
+Coherence As Texture -- Passive Textureless 3D Reconstruction by Self-interference,Wei-Yu Chen · Aswin C. Sankaranarayanan · Anat Levin · Matthew O’Toole, ,,https://onlinelibrary.wiley.com/doi/10.1002/lpor.202301155,,,,,nan
+Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment,Angchi Xu · Wei-Shi Zheng, ,https://arxiv.org/abs/2403.19225,,2403.19225.pdf,Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment,"Weakly-supervised action segmentation is a task of learning to partition a
+long video into several action segments, where training videos are only
+accompanied by transcripts (ordered list of actions). Most of existing methods
+need to infer pseudo segmentation for training by serial alignment between all
+frames and the transcript, which is time-consuming and hard to be parallelized
+while training. In this work, we aim to escape from this inefficient alignment
+with massive but redundant frames, and instead to directly localize a few
+action transitions for pseudo segmentation generation, where a transition
+refers to the change from an action segment to its next adjacent one in the
+transcript. As the true transitions are submerged in noisy boundaries due to
+intra-segment visual variation, we propose a novel Action-Transition-Aware
+Boundary Alignment (ATBA) framework to efficiently and effectively filter out
+noisy boundaries and detect transitions. In addition, to boost the semantic
+learning in the case that noise is inevitably present in the pseudo
+segmentation, we also introduce video-level losses to utilize the trusted
+video-level supervision. Extensive experiments show the effectiveness of our
+approach on both performance and training speed.",cs.CV,['cs.CV']
+Physics-aware Hand-object Interaction Denoising,Haowen Luo · Yunze Liu · Li Yi, ,https://arxiv.org/abs/2405.11481,,2405.11481.pdf,Physics-aware Hand-object Interaction Denoising,"The credibility and practicality of a reconstructed hand-object interaction
+sequence depend largely on its physical plausibility. However, due to high
+occlusions during hand-object interaction, physical plausibility remains a
+challenging criterion for purely vision-based tracking methods. To address this
+issue and enhance the results of existing hand trackers, this paper proposes a
+novel physically-aware hand motion de-noising method. Specifically, we
+introduce two learned loss terms that explicitly capture two crucial aspects of
+physical plausibility: grasp credibility and manipulation feasibility. These
+terms are used to train a physically-aware de-noising network. Qualitative and
+quantitative experiments demonstrate that our approach significantly improves
+both fine-grained physical plausibility and overall pose accuracy, surpassing
+current state-of-the-art de-noising methods.",cs.CV,['cs.CV']
+ToNNO: Tomographic Reconstruction of a Neural Network’s Output for Weakly Supervised Segmentation of 3D Medical Images,Marius Schmidt-Mengin · Alexis Benichoux · Shibeshih Belachew · Nikos Komodakis · Nikos Paragios, ,https://arxiv.org/abs/2404.13103,,2404.13103.pdf,ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images,"Annotating lots of 3D medical images for training segmentation models is
+time-consuming. The goal of weakly supervised semantic segmentation is to train
+segmentation models without using any ground truth segmentation masks. Our work
+addresses the case where only image-level categorical labels, indicating the
+presence or absence of a particular region of interest (such as tumours or
+lesions), are available. Most existing methods rely on class activation mapping
+(CAM). We propose a novel approach, ToNNO, which is based on the Tomographic
+reconstruction of a Neural Network's Output. Our technique extracts stacks of
+slices with different angles from the input 3D volume, feeds these slices to a
+2D encoder, and applies the inverse Radon transform in order to reconstruct a
+3D heatmap of the encoder's predictions. This generic method allows to perform
+dense prediction tasks on 3D volumes using any 2D image encoder. We apply it to
+weakly supervised medical image segmentation by training the 2D encoder to
+output high values for slices containing the regions of interest. We test it on
+four large scale medical image datasets and outperform 2D CAM methods. We then
+extend ToNNO by combining tomographic reconstruction with CAM methods,
+proposing Averaged CAM and Tomographic CAM, which obtain even better results.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG']"
+An Aggregation-Free Federated Learning for Tackling Data Heterogeneity,Yuan Wang · Huazhu Fu · Renuga Kanagavelu · Qingsong Wei · Yong Liu · Rick Goh, ,https://arxiv.org/abs/2404.18962,,2404.18962.pdf,An Aggregation-Free Federated Learning for Tackling Data Heterogeneity,"The performance of Federated Learning (FL) hinges on the effectiveness of
+utilizing knowledge from distributed datasets. Traditional FL methods adopt an
+aggregate-then-adapt framework, where clients update local models based on a
+global model aggregated by the server from the previous training round. This
+process can cause client drift, especially with significant cross-client data
+heterogeneity, impacting model performance and convergence of the FL algorithm.
+To address these challenges, we introduce FedAF, a novel aggregation-free FL
+algorithm. In this framework, clients collaboratively learn condensed data by
+leveraging peer knowledge, the server subsequently trains the global model
+using the condensed data and soft labels received from the clients. FedAF
+inherently avoids the issue of client drift, enhances the quality of condensed
+data amid notable data heterogeneity, and improves the global model
+performance. Extensive numerical studies on several popular benchmark datasets
+show FedAF surpasses various state-of-the-art FL algorithms in handling
+label-skew and feature-skew data heterogeneity, leading to superior global
+model accuracy and faster convergence.",cs.CV,"['cs.CV', 'cs.LG']"
+HoloVIC: Large-scale Dataset and Benchmark for Multi-Sensor Holographic Intersection and Vehicle-Infrastructure Cooperative,CONG MA · Qiao Lei · Chengkai Zhu · Kai Liu · Zelong Kong · Liqing · Xueqi Zhou · Yuheng KAN · Wei Wu, ,https://arxiv.org/abs/2403.02640,,2403.02640.pdf,HoloVIC: Large-scale Dataset and Benchmark for Multi-Sensor Holographic Intersection and Vehicle-Infrastructure Cooperative,"Vehicle-to-everything (V2X) is a popular topic in the field of Autonomous
+Driving in recent years. Vehicle-infrastructure cooperation (VIC) becomes one
+of the important research area. Due to the complexity of traffic conditions
+such as blind spots and occlusion, it greatly limits the perception
+capabilities of single-view roadside sensing systems. To further enhance the
+accuracy of roadside perception and provide better information to the vehicle
+side, in this paper, we constructed holographic intersections with various
+layouts to build a large-scale multi-sensor holographic vehicle-infrastructure
+cooperation dataset, called HoloVIC. Our dataset includes 3 different types of
+sensors (Camera, Lidar, Fisheye) and employs 4 sensor-layouts based on the
+different intersections. Each intersection is equipped with 6-18 sensors to
+capture synchronous data. While autonomous vehicles pass through these
+intersections for collecting VIC data. HoloVIC contains in total on 100k+
+synchronous frames from different sensors. Additionally, we annotated 3D
+bounding boxes based on Camera, Fisheye, and Lidar. We also associate the IDs
+of the same objects across different devices and consecutive frames in
+sequence. Based on HoloVIC, we formulated four tasks to facilitate the
+development of related research. We also provide benchmarks for these tasks.",cs.CV,['cs.CV']
+OneFormer3D: One Transformer for Unified Point Cloud Segmentation,Maksim Kolodiazhnyi · Anna Vorontsova · Anton Konushin · Danila Rukhovich,https://github.com/oneformer3d/oneformer3d,https://arxiv.org/abs/2311.14405,,2311.14405.pdf,OneFormer3D: One Transformer for Unified Point Cloud Segmentation,"Semantic, instance, and panoptic segmentation of 3D point clouds have been
+addressed using task-specific models of distinct design. Thereby, the
+similarity of all segmentation tasks and the implicit relationship between them
+have not been utilized effectively. This paper presents a unified, simple, and
+effective model addressing all these tasks jointly. The model, named
+OneFormer3D, performs instance and semantic segmentation consistently, using a
+group of learnable kernels, where each kernel is responsible for generating a
+mask for either an instance or a semantic category. These kernels are trained
+with a transformer-based decoder with unified instance and semantic queries
+passed as an input. Such a design enables training a model end-to-end in a
+single run, so that it achieves top performance on all three segmentation tasks
+simultaneously. Specifically, our OneFormer3D ranks 1st and sets a new
+state-of-the-art (+2.1 mAP50) in the ScanNet test leaderboard. We also
+demonstrate the state-of-the-art results in semantic, instance, and panoptic
+segmentation of ScanNet (+21 PQ), ScanNet200 (+3.8 mAP50), and S3DIS (+0.8
+mIoU) datasets.",cs.CV,['cs.CV']
+Federated Online Adaptation for Deep Stereo,Matteo Poggi · Fabio Tosi,https://fedstereo.github.io/,http://export.arxiv.org/abs/2405.14873,,2405.14873.pdf,Federated Online Adaptation for Deep Stereo,"We introduce a novel approach for adapting deep stereo networks in a
+collaborative manner. By building over principles of federated learning, we
+develop a distributed framework allowing for demanding the optimization process
+to a number of clients deployed in different environments. This makes it
+possible, for a deep stereo network running on resourced-constrained devices,
+to capitalize on the adaptation process carried out by other instances of the
+same architecture, and thus improve its accuracy in challenging environments
+even when it cannot carry out adaptation on its own. Experimental results show
+how federated adaptation performs equivalently to on-device adaptation, and
+even better when dealing with challenging environments.",cs.CV,['cs.CV']
+Learning Transferable Negative Prompts for Out-of-Distribution Detection,Tianqi Li · Guansong Pang · wenjun miao · Xiao Bai · Jin Zheng, ,,https://paperswithcode.com/paper/learning-transferable-negative-prompts-for,,,,,nan
+JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups,Simindokht Jahangard · Zhixi Cai · Shiki Wen · Hamid Rezatofighi, ,https://arxiv.org/abs/2404.04458,,2404.04458.pdf,JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups,"Understanding human social behaviour is crucial in computer vision and
+robotics. Micro-level observations like individual actions fall short,
+necessitating a comprehensive approach that considers individual behaviour,
+intra-group dynamics, and social group levels for a thorough understanding. To
+address dataset limitations, this paper introduces JRDB-Social, an extension of
+JRDB. Designed to fill gaps in human understanding across diverse indoor and
+outdoor social contexts, JRDB-Social provides annotations at three levels:
+individual attributes, intra-group interactions, and social group context. This
+dataset aims to enhance our grasp of human social dynamics for robotic
+applications. Utilizing the recent cutting-edge multi-modal large language
+models, we evaluated our benchmark to explore their capacity to decipher social
+human behaviour.",cs.CV,['cs.CV']
+Region-Based Representations Revisited,Michal Shlapentokh-Rothman · Ansel Blume · Yao Xiao · Yuqun Wu · Sethuraman T V · Heyi Tao · Jae Yong Lee · Wilfredo Torres-Calderon · Yu-Xiong Wang · Derek Hoiem, ,https://arxiv.org/abs/2402.02352,,2402.02352.pdf,Region-Based Representations Revisited,"We investigate whether region-based representations are effective for
+recognition. Regions were once a mainstay in recognition approaches, but pixel
+and patch-based features are now used almost exclusively. We show that recent
+class-agnostic segmenters like SAM can be effectively combined with strong
+unsupervised representations like DINOv2 and used for a wide variety of tasks,
+including semantic segmentation, object-based image retrieval, and multi-image
+analysis. Once the masks and features are extracted, these representations,
+even with linear decoders, enable competitive performance, making them well
+suited to applications that require custom queries. The compactness of the
+representation also makes it well-suited to video analysis and other problems
+requiring inference across many images.",cs.CV,['cs.CV']
+CycleINR: Cycle Implicit Neural Representation for Arbitrary-Scale Volumetric Super-Resolution of Medical Data,Wei Fang · Yuxing Tang · Heng Guo · Mingze Yuan · Tony C. W. MOK · Ke Yan · Jiawen Yao · Xin Chen · Zaiyi Liu · Le Lu · Ling Zhang · Minfeng Xu, ,https://arxiv.org/abs/2404.04878,,2404.04878.pdf,CycleINR: Cycle Implicit Neural Representation for Arbitrary-Scale Volumetric Super-Resolution of Medical Data,"In the realm of medical 3D data, such as CT and MRI images, prevalent
+anisotropic resolution is characterized by high intra-slice but diminished
+inter-slice resolution. The lowered resolution between adjacent slices poses
+challenges, hindering optimal viewing experiences and impeding the development
+of robust downstream analysis algorithms. Various volumetric super-resolution
+algorithms aim to surmount these challenges, enhancing inter-slice resolution
+and overall 3D medical imaging quality. However, existing approaches confront
+inherent challenges: 1) often tailored to specific upsampling factors, lacking
+flexibility for diverse clinical scenarios; 2) newly generated slices
+frequently suffer from over-smoothing, degrading fine details, and leading to
+inter-slice inconsistency. In response, this study presents CycleINR, a novel
+enhanced Implicit Neural Representation model for 3D medical data volumetric
+super-resolution. Leveraging the continuity of the learned implicit function,
+the CycleINR model can achieve results with arbitrary up-sampling rates,
+eliminating the need for separate training. Additionally, we enhance the grid
+sampling in CycleINR with a local attention mechanism and mitigate
+over-smoothing by integrating cycle-consistent loss. We introduce a new metric,
+Slice-wise Noise Level Inconsistency (SNLI), to quantitatively assess
+inter-slice noise level inconsistency. The effectiveness of our approach is
+demonstrated through image quality evaluations on an in-house dataset and a
+downstream task analysis on the Medical Segmentation Decathlon liver tumor
+dataset.",eess.IV,"['eess.IV', 'cs.CV']"
+"Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video",Hongchi Xia · Chih-Hao Lin · Wei-Chiu Ma · Shenlong Wang, ,https://arxiv.org/abs/2404.09833v1,,2404.09833v1.pdf,"Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video","Creating high-quality and interactive virtual environments, such as games and
+simulators, often involves complex and costly manual modeling processes. In
+this paper, we present Video2Game, a novel approach that automatically converts
+videos of real-world scenes into realistic and interactive game environments.
+At the heart of our system are three core components:(i) a neural radiance
+fields (NeRF) module that effectively captures the geometry and visual
+appearance of the scene; (ii) a mesh module that distills the knowledge from
+NeRF for faster rendering; and (iii) a physics module that models the
+interactions and physical dynamics among the objects. By following the
+carefully designed pipeline, one can construct an interactable and actionable
+digital replica of the real world. We benchmark our system on both indoor and
+large-scale outdoor scenes. We show that we can not only produce
+highly-realistic renderings in real-time, but also build interactive games on
+top.",cs.CV,"['cs.CV', 'cs.AI']"
+Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection,Jin Yang · Ping Wei · Huan Li · Ziyang Ren, ,https://arxiv.org/abs/2404.09263,,2404.09263.pdf,Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection,"Video moment retrieval and highlight detection are two highly valuable tasks
+in video understanding, but until recently they have been jointly studied.
+Although existing studies have made impressive advancement recently, they
+predominantly follow the data-driven bottom-up paradigm. Such paradigm
+overlooks task-specific and inter-task effects, resulting in poor model
+performance. In this paper, we propose a novel task-driven top-down framework
+TaskWeave for joint moment retrieval and highlight detection. The framework
+introduces a task-decoupled unit to capture task-specific and common
+representations. To investigate the interplay between the two tasks, we propose
+an inter-task feedback mechanism, which transforms the results of one task as
+guiding masks to assist the other task. Different from existing methods, we
+present a task-dependent joint loss function to optimize the model.
+Comprehensive experiments and in-depth ablation studies on QVHighlights, TVSum,
+and Charades-STA datasets corroborate the effectiveness and flexibility of the
+proposed framework. Codes are available at
+https://github.com/EdenGabriel/TaskWeave.",cs.CV,"['cs.CV', 'cs.AI']"
+Egocentric Full Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement,Jian Wang · Zhe Cao · Diogo Luvizon · Lingjie Liu · Kripasindhu Sarkar · Danhang Tang · Thabo Beeler · Christian Theobalt, ,https://arxiv.org/abs/2311.16495,,2311.16495.pdf,Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement,"In this work, we explore egocentric whole-body motion capture using a single
+fisheye camera, which simultaneously estimates human body and hand motion. This
+task presents significant challenges due to three factors: the lack of
+high-quality datasets, fisheye camera distortion, and human body
+self-occlusion. To address these challenges, we propose a novel approach that
+leverages FisheyeViT to extract fisheye image features, which are subsequently
+converted into pixel-aligned 3D heatmap representations for 3D human body pose
+prediction. For hand tracking, we incorporate dedicated hand detection and hand
+pose estimation networks for regressing 3D hand poses. Finally, we develop a
+diffusion-based whole-body motion prior model to refine the estimated
+whole-body motion while accounting for joint uncertainties. To train these
+networks, we collect a large synthetic dataset, EgoWholeBody, comprising
+840,000 high-quality egocentric images captured across a diverse range of
+whole-body motion sequences. Quantitative and qualitative evaluations
+demonstrate the effectiveness of our method in producing high-quality
+whole-body motion estimates from a single egocentric camera.",cs.CV,['cs.CV']
+PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation,Xinqiao Zhao · Ziqian Yang · Tianhong Dai · Bingfeng Zhang · Jimin Xiao, ,https://arxiv.org/abs/2405.06586,,2405.06586.pdf,Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approach,"Semantic segmentation is a core computer vision problem, but the high costs
+of data annotation have hindered its wide application. Weakly-Supervised
+Semantic Segmentation (WSSS) offers a cost-efficient workaround to extensive
+labeling in comparison to fully-supervised methods by using partial or
+incomplete labels. Existing WSSS methods have difficulties in learning the
+boundaries of objects leading to poor segmentation results. We propose a novel
+and effective framework that addresses these issues by leveraging visual
+foundation models inside the bounding box. Adopting a two-stage WSSS framework,
+our proposed network consists of a pseudo-label generation module and a
+segmentation module. The first stage leverages Segment Anything Model (SAM) to
+generate high-quality pseudo-labels. To alleviate the problem of delineating
+precise boundaries, we adopt SAM inside the bounding box with the help of
+another pre-trained foundation model (e.g., Grounding-DINO). Furthermore, we
+eliminate the necessity of using the supervision of image labels, by employing
+CLIP in classification. Then in the second stage, the generated high-quality
+pseudo-labels are used to train an off-the-shelf segmenter that achieves the
+state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014.",cs.CV,['cs.CV']
+Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation,Haojie Zhang · Yongyi Su · Xun Xu · Kui Jia, ,https://arxiv.org/abs/2312.03502,,2312.03502.pdf,Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation,"The success of large language models has inspired the computer vision
+community to explore image segmentation foundation model that is able to
+zero/few-shot generalize through prompt engineering. Segment-Anything(SAM),
+among others, is the state-of-the-art image segmentation foundation model
+demonstrating strong zero/few-shot generalization. Despite the success, recent
+studies reveal the weakness of SAM under strong distribution shift. In
+particular, SAM performs awkwardly on corrupted natural images, camouflaged
+images, medical images, etc. Motivated by the observations, we aim to develop a
+self-training based strategy to adapt SAM to target distribution. Given the
+unique challenges of large source dataset, high computation cost and incorrect
+pseudo label, we propose a weakly supervised self-training architecture with
+anchor regularization and low-rank finetuning to improve the robustness and
+computation efficiency of adaptation. We validate the effectiveness on 5 types
+of downstream segmentation tasks including natural clean/corrupted images,
+medical images, camouflaged images and robotic images. Our proposed method is
+task-agnostic in nature and outperforms pre-trained SAM and state-of-the-art
+domain adaptation methods on almost all downstream tasks with the same testing
+prompt inputs.",cs.CV,['cs.CV']
+SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,Boyuan Chen · Zhuo Xu · Sean Kirmani · brian ichter · Dorsa Sadigh · Leonidas Guibas · Fei Xia,https://spatial-vlm.github.io/,https://arxiv.org/abs/2401.12168,,2401.12168.pdf,SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,"Understanding and reasoning about spatial relationships is a fundamental
+capability for Visual Question Answering (VQA) and robotics. While Vision
+Language Models (VLM) have demonstrated remarkable performance in certain VQA
+benchmarks, they still lack capabilities in 3D spatial reasoning, such as
+recognizing quantitative relationships of physical objects like distances or
+size differences. We hypothesize that VLMs' limited spatial reasoning
+capability is due to the lack of 3D spatial knowledge in training data and aim
+to solve this problem by training VLMs with Internet-scale spatial reasoning
+data. To this end, we present a system to facilitate this approach. We first
+develop an automatic 3D spatial VQA data generation framework that scales up to
+2 billion VQA examples on 10 million real-world images. We then investigate
+various factors in the training recipe, including data quality, training
+pipeline, and VLM architecture. Our work features the first internet-scale 3D
+spatial reasoning dataset in metric space. By training a VLM on such data, we
+significantly enhance its ability on both qualitative and quantitative spatial
+VQA. Finally, we demonstrate that this VLM unlocks novel downstream
+applications in chain-of-thought spatial reasoning and robotics due to its
+quantitative estimation capability. Project website:
+https://spatial-vlm.github.io/",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG', 'cs.RO']"
+Learning to Transform Dynamically for Better Adversarial Transferability,Rongyi Zhu · Zeliang Zhang · Susan Liang · Zhuo Liu · Chenliang Xu, ,https://arxiv.org/abs/2405.14077,,2405.14077.pdf,Learning to Transform Dynamically for Better Adversarial Transferability,"Adversarial examples, crafted by adding perturbations imperceptible to
+humans, can deceive neural networks. Recent studies identify the adversarial
+transferability across various models, \textit{i.e.}, the cross-model attack
+ability of adversarial samples. To enhance such adversarial transferability,
+existing input transformation-based methods diversify input data with
+transformation augmentation. However, their effectiveness is limited by the
+finite number of available transformations. In our study, we introduce a novel
+approach named Learning to Transform (L2T). L2T increases the diversity of
+transformed images by selecting the optimal combination of operations from a
+pool of candidates, consequently improving adversarial transferability. We
+conceptualize the selection of optimal transformation combinations as a
+trajectory optimization problem and employ a reinforcement learning strategy to
+effectively solve the problem. Comprehensive experiments on the ImageNet
+dataset, as well as practical tests with Google Vision and GPT-4V, reveal that
+L2T surpasses current methodologies in enhancing adversarial transferability,
+thereby confirming its effectiveness and practical significance. The code is
+available at https://github.com/RongyiZhu/L2T.",cs.CV,"['cs.CV', 'cs.AI']"
+Can’t make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models,Himangi Mittal · Nakul Agarwal · Shao-Yuan Lo · Kwonjoon Lee, ,https://arxiv.org/abs/2405.20305,,2405.20305.pdf,Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models,"We introduce PlausiVL, a large video-language model for anticipating action
+sequences that are plausible in the real-world. While significant efforts have
+been made towards anticipating future actions, prior approaches do not take
+into account the aspect of plausibility in an action sequence. To address this
+limitation, we explore the generative capability of a large video-language
+model in our work and further, develop the understanding of plausibility in an
+action sequence by introducing two objective functions, a counterfactual-based
+plausible action sequence learning loss and a long-horizon action repetition
+loss. We utilize temporal logical constraints as well as verb-noun action pair
+logical constraints to create implausible/counterfactual action sequences and
+use them to train the model with plausible action sequence learning loss. This
+loss helps the model to differentiate between plausible and not plausible
+action sequences and also helps the model to learn implicit temporal cues
+crucial for the task of action anticipation. The long-horizon action repetition
+loss puts a higher penalty on the actions that are more prone to repetition
+over a longer temporal window. With this penalization, the model is able to
+generate diverse, plausible action sequences. We evaluate our approach on two
+large-scale datasets, Ego4D and EPIC-Kitchens-100, and show improvements on the
+task of action anticipation.",cs.CV,['cs.CV']
+Adapting to Length Shift: FlexiLength Network for Trajectory Prediction,Yi Xu · Yun Fu, ,https://arxiv.org/abs/2404.00742,,2404.00742.pdf,Adapting to Length Shift: FlexiLength Network for Trajectory Prediction,"Trajectory prediction plays an important role in various applications,
+including autonomous driving, robotics, and scene understanding. Existing
+approaches mainly focus on developing compact neural networks to increase
+prediction precision on public datasets, typically employing a standardized
+input duration. However, a notable issue arises when these models are evaluated
+with varying observation lengths, leading to a significant performance drop, a
+phenomenon we term the Observation Length Shift. To address this issue, we
+introduce a general and effective framework, the FlexiLength Network (FLN), to
+enhance the robustness of existing trajectory prediction techniques against
+varying observation periods. Specifically, FLN integrates trajectory data with
+diverse observation lengths, incorporates FlexiLength Calibration (FLC) to
+acquire temporal invariant representations, and employs FlexiLength Adaptation
+(FLA) to further refine these representations for more accurate future
+trajectory predictions. Comprehensive experiments on multiple datasets, ie,
+ETH/UCY, nuScenes, and Argoverse 1, demonstrate the effectiveness and
+flexibility of our proposed FLN framework.",cs.CV,['cs.CV']
+Learning Group Activity Features Through Person Attribute Prediction,Chihiro Nakatani · Hiroaki Kawashima · Norimichi Ukita, ,https://arxiv.org/abs/2403.02753,,2403.02753.pdf,Learning Group Activity Features Through Person Attribute Prediction,"This paper proposes Group Activity Feature (GAF) learning in which features
+of multi-person activity are learned as a compact latent vector. Unlike prior
+work in which the manual annotation of group activities is required for
+supervised learning, our method learns the GAF through person attribute
+prediction without group activity annotations. By learning the whole network in
+an end-to-end manner so that the GAF is required for predicting the person
+attributes of people in a group, the GAF is trained as the features of
+multi-person activity. As a person attribute, we propose to use a person's
+action class and appearance features because the former is easy to annotate due
+to its simpleness, and the latter requires no manual annotation. In addition,
+we introduce a location-guided attribute prediction to disentangle the complex
+GAF for extracting the features of each target person properly. Various
+experimental results validate that our method outperforms SOTA methods
+quantitatively and qualitatively on two public datasets. Visualization of our
+GAF also demonstrates that our method learns the GAF representing fined-grained
+group activity classes. Code: https://github.com/chihina/GAFL-CVPR2024.",cs.CV,['cs.CV']
+Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation,Xingqun Qi · Jiahao Pan · Peng Li · Ruibin Yuan · Xiaowei Chi · Mengfei Li · Wenhan Luo · Wei Xue · Shanghang Zhang · Qifeng Liu · Yike Guo, ,https://arxiv.org/abs/2311.17532,,2311.17532.pdf,Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation,"Generating vivid and emotional 3D co-speech gestures is crucial for virtual
+avatar animation in human-machine interaction applications. While the existing
+methods enable generating the gestures to follow a single emotion label, they
+overlook that long gesture sequence modeling with emotion transition is more
+practical in real scenes. In addition, the lack of large-scale available
+datasets with emotional transition speech and corresponding 3D human gestures
+also limits the addressing of this task. To fulfill this goal, we first
+incorporate the ChatGPT-4 and an audio inpainting approach to construct the
+high-fidelity emotion transition human speeches. Considering obtaining the
+realistic 3D pose annotations corresponding to the dynamically inpainted
+emotion transition audio is extremely difficult, we propose a novel weakly
+supervised training strategy to encourage authority gesture transitions.
+Specifically, to enhance the coordination of transition gestures w.r.t
+different emotional ones, we model the temporal association representation
+between two different emotional gesture sequences as style guidance and infuse
+it into the transition generation. We further devise an emotion mixture
+mechanism that provides weak supervision based on a learnable mixed emotion
+label for transition gestures. Last, we present a keyframe sampler to supply
+effective initial posture cues in long sequences, enabling us to generate
+diverse gestures. Extensive experiments demonstrate that our method outperforms
+the state-of-the-art models constructed by adapting single emotion-conditioned
+counterparts on our newly defined emotion transition task and datasets. Our
+code and dataset will be released on the project page:
+https://xingqunqi-lab.github.io/Emo-Transition-Gesture/.",cs.CV,['cs.CV']
+FocusMAE: Gallbladder Cancer Detection from Ultrasound Videos with Focused Masked Autoencoders,Soumen Basu · Mayuna Gupta · Chetan Madan · Pankaj Gupta · Chetan Arora,https://gbc-iitd.github.io/focusmae,https://arxiv.org/abs/2403.08848,,2403.08848.pdf,FocusMAE: Gallbladder Cancer Detection from Ultrasound Videos with Focused Masked Autoencoders,"In recent years, automated Gallbladder Cancer (GBC) detection has gained the
+attention of researchers. Current state-of-the-art (SOTA) methodologies relying
+on ultrasound sonography (US) images exhibit limited generalization,
+emphasizing the need for transformative approaches. We observe that individual
+US frames may lack sufficient information to capture disease manifestation.
+This study advocates for a paradigm shift towards video-based GBC detection,
+leveraging the inherent advantages of spatiotemporal representations. Employing
+the Masked Autoencoder (MAE) for representation learning, we address
+shortcomings in conventional image-based methods. We propose a novel design
+called FocusMAE to systematically bias the selection of masking tokens from
+high-information regions, fostering a more refined representation of
+malignancy. Additionally, we contribute the most extensive US video dataset for
+GBC detection. We also note that, this is the first study on US video-based GBC
+detection. We validate the proposed methods on the curated dataset, and report
+a new state-of-the-art (SOTA) accuracy of 96.4% for the GBC detection problem,
+against an accuracy of 84% by current Image-based SOTA - GBCNet, and RadFormer,
+and 94.7% by Video-based SOTA - AdaMAE. We further demonstrate the generality
+of the proposed FocusMAE on a public CT-based Covid detection dataset,
+reporting an improvement in accuracy by 3.3% over current baselines. The source
+code and pretrained models are available at:
+https://gbc-iitd.github.io/focusmae",eess.IV,"['eess.IV', 'cs.CV']"
+Learning to Predict Activity Progress by Self-Supervised Video Alignment,Gerard Donahue · Ehsan Elhamifar, ,https://arxiv.org/abs/2405.15160,,2405.15160.pdf,ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning,"This paper presents a new self-supervised video representation learning
+framework, ARVideo, which autoregressively predicts the next video token in a
+tailored sequence order. Two key designs are included. First, we organize
+autoregressive video tokens into clusters that span both spatially and
+temporally, thereby enabling a richer aggregation of contextual information
+compared to the standard spatial-only or temporal-only clusters. Second, we
+adopt a randomized spatiotemporal prediction order to facilitate learning from
+multi-dimensional data, addressing the limitations of a handcrafted
+spatial-first or temporal-first sequence order. Extensive experiments establish
+ARVideo as an effective paradigm for self-supervised video representation
+learning. For example, when trained with the ViT-B backbone, ARVideo
+competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something
+V2, which are on par with the strong benchmark set by VideoMAE. Importantly,
+ARVideo also demonstrates higher training efficiency, i.e., it trains 14%
+faster and requires 58% less GPU memory compared to VideoMAE.",cs.CV,['cs.CV']
+Revisiting Global Translation Estimation with Feature Tracks,Peilin Tao · Hainan Cui · Mengqi Rong · Shuhan Shen, ,https://arxiv.org/abs/2403.14118,,2403.14118.pdf,From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation,"Machine Translation Quality Estimation (MTQE) is the task of estimating the
+quality of machine-translated text in real time without the need for reference
+translations, which is of great importance for the development of MT. After two
+decades of evolution, QE has yielded a wealth of results. This article provides
+a comprehensive overview of QE datasets, annotation methods, shared tasks,
+methodologies, challenges, and future research directions. It begins with an
+introduction to the background and significance of QE, followed by an
+explanation of the concepts and evaluation metrics for word-level QE,
+sentence-level QE, document-level QE, and explainable QE. The paper categorizes
+the methods developed throughout the history of QE into those based on
+handcrafted features, deep learning, and Large Language Models (LLMs), with a
+further division of deep learning-based methods into classic deep learning and
+those incorporating pre-trained language models (LMs). Additionally, the
+article details the advantages and limitations of each method and offers a
+straightforward comparison of different approaches. Finally, the paper
+discusses the current challenges in QE research and provides an outlook on
+future research directions.",cs.CL,['cs.CL']
+Directed Decentralized Collaboration for Personalized Federated Learning,Yingqi Liu · Yifan Shi · Qinglun Li · Baoyuan Wu · Xueqian Wang · Li Shen, ,https://arxiv.org/abs/2405.17876,,2405.17876.pdf,Decentralized Directed Collaboration for Personalized Federated Learning,"Personalized Federated Learning (PFL) is proposed to find the greatest
+personalized models for each client. To avoid the central failure and
+communication bottleneck in the server-based FL, we concentrate on the
+Decentralized Personalized Federated Learning (DPFL) that performs distributed
+model training in a Peer-to-Peer (P2P) manner. Most personalized works in DPFL
+are based on undirected and symmetric topologies, however, the data,
+computation and communication resources heterogeneity result in large variances
+in the personalized models, which lead the undirected aggregation to suboptimal
+personalized performance and unguaranteed convergence. To address these issues,
+we propose a directed collaboration DPFL framework by incorporating stochastic
+gradient push and partial model personalized, called \textbf{D}ecentralized
+\textbf{Fed}erated \textbf{P}artial \textbf{G}radient \textbf{P}ush
+(\textbf{DFedPGP}). It personalizes the linear classifier in the modern deep
+model to customize the local solution and learns a consensus representation in
+a fully decentralized manner. Clients only share gradients with a subset of
+neighbors based on the directed and asymmetric topologies, which guarantees
+flexible choices for resource efficiency and better convergence. Theoretically,
+we show that the proposed DFedPGP achieves a superior convergence rate of
+$\mathcal{O}(\frac{1}{\sqrt{T}})$ in the general non-convex setting, and prove
+the tighter connectivity among clients will speed up the convergence. The
+proposed method achieves state-of-the-art (SOTA) accuracy in both data and
+computation heterogeneity scenarios, demonstrating the efficiency of the
+directed collaboration and partial gradient push.",cs.LG,"['cs.LG', 'cs.DC', 'math.OC']"
+Towards Calibrated Multi-label Deep Neural Networks,Jiacheng Cheng · Nuno Vasconcelos, ,,https://paperswithcode.com/paper/towards-calibrated-deep-clustering-network,,,,,nan
+PolarRec: Improving Radio Interferometric Data Reconstruction Using Polar Coordinates,Ruoqi Wang · Zhuoyang Chen · Jiayi Zhu · Qiong Luo · Feng Wang, ,https://arxiv.org/abs/2308.14610,,2308.14610.pdf,PolarRec: Radio Interferometric Data Reconstruction with Polar Coordinate Representation,"In radio astronomy, visibility data, which are measurements of wave signals
+from radio telescopes, are transformed into images for observation of distant
+celestial objects. However, these resultant images usually contain both real
+sources and artifacts, due to signal sparsity and other factors. One way to
+obtain cleaner images is to reconstruct samples into dense forms before
+imaging. Unfortunately, existing reconstruction methods often miss some
+components of visibility in frequency domain, so blurred object edges and
+persistent artifacts remain in the images. Furthermore, the computation
+overhead is high on irregular visibility samples due to the data skew. To
+address these problems, we propose PolarRec, a transformer-encoder-conditioned
+reconstruction pipeline with visibility samples converted into the polar
+coordinate representation. This representation matches the way in which radio
+telescopes observe a celestial area as the Earth rotates. As a result,
+visibility samples distribute in the polar system more uniformly than in the
+Cartesian space. Therefore, we propose to use radial distance in the loss
+function, to help reconstruct complete visibility effectively. Also, we group
+visibility samples by their polar angles and propose a group-based encoding
+scheme to improve the efficiency. Our experiments demonstrate that PolarRec
+markedly improves imaging results by faithfully reconstructing all frequency
+components in the visibility domain while significantly reducing the
+computation cost in visibility data encoding. We believe this high-quality and
+high-efficiency imaging of PolarRec will better facilitate astronomers to
+conduct their research.",astro-ph.IM,"['astro-ph.IM', 'cs.AI', 'cs.CV']"
+SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,Rongyuan Wu · Tao Yang · Lingchen Sun · Zhengqiang ZHANG · Shuai Li · Lei Zhang, ,https://arxiv.org/abs/2311.16518,,2311.16518.pdf,SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,"Owe to the powerful generative priors, the pre-trained text-to-image (T2I)
+diffusion models have become increasingly popular in solving the real-world
+image super-resolution problem. However, as a consequence of the heavy quality
+degradation of input low-resolution (LR) images, the destruction of local
+structures can lead to ambiguous image semantics. As a result, the content of
+reproduced high-resolution image may have semantic errors, deteriorating the
+super-resolution performance. To address this issue, we present a
+semantics-aware approach to better preserve the semantic fidelity of generative
+real-world image super-resolution. First, we train a degradation-aware prompt
+extractor, which can generate accurate soft and hard semantic prompts even
+under strong degradation. The hard semantic prompts refer to the image tags,
+aiming to enhance the local perception ability of the T2I model, while the soft
+semantic prompts compensate for the hard ones to provide additional
+representation information. These semantic prompts can encourage the T2I model
+to generate detailed and semantically accurate results. Furthermore, during the
+inference process, we integrate the LR images into the initial sampling noise
+to mitigate the diffusion model's tendency to generate excessive random
+details. The experiments show that our method can reproduce more realistic
+image details and hold better the semantics.",cs.CV,['cs.CV']
+PanoContext-Former: Panoramic Total Scene Understanding with a Transformer,Yuan Dong · Chuan Fang · Liefeng Bo · Zilong Dong · Ping Tan,https://fangchuan.github.io/PanoContext-Former/,https://arxiv.org/abs/2312.07378v1,,2312.07378v1.pdf,X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer,"The field of 4D point cloud understanding is rapidly developing with the goal
+of analyzing dynamic 3D point cloud sequences. However, it remains a
+challenging task due to the sparsity and lack of texture in point clouds.
+Moreover, the irregularity of point cloud poses a difficulty in aligning
+temporal information within video sequences. To address these issues, we
+propose a novel cross-modal knowledge transfer framework, called
+X4D-SceneFormer. This framework enhances 4D-Scene understanding by transferring
+texture priors from RGB sequences using a Transformer architecture with
+temporal relationship mining. Specifically, the framework is designed with a
+dual-branch architecture, consisting of an 4D point cloud transformer and a
+Gradient-aware Image Transformer (GIT). During training, we employ multiple
+knowledge transfer techniques, including temporal consistency losses and masked
+self-attention, to strengthen the knowledge transfer between modalities. This
+leads to enhanced performance during inference using single-modal 4D point
+cloud inputs. Extensive experiments demonstrate the superior performance of our
+framework on various 4D point cloud video understanding tasks, including action
+recognition, action segmentation and semantic segmentation. The results achieve
+1st places, i.e., 85.3% (+7.9%) accuracy and 47.3% (+5.0%) mIoU for 4D action
+segmentation and semantic segmentation, on the HOI4D
+challenge\footnote{\url{http://www.hoi4d.top/}.}, outperforming previous
+state-of-the-art by a large margin. We release the code at
+https://github.com/jinglinglingling/X4D",cs.CV,['cs.CV']
+Text-image Alignment for Diffusion-based Perception,Neehar Kondapaneni · Markus Marks · Manuel Knott · Rogério Guimarães · Pietro Perona,https://www.vision.caltech.edu/tadp/,https://arxiv.org/abs/2310.00031,,2310.00031.pdf,Text-image Alignment for Diffusion-based Perception,"Diffusion models are generative models with impressive text-to-image
+synthesis capabilities and have spurred a new wave of creative methods for
+classical machine learning tasks. However, the best way to harness the
+perceptual knowledge of these generative models for visual tasks is still an
+open question. Specifically, it is unclear how to use the prompting interface
+when applying diffusion backbones to vision tasks. We find that automatically
+generated captions can improve text-image alignment and significantly enhance a
+model's cross-attention maps, leading to better perceptual performance. Our
+approach improves upon the current state-of-the-art (SOTA) in diffusion-based
+semantic segmentation on ADE20K and the current overall SOTA for depth
+estimation on NYUv2. Furthermore, our method generalizes to the cross-domain
+setting. We use model personalization and caption modifications to align our
+model to the target domain and find improvements over unaligned baselines. Our
+cross-domain object detection model, trained on Pascal VOC, achieves SOTA
+results on Watercolor2K. Our cross-domain segmentation method, trained on
+Cityscapes, achieves SOTA results on Dark Zurich-val and Nighttime Driving.
+Project page: https://www.vision.caltech.edu/tadp/. Code:
+https://github.com/damaggu/TADP.",cs.CV,['cs.CV']
+DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Iterative Diffusion-Based Refinement,Jiuming Liu · Guangming Wang · Weicai Ye · Chaokang Jiang · Jinru Han · Zhe Liu · Guofeng Zhang · Dalong Du · Hesheng Wang, ,https://arxiv.org/abs/2311.17456,,2311.17456.pdf,DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Diffusion Model,"Scene flow estimation, which aims to predict per-point 3D displacements of
+dynamic scenes, is a fundamental task in the computer vision field. However,
+previous works commonly suffer from unreliable correlation caused by locally
+constrained searching ranges, and struggle with accumulated inaccuracy arising
+from the coarse-to-fine structure. To alleviate these problems, we propose a
+novel uncertainty-aware scene flow estimation network (DifFlow3D) with the
+diffusion probabilistic model. Iterative diffusion-based refinement is designed
+to enhance the correlation robustness and resilience to challenging cases, e.g.
+dynamics, noisy inputs, repetitive patterns, etc. To restrain the generation
+diversity, three key flow-related features are leveraged as conditions in our
+diffusion model. Furthermore, we also develop an uncertainty estimation module
+within diffusion to evaluate the reliability of estimated scene flow. Our
+DifFlow3D achieves state-of-the-art performance, with 24.0% and 29.1% EPE3D
+reduction respectively on FlyingThings3D and KITTI 2015 datasets. Notably, our
+method achieves an unprecedented millimeter-level accuracy (0.0078m in EPE3D)
+on the KITTI dataset. Additionally, our diffusion-based refinement paradigm can
+be readily integrated as a plug-and-play module into existing scene flow
+networks, significantly increasing their estimation accuracy. Codes are
+released at https://github.com/IRMVLab/DifFlow3D.",cs.CV,['cs.CV']
+Mind Artist: Creating Artistic Snapshots with Human Thought,Jiaxuan Chen · Yu Qi · Yueming Wang · Gang Pan, ,https://ar5iv.labs.arxiv.org/html/2309.15729,,2309.15729.pdf,MindGPT: Interpreting What You See with Non-invasive Brain Recordings,"Decoding of seen visual contents with non-invasive brain recordings has
+important scientific and practical values. Efforts have been made to recover
+the seen images from brain signals. However, most existing approaches cannot
+faithfully reflect the visual contents due to insufficient image quality or
+semantic mismatches. Compared with reconstructing pixel-level visual images,
+speaking is a more efficient and effective way to explain visual information.
+Here we introduce a non-invasive neural decoder, termed as MindGPT, which
+interprets perceived visual stimuli into natural languages from fMRI signals.
+Specifically, our model builds upon a visually guided neural encoder with a
+cross-attention mechanism, which permits us to guide latent neural
+representations towards a desired language semantic direction in an end-to-end
+manner by the collaborative use of the large language model GPT. By doing so,
+we found that the neural representations of the MindGPT are explainable, which
+can be used to evaluate the contributions of visual properties to language
+semantics. Our experiments show that the generated word sequences truthfully
+represented the visual information (with essential details) conveyed in the
+seen stimuli. The results also suggested that with respect to language decoding
+tasks, the higher visual cortex (HVC) is more semantically informative than the
+lower visual cortex (LVC), and using only the HVC can recover most of the
+semantic information. The code of the MindGPT model will be publicly available
+at https://github.com/JxuanC/MindGPT.",cs.CV,"['cs.CV', 'cs.AI']"
+Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness,Sibo Wang · Jie Zhang · Zheng Yuan · Shiguang Shan,https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness,https://arxiv.org/html/2401.04350v3,,2401.04350v3.pdf,Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness,"Large-scale pre-trained vision-language models like CLIP have demonstrated
+impressive performance across various tasks, and exhibit remarkable zero-shot
+generalization capability, while they are also vulnerable to imperceptible
+adversarial examples. Existing works typically employ adversarial training
+(fine-tuning) as a defense method against adversarial examples. However, direct
+application to the CLIP model may result in overfitting, compromising the
+model's capacity for generalization. In this paper, we propose Pre-trained
+Model Guided Adversarial Fine-Tuning (PMG-AFT) method, which leverages
+supervision from the original pre-trained model by carefully designing an
+auxiliary branch, to enhance the model's zero-shot adversarial robustness.
+Specifically, PMG-AFT minimizes the distance between the features of
+adversarial examples in the target model and those in the pre-trained model,
+aiming to preserve the generalization features already captured by the
+pre-trained model. Extensive Experiments on 15 zero-shot datasets demonstrate
+that PMG-AFT significantly outperforms the state-of-the-art method, improving
+the top-1 robust accuracy by an average of 4.99%. Furthermore, our approach
+consistently improves clean accuracy by an average of 8.72%. Our code is
+available at
+https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness.",cs.CV,['cs.CV']
+ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image,Marco Pesavento · Yuanlu Xu · Nikolaos Sarafianos · Robert Maier · Ziyan Wang · Chun-Han Yao · Marco Volino · Edmond Boyer · Adrian Hilton · Tony Tung, ,https://arxiv.org/abs/2403.10357,,2403.10357.pdf,ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image,"Recent progress in human shape learning, shows that neural implicit models
+are effective in generating 3D human surfaces from limited number of views, and
+even from a single RGB image. However, existing monocular approaches still
+struggle to recover fine geometric details such as face, hands or cloth
+wrinkles. They are also easily prone to depth ambiguities that result in
+distorted geometries along the camera optical axis. In this paper, we explore
+the benefits of incorporating depth observations in the reconstruction process
+by introducing ANIM, a novel method that reconstructs arbitrary 3D human shapes
+from single-view RGB-D images with an unprecedented level of accuracy. Our
+model learns geometric details from both multi-resolution pixel-aligned and
+voxel-aligned features to leverage depth information and enable spatial
+relationships, mitigating depth ambiguities. We further enhance the quality of
+the reconstructed shape by introducing a depth-supervision strategy, which
+improves the accuracy of the signed distance field estimation of points that
+lie on the reconstructed surface. Experiments demonstrate that ANIM outperforms
+state-of-the-art works that use RGB, surface normals, point cloud or RGB-D data
+as input. In addition, we introduce ANIM-Real, a new multi-modal dataset
+comprising high-quality scans paired with consumer-grade RGB-D camera, and our
+protocol to fine-tune ANIM, enabling high-quality reconstruction from
+real-world human capture.",cs.CV,"['cs.CV', 'cs.GR']"
+GLOW: Global Layout Aware Attacks on Object Detection,Jun Bao · Buyu Liu · Kui Ren · Jun Yu, ,,https://paperswithcode.com/search?q=author:Jun+Yu,,,,,nan
+ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention,Jiawei Wang · Changjian Li,https://enigma-li.github.io/projects/contextSeg/contextSeg.html,https://arxiv.org/abs/2311.16682,,2311.16682.pdf,ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention,"Sketch semantic segmentation is a well-explored and pivotal problem in
+computer vision involving the assignment of pre-defined part labels to
+individual strokes. This paper presents ContextSeg - a simple yet highly
+effective approach to tackling this problem with two stages. In the first
+stage, to better encode the shape and positional information of strokes, we
+propose to predict an extra dense distance field in an autoencoder network to
+reinforce structural information learning. In the second stage, we treat an
+entire stroke as a single entity and label a group of strokes within the same
+semantic part using an auto-regressive Transformer with the default attention
+mechanism. By group-based labeling, our method can fully leverage the context
+information when making decisions for the remaining groups of strokes. Our
+method achieves the best segmentation accuracy compared with state-of-the-art
+approaches on two representative datasets and has been extensively evaluated
+demonstrating its superior performance. Additionally, we offer insights into
+solving part imbalance in training data and the preliminary experiment on
+cross-category training, which can inspire future research in this field.",cs.CV,"['cs.CV', 'cs.GR']"
+GEARS: Local Geometry-aware Hand-object Interaction Synthesis,Keyang Zhou · Bharat Lal Bhatnagar · Jan Lenssen · Gerard Pons-Moll, ,https://arxiv.org/abs/2404.01758,,2404.01758.pdf,GEARS: Local Geometry-aware Hand-object Interaction Synthesis,"Generating realistic hand motion sequences in interaction with objects has
+gained increasing attention with the growing interest in digital humans. Prior
+work has illustrated the effectiveness of employing occupancy-based or
+distance-based virtual sensors to extract hand-object interaction features.
+Nonetheless, these methods show limited generalizability across object
+categories, shapes and sizes. We hypothesize that this is due to two reasons:
+1) the limited expressiveness of employed virtual sensors, and 2) scarcity of
+available training data. To tackle this challenge, we introduce a novel
+joint-centered sensor designed to reason about local object geometry near
+potential interaction regions. The sensor queries for object surface points in
+the neighbourhood of each hand joint. As an important step towards mitigating
+the learning complexity, we transform the points from global frame to hand
+template frame and use a shared module to process sensor features of each
+individual joint. This is followed by a spatio-temporal transformer network
+aimed at capturing correlation among the joints in different dimensions.
+Moreover, we devise simple heuristic rules to augment the limited training
+sequences with vast static hand grasping samples. This leads to a broader
+spectrum of grasping types observed during training, in turn enhancing our
+model's generalization capability. We evaluate on two public datasets, GRAB and
+InterCap, where our method shows superiority over baselines both quantitatively
+and perceptually.",cs.CV,['cs.CV']
+Training Generative Image Super-Resolution Models by Wavelet-Domain Losses Enables Better Control of Artifacts,Cansu Korkmaz · Ahmet Murat Tekalp · Zafer Dogan,https://github.com/mandalinadagi/WGSR,,https://paperswithcode.com/paper/training-generative-image-super-resolution,,,,,nan
+OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning,Geng Xinyu · Jiaming Wang · Jiawei Gong · yuerong xue · Jun Xu · Fanglin Chen · Xiaolin Huang, ,https://arxiv.org/abs/2403.13351v1,,2403.13351v1.pdf,OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning,"Redundancy is a persistent challenge in Capsule Networks (CapsNet),leading to
+high computational costs and parameter counts. Although previous works have
+introduced pruning after the initial capsule layer, dynamic routing's fully
+connected nature and non-orthogonal weight matrices reintroduce redundancy in
+deeper layers. Besides, dynamic routing requires iterating to converge, further
+increasing computational demands. In this paper, we propose an Orthogonal
+Capsule Network (OrthCaps) to reduce redundancy, improve routing performance
+and decrease parameter counts. Firstly, an efficient pruned capsule layer is
+introduced to discard redundant capsules. Secondly, dynamic routing is replaced
+with orthogonal sparse attention routing, eliminating the need for iterations
+and fully connected structures. Lastly, weight matrices during routing are
+orthogonalized to sustain low capsule similarity, which is the first approach
+to introduce orthogonality into CapsNet as far as we know. Our experiments on
+baseline datasets affirm the efficiency and robustness of OrthCaps in
+classification tasks, in which ablation studies validate the criticality of
+each component. Remarkably, OrthCaps-Shallow outperforms other Capsule Network
+benchmarks on four datasets, utilizing only 110k parameters, which is a mere
+1.25% of a standard Capsule Network's total. To the best of our knowledge, it
+achieves the smallest parameter count among existing Capsule Networks.
+Similarly, OrthCaps-Deep demonstrates competitive performance across four
+datasets, utilizing only 1.2% of the parameters required by its counterparts.",cs.CV,['cs.CV']
+Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping,Alex Costanzino · Pierluigi Zama Ramirez · Giuseppe Lisanti · Luigi Di Stefano,https://cvlab-unibo.github.io/CrossmodalFeatureMapping/,https://arxiv.org/abs/2312.04521,,2312.04521.pdf,Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping,"The paper explores the industrial multimodal Anomaly Detection (AD) task,
+which exploits point clouds and RGB images to localize anomalies. We introduce
+a novel light and fast framework that learns to map features from one modality
+to the other on nominal samples. At test time, anomalies are detected by
+pinpointing inconsistencies between observed and mapped features. Extensive
+experiments show that our approach achieves state-of-the-art detection and
+segmentation performance in both the standard and few-shot settings on the
+MVTec 3D-AD dataset while achieving faster inference and occupying less memory
+than previous multimodal AD methods. Moreover, we propose a layer-pruning
+technique to improve memory and time efficiency with a marginal sacrifice in
+performance.",cs.CV,['cs.CV']
+Discover and Mitigate Multiple Biased Subgroups in Image Classifiers,Zeliang Zhang · Mingqian Feng · Zhiheng Li · Chenliang Xu, ,https://arxiv.org/abs/2403.12777,,2403.12777.pdf,Discover and Mitigate Multiple Biased Subgroups in Image Classifiers,"Machine learning models can perform well on in-distribution data but often
+fail on biased subgroups that are underrepresented in the training data,
+hindering the robustness of models for reliable applications. Such subgroups
+are typically unknown due to the absence of subgroup labels. Discovering biased
+subgroups is the key to understanding models' failure modes and further
+improving models' robustness. Most previous works of subgroup discovery make an
+implicit assumption that models only underperform on a single biased subgroup,
+which does not hold on in-the-wild data where multiple biased subgroups exist.
+  In this work, we propose Decomposition, Interpretation, and Mitigation (DIM),
+a novel method to address a more challenging but also more practical problem of
+discovering multiple biased subgroups in image classifiers. Our approach
+decomposes the image features into multiple components that represent multiple
+subgroups. This decomposition is achieved via a bilinear dimension reduction
+method, Partial Least Square (PLS), guided by useful supervision from the image
+classifier. We further interpret the semantic meaning of each subgroup
+component by generating natural language descriptions using vision-language
+foundation models. Finally, DIM mitigates multiple biased subgroups
+simultaneously via two strategies, including the data- and model-centric
+strategies. Extensive experiments on CIFAR-100 and Breeds datasets demonstrate
+the effectiveness of DIM in discovering and mitigating multiple biased
+subgroups. Furthermore, DIM uncovers the failure modes of the classifier on
+Hard ImageNet, showcasing its broader applicability to understanding model bias
+in image classifiers. The code is available at
+https://github.com/ZhangAIPI/DIM.",cs.CV,"['cs.CV', 'cs.AI']"
+RMT: Retentive Networks Meet Vision Transformers,Qihang Fan · Huaibo Huang · Mingrui Chen · Hongmin Liu · Ran He,https://github.com/qhfan/RMT,https://arxiv.org/abs/2309.11523,,2309.11523.pdf,RMT: Retentive Networks Meet Vision Transformers,"Vision Transformer (ViT) has gained increasing attention in the computer
+vision community in recent years. However, the core component of ViT,
+Self-Attention, lacks explicit spatial priors and bears a quadratic
+computational complexity, thereby constraining the applicability of ViT. To
+alleviate these issues, we draw inspiration from the recent Retentive Network
+(RetNet) in the field of NLP, and propose RMT, a strong vision backbone with
+explicit spatial prior for general purposes. Specifically, we extend the
+RetNet's temporal decay mechanism to the spatial domain, and propose a spatial
+decay matrix based on the Manhattan distance to introduce the explicit spatial
+prior to Self-Attention. Additionally, an attention decomposition form that
+adeptly adapts to explicit spatial prior is proposed, aiming to reduce the
+computational burden of modeling global information without disrupting the
+spatial decay matrix. Based on the spatial decay matrix and the attention
+decomposition form, we can flexibly integrate explicit spatial prior into the
+vision backbone with linear complexity. Extensive experiments demonstrate that
+RMT exhibits exceptional performance across various vision tasks. Specifically,
+without extra training data, RMT achieves **84.8%** and **86.1%** top-1 acc on
+ImageNet-1k with **27M/4.5GFLOPs** and **96M/18.2GFLOPs**. For downstream
+tasks, RMT achieves **54.5** box AP and **47.2** mask AP on the COCO detection
+task, and **52.8** mIoU on the ADE20K semantic segmentation task. Code is
+available at https://github.com/qhfan/RMT",cs.CV,['cs.CV']
+No More Ambiguity in 360$^\circ$ Room Layout via Bi-Layout Estimation,Yu-Ju Tsai · Jin-Cheng Jhang · JINGJING ZHENG · Wei Wang · Albert Chen · Min Sun · Cheng-Hao Kuo · Ming-Hsuan Yang, ,https://arxiv.org/abs/2404.09993,,2404.09993.pdf,No More Ambiguity in 360° Room Layout via Bi-Layout Estimation,"Inherent ambiguity in layout annotations poses significant challenges to
+developing accurate 360{\deg} room layout estimation models. To address this
+issue, we propose a novel Bi-Layout model capable of predicting two distinct
+layout types. One stops at ambiguous regions, while the other extends to
+encompass all visible areas. Our model employs two global context embeddings,
+where each embedding is designed to capture specific contextual information for
+each layout type. With our novel feature guidance module, the image feature
+retrieves relevant context from these embeddings, generating layout-aware
+features for precise bi-layout predictions. A unique property of our Bi-Layout
+model is its ability to inherently detect ambiguous regions by comparing the
+two predictions. To circumvent the need for manual correction of ambiguous
+annotations during testing, we also introduce a new metric for disambiguating
+ground truth layouts. Our method demonstrates superior performance on benchmark
+datasets, notably outperforming leading approaches. Specifically, on the
+MatterportLayout dataset, it improves 3DIoU from 81.70% to 82.57% across the
+full test set and notably from 54.80% to 59.97% in subsets with significant
+ambiguity. Project page: https://liagm.github.io/Bi_Layout/",cs.CV,['cs.CV']
+AVID: Any-Length Video Inpainting with Diffusion Model,Zhixing Zhang · Bichen Wu · Xiaoyan Wang · Yaqiao Luo · Luxin Zhang · Yinan Zhao · Peter Vajda · Dimitris N. Metaxas · Licheng Yu,https://zhang-zx.github.io/AVID/,https://arxiv.org/abs/2312.03816,,2312.03816.pdf,AVID: Any-Length Video Inpainting with Diffusion Model,"Recent advances in diffusion models have successfully enabled text-guided
+image inpainting. While it seems straightforward to extend such editing
+capability into the video domain, there have been fewer works regarding
+text-guided video inpainting. Given a video, a masked region at its initial
+frame, and an editing prompt, it requires a model to do infilling at each frame
+following the editing guidance while keeping the out-of-mask region intact.
+There are three main challenges in text-guided video inpainting: ($i$) temporal
+consistency of the edited video, ($ii$) supporting different inpainting types
+at different structural fidelity levels, and ($iii$) dealing with variable
+video length. To address these challenges, we introduce Any-Length Video
+Inpainting with Diffusion Model, dubbed as AVID. At its core, our model is
+equipped with effective motion modules and adjustable structure guidance, for
+fixed-length video inpainting. Building on top of that, we propose a novel
+Temporal MultiDiffusion sampling pipeline with a middle-frame attention
+guidance mechanism, facilitating the generation of videos with any desired
+duration. Our comprehensive experiments show our model can robustly deal with
+various inpainting types at different video duration ranges, with high quality.
+More visualization results are made publicly available at
+https://zhang-zx.github.io/AVID/ .",cs.CV,['cs.CV']
+PaReNeRF: Toward Fast Large-scale Dynamic NeRF with Patch-based Reference,Xiao Tang · Min Yang · Penghui Sun · Hui Li · Yuchao Dai · feng zhu · Hojae Lee, ,https://arxiv.org/abs/2405.08609,,2405.08609.pdf,Dynamic NeRF: A Review,"Neural Radiance Field(NeRF) is an novel implicit method to achieve the 3D
+reconstruction and representation with a high resolution. After the first
+research of NeRF is proposed, NeRF has gained a robust developing power and is
+booming in the 3D modeling, representation and reconstruction areas. However
+the first and most of the followed research projects based on NeRF is static,
+which are weak in the practical applications. Therefore, more researcher are
+interested and focused on the study of dynamic NeRF that is more feasible and
+useful in practical applications or situations. Compared with the static NeRF,
+implementing the Dynamic NeRF is more difficult and complex. But Dynamic is
+more potential in the future even is the basic of Editable NeRF. In this
+review, we made a detailed and abundant statement for the development and
+important implementation principles of Dynamci NeRF. The analysis of main
+principle and development of Dynamic NeRF is from 2021 to 2023, including the
+most of the Dynamic NeRF projects. What is more, with colorful and novel
+special designed figures and table, We also made a detailed comparison and
+analysis of different features of various of Dynamic. Besides, we analyzed and
+discussed the key methods to implement a Dynamic NeRF. The volume of the
+reference papers is large. The statements and comparisons are multidimensional.
+With a reading of this review, the whole development history and most of the
+main design method or principles of Dynamic NeRF can be easy understood and
+gained.",cs.CV,['cs.CV']
+LoCoNet: Long-Short Context Network for Active Speaker Detection,Xizi Wang · Feng Cheng · Gedas Bertasius, ,https://ar5iv.labs.arxiv.org/html/2301.08237,,2301.08237.pdf,LoCoNet: Long-Short Context Network for Active Speaker Detection,"Active Speaker Detection (ASD) aims to identify who is speaking in each frame
+of a video. ASD reasons from audio and visual information from two contexts:
+long-term intra-speaker context and short-term inter-speaker context. Long-term
+intra-speaker context models the temporal dependencies of the same speaker,
+while short-term inter-speaker context models the interactions of speakers in
+the same scene. These two contexts are complementary to each other and can help
+infer the active speaker. Motivated by these observations, we propose LoCoNet,
+a simple yet effective Long-Short Context Network that models the long-term
+intra-speaker context and short-term inter-speaker context. We use
+self-attention to model long-term intra-speaker context due to its
+effectiveness in modeling long-range dependencies, and convolutional blocks
+that capture local patterns to model short-term inter-speaker context.
+Extensive experiments show that LoCoNet achieves state-of-the-art performance
+on multiple datasets, achieving an mAP of 95.2%(+1.1%) on AVA-ActiveSpeaker,
+68.1%(+22%) on Columbia dataset, 97.2%(+2.8%) on Talkies dataset and
+59.7%(+8.0%) on Ego4D dataset. Moreover, in challenging cases where multiple
+speakers are present, or face of active speaker is much smaller than other
+faces in the same scene, LoCoNet outperforms previous state-of-the-art methods
+by 3.4% on the AVA-ActiveSpeaker dataset. The code will be released at
+https://github.com/SJTUwxz/LoCoNet_ASD.",cs.CV,['cs.CV']
+Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization,Ziying Xia · Jian Cheng · Siyu Liu · Yongxiang Hu · Shiguang Wang · Zhang Yijie · Wanli Dang,https://github.com/zyxia1009/CVPR2024-TSPNet,,https://link.springer.com/article/10.1007/s11063-024-11598-w,,,,,nan
+3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos,Jiakai Sun · Han Jiao · Guangyuan Li · Zhanjie Zhang · Lei Zhao · Wei Xing,https://sjojok.github.io/3dgstream/,https://arxiv.org/abs/2403.01444,,2403.01444.pdf,3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos,"Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
+from multi-view videos remains a challenging endeavor. Despite the remarkable
+advancements achieved by current neural rendering techniques, these methods
+generally require complete video sequences for offline training and are not
+capable of real-time rendering. To address these constraints, we introduce
+3DGStream, a method designed for efficient FVV streaming of real-world dynamic
+scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
+seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
+Gaussians (3DGs) to represent the scene. Instead of the na\""ive approach of
+directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
+Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
+the training time and storage required for each FVV frame. Furthermore, we
+propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
+scenes. Experiments demonstrate that 3DGStream achieves competitive performance
+in terms of rendering speed, image quality, training time, and model storage
+when compared with state-of-the-art methods.",cs.CV,['cs.CV']
+Aerial Lifting: Neural Urban Semantic and Building Instance Lifting from Aerial Imagery,Yuqi Zhang · Guanying Chen · Jiaxing Chen · Shuguang Cui,https://zyqz97.github.io/Aerial_Lifting/,https://arxiv.org/abs/2403.11812,,2403.11812.pdf,Aerial Lifting: Neural Urban Semantic and Building Instance Lifting from Aerial Imagery,"We present a neural radiance field method for urban-scale semantic and
+building-level instance segmentation from aerial images by lifting noisy 2D
+labels to 3D. This is a challenging problem due to two primary reasons.
+Firstly, objects in urban aerial images exhibit substantial variations in size,
+including buildings, cars, and roads, which pose a significant challenge for
+accurate 2D segmentation. Secondly, the 2D labels generated by existing
+segmentation methods suffer from the multi-view inconsistency problem,
+especially in the case of aerial images, where each image captures only a small
+portion of the entire scene. To overcome these limitations, we first introduce
+a scale-adaptive semantic label fusion strategy that enhances the segmentation
+of objects of varying sizes by combining labels predicted from different
+altitudes, harnessing the novel-view synthesis capabilities of NeRF. We then
+introduce a novel cross-view instance label grouping strategy based on the 3D
+scene representation to mitigate the multi-view inconsistency problem in the 2D
+instance labels. Furthermore, we exploit multi-view reconstructed depth priors
+to improve the geometric quality of the reconstructed radiance field, resulting
+in enhanced segmentation results. Experiments on multiple real-world
+urban-scale datasets demonstrate that our approach outperforms existing
+methods, highlighting its effectiveness.",cs.CV,['cs.CV']
+NetTrack: Tracking Highly Dynamic Objects with a Net,Guangze Zheng · Shijie Lin · Haobo Zuo · Changhong Fu · Jia Pan, ,https://arxiv.org/abs/2403.11186,,2403.11186.pdf,NetTrack: Tracking Highly Dynamic Objects with a Net,"The complex dynamicity of open-world objects presents non-negligible
+challenges for multi-object tracking (MOT), often manifested as severe
+deformations, fast motion, and occlusions. Most methods that solely depend on
+coarse-grained object cues, such as boxes and the overall appearance of the
+object, are susceptible to degradation due to distorted internal relationships
+of dynamic objects. To address this problem, this work proposes NetTrack, an
+efficient, generic, and affordable tracking framework to introduce fine-grained
+learning that is robust to dynamicity. Specifically, NetTrack constructs a
+dynamicity-aware association with a fine-grained Net, leveraging point-level
+visual cues. Correspondingly, a fine-grained sampler and matching method have
+been incorporated. Furthermore, NetTrack learns object-text correspondence for
+fine-grained localization. To evaluate MOT in extremely dynamic open-world
+scenarios, a bird flock tracking (BFT) dataset is constructed, which exhibits
+high dynamicity with diverse species and open-world scenarios. Comprehensive
+evaluation on BFT validates the effectiveness of fine-grained learning on
+object dynamicity, and thorough transfer experiments on challenging open-world
+benchmarks, i.e., TAO, TAO-OW, AnimalTrack, and GMOT-40, validate the strong
+generalization ability of NetTrack even without finetuning. Project page:
+https://george-zhuang.github.io/nettrack/.",cs.CV,['cs.CV']
+"Advancing Saliency Ranking with Human Fixations: Dataset, Models and Benchmarks",Bowen Deng · Siyang Song · Andrew French · Denis Schluppeck · Michael Pound, ,,https://github.com/topics/saliency-ranking-dateset,,,,,nan
+Breathing Life Into Sketches Using Text-to-Video Priors,Rinon Gal · Yael Vinker · Yuval Alaluf · Amit H. Bermano · Daniel Cohen-Or · Ariel Shamir · Gal Chechik, ,https://arxiv.org/abs/2311.13608,,2311.13608.pdf,Breathing Life Into Sketches Using Text-to-Video Priors,"A sketch is one of the most intuitive and versatile tools humans use to
+convey their ideas visually. An animated sketch opens another dimension to the
+expression of ideas and is widely used by designers for a variety of purposes.
+Animating sketches is a laborious process, requiring extensive experience and
+professional design skills. In this work, we present a method that
+automatically adds motion to a single-subject sketch (hence, ""breathing life
+into it""), merely by providing a text prompt indicating the desired motion. The
+output is a short animation provided in vector representation, which can be
+easily edited. Our method does not require extensive training, but instead
+leverages the motion prior of a large pretrained text-to-video diffusion model
+using a score-distillation loss to guide the placement of strokes. To promote
+natural and smooth motion and to better preserve the sketch's appearance, we
+model the learned motion through two components. The first governs small local
+deformations and the second controls global affine transformations.
+Surprisingly, we find that even models that struggle to generate sketch videos
+on their own can still serve as a useful backbone for animating abstract
+representations.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+BrainWash: A Poisoning Attack to Forget in Continual Learning,Ali Abbasi · Parsa Nooralinejad · Hamed Pirsiavash · Soheil Kolouri, ,https://arxiv.org/abs/2311.11995,,2311.11995.pdf,BrainWash: A Poisoning Attack to Forget in Continual Learning,"Continual learning has gained substantial attention within the deep learning
+community, offering promising solutions to the challenging problem of
+sequential learning. Yet, a largely unexplored facet of this paradigm is its
+susceptibility to adversarial attacks, especially with the aim of inducing
+forgetting. In this paper, we introduce ""BrainWash,"" a novel data poisoning
+method tailored to impose forgetting on a continual learner. By adding the
+BrainWash noise to a variety of baselines, we demonstrate how a trained
+continual learner can be induced to forget its previously learned tasks
+catastrophically, even when using these continual learning baselines. An
+important feature of our approach is that the attacker requires no access to
+previous tasks' data and is armed merely with the model's current parameters
+and the data belonging to the most recent task. Our extensive experiments
+highlight the efficacy of BrainWash, showcasing degradation in performance
+across various regularization-based continual learning methods.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CR']"
+ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis,Xiangjun Gao · Xiaoyu Li · Chaopeng Zhang · Qi Zhang · Yan-Pei Cao · Ying Shan · Long Quan, ,https://arxiv.org/abs/2311.17123,,2311.17123.pdf,ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis,"In this work, we propose a method to address the challenge of rendering a 3D
+human from a single image in a free-view manner. Some existing approaches could
+achieve this by using generalizable pixel-aligned implicit fields to
+reconstruct a textured mesh of a human or by employing a 2D diffusion model as
+guidance with the Score Distillation Sampling (SDS) method, to lift the 2D
+image into 3D space. However, a generalizable implicit field often results in
+an over-smooth texture field, while the SDS method tends to lead to a
+texture-inconsistent novel view with the input image. In this paper, we
+introduce a texture-consistent back view synthesis module that could transfer
+the reference image content to the back view through depth and text-guided
+attention injection. Moreover, to alleviate the color distortion that occurs in
+the side region, we propose a visibility-aware patch consistency regularization
+for texture mapping and refinement combined with the synthesized back view
+texture. With the above techniques, we could achieve high-fidelity and
+texture-consistent human rendering from a single image. Experiments conducted
+on both real and synthetic data demonstrate the effectiveness of our method and
+show that our approach outperforms previous baseline methods.",cs.CV,"['cs.CV', 'cs.AI']"
+NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging,Takahiro Shirakawa · Seiichi Uchida, ,https://arxiv.org/abs/2403.03485,,2403.03485.pdf,NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging,"Layout-aware text-to-image generation is a task to generate multi-object
+images that reflect layout conditions in addition to text conditions. The
+current layout-aware text-to-image diffusion models still have several issues,
+including mismatches between the text and layout conditions and quality
+degradation of generated images. This paper proposes a novel layout-aware
+text-to-image diffusion model called NoiseCollage to tackle these issues.
+During the denoising process, NoiseCollage independently estimates noises for
+individual objects and then crops and merges them into a single noise. This
+operation helps avoid condition mismatches; in other words, it can put the
+right objects in the right places. Qualitative and quantitative evaluations
+show that NoiseCollage outperforms several state-of-the-art models. These
+successful results indicate that the crop-and-merge operation of noises is a
+reasonable strategy to control image generation. We also show that NoiseCollage
+can be integrated with ControlNet to use edges, sketches, and pose skeletons as
+additional conditions. Experimental results show that this integration boosts
+the layout accuracy of ControlNet. The code is available at
+https://github.com/univ-esuty/noisecollage.",cs.CV,['cs.CV']
+SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution,Zhixuan Liang · Yao Mu · Hengbo Ma · Masayoshi Tomizuka · Mingyu Ding · Ping Luo,https://skilldiffuser.github.io/,https://arxiv.org/abs/2312.11598,,2312.11598.pdf,SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution,"Diffusion models have demonstrated strong potential for robotic trajectory
+planning. However, generating coherent trajectories from high-level
+instructions remains challenging, especially for long-range composition tasks
+requiring multiple sequential skills. We propose SkillDiffuser, an end-to-end
+hierarchical planning framework integrating interpretable skill learning with
+conditional diffusion planning to address this problem. At the higher level,
+the skill abstraction module learns discrete, human-understandable skill
+representations from visual observations and language instructions. These
+learned skill embeddings are then used to condition the diffusion model to
+generate customized latent trajectories aligned with the skills. This allows
+generating diverse state trajectories that adhere to the learnable skills. By
+integrating skill learning with conditional trajectory generation,
+SkillDiffuser produces coherent behavior following abstract instructions across
+diverse tasks. Experiments on multi-task robotic manipulation benchmarks like
+Meta-World and LOReL demonstrate state-of-the-art performance and
+human-interpretable skill representations from SkillDiffuser. More
+visualization results and information could be found on our website.",cs.RO,"['cs.RO', 'cs.CV', 'cs.LG']"
+CurveCloudNet: Processing Point Clouds with 1D Structure,Colton Stearns · Alex Fu · Jiateng Liu · Jeong Joon Park · Davis Rempe · Despoina Paschalidou · Leonidas Guibas, ,https://arxiv.org/abs/2312.12743,,2312.12743.pdf,PointeNet: A Lightweight Framework for Effective and Efficient Point Cloud Analysis,"Current methodologies in point cloud analysis predominantly explore 3D
+geometries, often achieved through the introduction of intricate learnable
+geometric extractors in the encoder or by deepening networks with repeated
+blocks. However, these approaches inevitably lead to a significant number of
+learnable parameters, resulting in substantial computational costs and imposing
+memory burdens on CPU/GPU. Additionally, the existing strategies are primarily
+tailored for object-level point cloud classification and segmentation tasks,
+with limited extensions to crucial scene-level applications, such as autonomous
+driving. In response to these limitations, we introduce PointeNet, an efficient
+network designed specifically for point cloud analysis. PointeNet distinguishes
+itself with its lightweight architecture, low training cost, and plug-and-play
+capability, effectively capturing representative features. The network consists
+of a Multivariate Geometric Encoding (MGE) module and an optional
+Distance-aware Semantic Enhancement (DSE) module. The MGE module employs
+operations of sampling, grouping, and multivariate geometric aggregation to
+lightweightly capture and adaptively aggregate multivariate geometric features,
+providing a comprehensive depiction of 3D geometries. The DSE module, designed
+for real-world autonomous driving scenarios, enhances the semantic perception
+of point clouds, particularly for distant points. Our method demonstrates
+flexibility by seamlessly integrating with a classification/segmentation head
+or embedding into off-the-shelf 3D object detection networks, achieving notable
+performance improvements at a minimal cost. Extensive experiments on
+object-level datasets, including ModelNet40, ScanObjectNN, ShapeNetPart, and
+the scene-level dataset KITTI, demonstrate the superior performance of
+PointeNet over state-of-the-art methods in point cloud analysis.",cs.CV,['cs.CV']
+LAN: Learning to Adapt Noise for Image Denoising,Changjin Kim · Tae Hyun Kim · Sungyong Baik, ,https://arxiv.org/abs/2403.15132,,2403.15132.pdf,Transfer CLIP for Generalizable Image Denoising,"Image denoising is a fundamental task in computer vision. While prevailing
+deep learning-based supervised and self-supervised methods have excelled in
+eliminating in-distribution noise, their susceptibility to out-of-distribution
+(OOD) noise remains a significant challenge. The recent emergence of
+contrastive language-image pre-training (CLIP) model has showcased exceptional
+capabilities in open-world image recognition and segmentation. Yet, the
+potential for leveraging CLIP to enhance the robustness of low-level tasks
+remains largely unexplored. This paper uncovers that certain dense features
+extracted from the frozen ResNet image encoder of CLIP exhibit
+distortion-invariant and content-related properties, which are highly desirable
+for generalizable denoising. Leveraging these properties, we devise an
+asymmetrical encoder-decoder denoising network, which incorporates dense
+features including the noisy image and its multi-scale features from the frozen
+ResNet encoder of CLIP into a learnable image decoder to achieve generalizable
+denoising. The progressive feature augmentation strategy is further proposed to
+mitigate feature overfitting and improve the robustness of the learnable
+decoder. Extensive experiments and comparisons conducted across diverse OOD
+noises, including synthetic noise, real-world sRGB noise, and low-dose CT image
+noise, demonstrate the superior generalization ability of our method.",cs.CV,"['cs.CV', 'eess.IV']"
+Structured Gradient-based Interpretations via Norm-Regularized Adversarial Training,Shizhan Gong · Qi Dou · Farzan Farnia, ,https://arxiv.org/abs/2404.04647,,2404.04647.pdf,Structured Gradient-based Interpretations via Norm-Regularized Adversarial Training,"Gradient-based saliency maps have been widely used to explain the decisions
+of deep neural network classifiers. However, standard gradient-based
+interpretation maps, including the simple gradient and integrated gradient
+algorithms, often lack desired structures such as sparsity and connectedness in
+their application to real-world computer vision models. A frequently used
+approach to inducing sparsity structures into gradient-based saliency maps is
+to alter the simple gradient scheme using sparsification or norm-based
+regularization. A drawback with such post-processing methods is their
+frequently-observed significant loss in fidelity to the original simple
+gradient map. In this work, we propose to apply adversarial training as an
+in-processing scheme to train neural networks with structured simple gradient
+maps. We show a duality relation between the regularized norms of the
+adversarial perturbations and gradient-based maps, based on which we design
+adversarial training loss functions promoting sparsity and group-sparsity
+properties in simple gradient maps. We present several numerical results to
+show the influence of our proposed norm-based adversarial training methods on
+the standard gradient-based maps of standard neural network architectures on
+benchmark image datasets.",cs.CV,['cs.CV']
+MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading,Abdallah Dib · Luiz Gustavo Hafemann · Emeline Got · Trevor Anderson · Amin Fadaeinejad · Rafael M. O. Cruz · Marc-André Carbonneau, ,https://arxiv.org/abs/2312.13091v2,,2312.13091v2.pdf,MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading,"Reconstructing an avatar from a portrait image has many applications in
+multimedia, but remains a challenging research problem. Extracting reflectance
+maps and geometry from one image is ill-posed: recovering geometry is a
+one-to-many mapping problem and reflectance and light are difficult to
+disentangle. Accurate geometry and reflectance can be captured under the
+controlled conditions of a light stage, but it is costly to acquire large
+datasets in this fashion. Moreover, training solely with this type of data
+leads to poor generalization with in-the-wild images. This motivates the
+introduction of MoSAR, a method for 3D avatar generation from monocular images.
+We propose a semi-supervised training scheme that improves generalization by
+learning from both light stage and in-the-wild datasets. This is achieved using
+a novel differentiable shading formulation. We show that our approach
+effectively disentangles the intrinsic face parameters, producing relightable
+avatars. As a result, MoSAR estimates a richer set of skin reflectance maps,
+and generates more realistic avatars than existing state-of-the-art methods. We
+also introduce a new dataset, named FFHQ-UV-Intrinsics, the first public
+dataset providing intrinsic face attributes at scale (diffuse, specular,
+ambient occlusion and translucency maps) for a total of 10k subjects. The
+project website and the dataset are available on the following link:
+https://ubisoft-laforge.github.io/character/mosar/",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG', '68T45 (Primary) 68T07, 68T01 (Secondary)', 'I.2.10; I.4; I.3.3; I.5']"
+Cinematic Behavior Transfer via NeRF-based Differentiable Filming,Xuekun Jiang · Anyi Rao · Jingbo Wang · Dahua Lin · Bo Dai, ,https://arxiv.org/abs/2311.17754,,2311.17754.pdf,Cinematic Behavior Transfer via NeRF-based Differentiable Filming,"In the evolving landscape of digital media and video production, the precise
+manipulation and reproduction of visual elements like camera movements and
+character actions are highly desired. Existing SLAM methods face limitations in
+dynamic scenes and human pose estimation often focuses on 2D projections,
+neglecting 3D statuses. To address these issues, we first introduce a reverse
+filming behavior estimation technique. It optimizes camera trajectories by
+leveraging NeRF as a differentiable renderer and refining SMPL tracks. We then
+introduce a cinematic transfer pipeline that is able to transfer various shot
+types to a new 2D video or a 3D virtual environment. The incorporation of 3D
+engine workflow enables superior rendering and control abilities, which also
+achieves a higher rating in the user study.",cs.CV,"['cs.CV', 'cs.GR', 'cs.HC', 'cs.MM']"
+Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation,Xiaohan Lei · Min Wang · Wengang Zhou · Li Li · Houqiang Li,https://xiaohanlei.github.io/projects/IEVE/,https://arxiv.org/abs/2402.17587,,2402.17587.pdf,Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation,"As a new embodied vision task, Instance ImageGoal Navigation (IIN) aims to
+navigate to a specified object depicted by a goal image in an unexplored
+environment.
+  The main challenge of this task lies in identifying the target object from
+different viewpoints while rejecting similar distractors.
+  Existing ImageGoal Navigation methods usually adopt the simple
+Exploration-Exploitation framework and ignore the identification of specific
+instance during navigation.
+  In this work, we propose to imitate the human behaviour of ``getting closer
+to confirm"" when distinguishing objects from a distance.
+  Specifically, we design a new modular navigation framework named
+Instance-aware Exploration-Verification-Exploitation (IEVE) for instance-level
+image goal navigation.
+  Our method allows for active switching among the exploration, verification,
+and exploitation actions, thereby facilitating the agent in making reasonable
+decisions under different situations.
+  On the challenging HabitatMatterport 3D semantic (HM3D-SEM) dataset, our
+method surpasses previous state-of-the-art work, with a classical segmentation
+model (0.684 vs. 0.561 success) or a robust model (0.702 vs. 0.561 success)",cs.CV,"['cs.CV', 'cs.RO']"
+TextNeRF: A Novel Scene-Text Image Synthesis Method based on Neural Radiance Fields,Jialei Cui · Jianwei Du · Wenzhuo Liu · Zhouhui Lian, ,https://arxiv.org/abs/2403.01325,,2403.01325.pdf,NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning,"Neural Radiance Fields (NeRF) have garnered remarkable success in novel view
+synthesis. Nonetheless, the task of generating high-quality images for novel
+views persists as a critical challenge. While the existing efforts have
+exhibited commendable progress, capturing intricate details, enhancing
+textures, and achieving superior Peak Signal-to-Noise Ratio (PSNR) metrics
+warrant further focused attention and advancement. In this work, we propose
+NeRF-VPT, an innovative method for novel view synthesis to address these
+challenges. Our proposed NeRF-VPT employs a cascading view prompt tuning
+paradigm, wherein RGB information gained from preceding rendering outcomes
+serves as instructive visual prompts for subsequent rendering stages, with the
+aspiration that the prior knowledge embedded in the prompts can facilitate the
+gradual enhancement of rendered image quality. NeRF-VPT only requires sampling
+RGB data from previous stage renderings as priors at each training stage,
+without relying on extra guidance or complex techniques. Thus, our NeRF-VPT is
+plug-and-play and can be readily integrated into existing methods. By
+conducting comparative analyses of our NeRF-VPT against several NeRF-based
+approaches on demanding real-scene benchmarks, such as Realistic Synthetic 360,
+Real Forward-Facing, Replica dataset, and a user-captured dataset, we
+substantiate that our NeRF-VPT significantly elevates baseline performance and
+proficiently generates more high-quality novel view images than all the
+compared state-of-the-art methods. Furthermore, the cascading learning of
+NeRF-VPT introduces adaptability to scenarios with sparse inputs, resulting in
+a significant enhancement of accuracy for sparse-view novel view synthesis. The
+source code and dataset are available at
+\url{https://github.com/Freedomcls/NeRF-VPT}.",cs.CV,['cs.CV']
+Sparse Global Matching for Video Frame Interpolation with Large Motion,Chunxu Liu · Guozhen Zhang · Rui Zhao · Limin Wang, ,https://arxiv.org/abs/2404.06913,,2404.06913.pdf,Sparse Global Matching for Video Frame Interpolation with Large Motion,"Large motion poses a critical challenge in Video Frame Interpolation (VFI)
+task. Existing methods are often constrained by limited receptive fields,
+resulting in sub-optimal performance when handling scenarios with large motion.
+In this paper, we introduce a new pipeline for VFI, which can effectively
+integrate global-level information to alleviate issues associated with large
+motion. Specifically, we first estimate a pair of initial intermediate flows
+using a high-resolution feature map for extracting local details. Then, we
+incorporate a sparse global matching branch to compensate for flow estimation,
+which consists of identifying flaws in initial flows and generating sparse flow
+compensation with a global receptive field. Finally, we adaptively merge the
+initial flow estimation with global flow compensation, yielding a more accurate
+intermediate flow. To evaluate the effectiveness of our method in handling
+large motion, we carefully curate a more challenging subset from commonly used
+benchmarks. Our method demonstrates the state-of-the-art performance on these
+VFI subsets with large motion.",cs.CV,['cs.CV']
+StraightPCF: Straight Point Cloud Filtering,Dasith de Silva Edirimuni · Xuequan Lu · Gang Li · Lei Wei · Antonio Robles-Kelly · Hongdong Li,https://ddsediri.github.io/ projects/StraightPCF/,https://arxiv.org/abs/2405.08322,,2405.08322.pdf,StraightPCF: Straight Point Cloud Filtering,"Point cloud filtering is a fundamental 3D vision task, which aims to remove
+noise while recovering the underlying clean surfaces. State-of-the-art methods
+remove noise by moving noisy points along stochastic trajectories to the clean
+surfaces. These methods often require regularization within the training
+objective and/or during post-processing, to ensure fidelity. In this paper, we
+introduce StraightPCF, a new deep learning based method for point cloud
+filtering. It works by moving noisy points along straight paths, thus reducing
+discretization errors while ensuring faster convergence to the clean surfaces.
+We model noisy patches as intermediate states between high noise patch variants
+and their clean counterparts, and design the VelocityModule to infer a constant
+flow velocity from the former to the latter. This constant flow leads to
+straight filtering trajectories. In addition, we introduce a DistanceModule
+that scales the straight trajectory using an estimated distance scalar to
+attain convergence near the clean surface. Our network is lightweight and only
+has $\sim530K$ parameters, being 17% of IterativePFN (a most recent point cloud
+filtering network). Extensive experiments on both synthetic and real-world data
+show our method achieves state-of-the-art results. Our method also demonstrates
+nice distributions of filtered points without the need for regularization. The
+implementation code can be found at: https://github.com/ddsediri/StraightPCF.",cs.CV,['cs.CV']
+MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection,Jakub Micorek · Horst Possegger · Dominik Narnhofer · Horst Bischof · Mateusz Kozinski,https://github.com/jakubmicorek/MULDE-Multiscale-Log-Density-Estimation-via-Denoising-Score-Matching-for-Video-Anomaly-Detection,https://arxiv.org/abs/2403.14497,,2403.14497.pdf,MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection,"We propose a novel approach to video anomaly detection: we treat feature
+vectors extracted from videos as realizations of a random variable with a fixed
+distribution and model this distribution with a neural network. This lets us
+estimate the likelihood of test videos and detect video anomalies by
+thresholding the likelihood estimates. We train our video anomaly detector
+using a modification of denoising score matching, a method that injects
+training data with noise to facilitate modeling its distribution. To eliminate
+hyperparameter selection, we model the distribution of noisy video features
+across a range of noise levels and introduce a regularizer that tends to align
+the models for different levels of noise. At test time, we combine anomaly
+indications at multiple noise scales with a Gaussian mixture model. Running our
+video anomaly detector induces minimal delays as inference requires merely
+extracting the features and forward-propagating them through a shallow neural
+network and a Gaussian mixture model. Our experiments on five popular video
+anomaly detection benchmarks demonstrate state-of-the-art performance, both in
+the object-centric and in the frame-centric setup.",cs.CV,['cs.CV']
+Watermark-embedded Adversarial Examples for Copyright Protection against Diffusion Models,Peifei Zhu · Tsubasa Takahashi · Hirokatsu Kataoka, ,https://arxiv.org/abs/2404.09401,,2404.09401.pdf,Watermark-embedded Adversarial Examples for Copyright Protection against Diffusion Models,"Diffusion Models (DMs) have shown remarkable capabilities in various
+image-generation tasks. However, there are growing concerns that DMs could be
+used to imitate unauthorized creations and thus raise copyright issues. To
+address this issue, we propose a novel framework that embeds personal
+watermarks in the generation of adversarial examples. Such examples can force
+DMs to generate images with visible watermarks and prevent DMs from imitating
+unauthorized images. We construct a generator based on conditional adversarial
+networks and design three losses (adversarial loss, GAN loss, and perturbation
+loss) to generate adversarial examples that have subtle perturbation but can
+effectively attack DMs to prevent copyright violations. Training a generator
+for a personal watermark by our method only requires 5-10 samples within 2-3
+minutes, and once the generator is trained, it can generate adversarial
+examples with that watermark significantly fast (0.2s per image). We conduct
+extensive experiments in various conditional image-generation scenarios.
+Compared to existing methods that generate images with chaotic textures, our
+method adds visible watermarks on the generated images, which is a more
+straightforward way to indicate copyright violations. We also observe that our
+adversarial examples exhibit good transferability across unknown generative
+models. Therefore, this work provides a simple yet powerful way to protect
+copyright from DM-based imitation.",cs.CV,"['cs.CV', 'cs.AI']"
+Dr. Bokeh: DiffeRentiable Occlusion-aware Bokeh Rendering,Yichen Sheng · Zixun Yu · Lu Ling · Zhiwen Cao · Xuaner Zhang · Xin Lu · Ke Xian · Haiting Lin · Bedrich Benes, ,https://arxiv.org/abs/2308.08843,,2308.08843.pdf,Dr.Bokeh: DiffeRentiable Occlusion-aware Bokeh Rendering,"Bokeh is widely used in photography to draw attention to the subject while
+effectively isolating distractions in the background. Computational methods
+simulate bokeh effects without relying on a physical camera lens. However, in
+the realm of digital bokeh synthesis, the two main challenges for bokeh
+synthesis are color bleeding and partial occlusion at object boundaries. Our
+primary goal is to overcome these two major challenges using physics principles
+that define bokeh formation. To achieve this, we propose a novel and accurate
+filtering-based bokeh rendering equation and a physically-based occlusion-aware
+bokeh renderer, dubbed Dr.Bokeh, which addresses the aforementioned challenges
+during the rendering stage without the need of post-processing or data-driven
+approaches. Our rendering algorithm first preprocesses the input RGBD to obtain
+a layered scene representation. Dr.Bokeh then takes the layered representation
+and user-defined lens parameters to render photo-realistic lens blur. By
+softening non-differentiable operations, we make Dr.Bokeh differentiable such
+that it can be plugged into a machine-learning framework. We perform
+quantitative and qualitative evaluations on synthetic and real-world images to
+validate the effectiveness of the rendering quality and the differentiability
+of our method. We show Dr.Bokeh not only outperforms state-of-the-art bokeh
+rendering algorithms in terms of photo-realism but also improves the depth
+quality from depth-from-defocus.",cs.GR,['cs.GR']
+XScale-NVS: Cross-Scale Novel View Synthesis with Hash Featurized Manifold,Guangyu Wang · Jinzhi Zhang · Fan Wang · Ruqi Huang · Lu Fang, ,https://arxiv.org/abs/2403.19517,,2403.19517.pdf,XScale-NVS: Cross-Scale Novel View Synthesis with Hash Featurized Manifold,"We propose XScale-NVS for high-fidelity cross-scale novel view synthesis of
+real-world large-scale scenes. Existing representations based on explicit
+surface suffer from discretization resolution or UV distortion, while implicit
+volumetric representations lack scalability for large scenes due to the
+dispersed weight distribution and surface ambiguity. In light of the above
+challenges, we introduce hash featurized manifold, a novel hash-based
+featurization coupled with a deferred neural rendering framework. This approach
+fully unlocks the expressivity of the representation by explicitly
+concentrating the hash entries on the 2D manifold, thus effectively
+representing highly detailed contents independent of the discretization
+resolution. We also introduce a novel dataset, namely GigaNVS, to benchmark
+cross-scale, high-resolution novel view synthesis of realworld large-scale
+scenes. Our method significantly outperforms competing baselines on various
+real-world scenes, yielding an average LPIPS that is 40% lower than prior
+state-of-the-art on the challenging GigaNVS benchmark. Please see our project
+page at: xscalenvs.github.io.",cs.CV,['cs.CV']
+Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions,Oindrila Saha · Grant Horn · Subhransu Maji,https://github.com/cvl-umass/AdaptCLIPZS/,https://arxiv.org/abs/2401.02460,,2401.02460.pdf,Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions,"The zero-shot performance of existing vision-language models (VLMs) such as
+CLIP is limited by the availability of large-scale, aligned image and text
+datasets in specific domains. In this work, we leverage two complementary
+sources of information -- descriptions of categories generated by large
+language models (LLMs) and abundant, fine-grained image classification datasets
+-- to improve the zero-shot classification performance of VLMs across
+fine-grained domains. On the technical side, we develop methods to train VLMs
+with this ""bag-level"" image-text supervision. We find that simply using these
+attributes at test-time does not improve performance, but our training
+strategy, for example, on the iNaturalist dataset, leads to an average
+improvement of 4-5% in zero-shot classification accuracy for novel categories
+of birds and flowers. Similar improvements are observed in domains where a
+subset of the categories was used to fine-tune the model. By prompting LLMs in
+various ways, we generate descriptions that capture visual appearance, habitat,
+and geographic regions and pair them with existing attributes such as the
+taxonomic structure of the categories. We systematically evaluate their ability
+to improve zero-shot categorization in natural domains. Our findings suggest
+that geographic priors can be just as effective and are complementary to visual
+appearance. Our method also outperforms prior work on prompt-based tuning of
+VLMs. We release the benchmark, consisting of 14 datasets at
+https://github.com/cvl-umass/AdaptCLIPZS , which will contribute to future
+research in zero-shot recognition.",cs.CV,['cs.CV']
+Contrastive Learning for DeepFake Classification and Localization via Multi-Label Ranking,Cheng-Yao Hong · Yen-Chi Hsu · Tyng-Luh Liu, ,https://arxiv.org/abs/2401.01448,,2401.01448.pdf,ProbMCL: Simple Probabilistic Contrastive Learning for Multi-label Visual Classification,"Multi-label image classification presents a challenging task in many domains,
+including computer vision and medical imaging. Recent advancements have
+introduced graph-based and transformer-based methods to improve performance and
+capture label dependencies. However, these methods often include complex
+modules that entail heavy computation and lack interpretability. In this paper,
+we propose Probabilistic Multi-label Contrastive Learning (ProbMCL), a novel
+framework to address these challenges in multi-label image classification
+tasks. Our simple yet effective approach employs supervised contrastive
+learning, in which samples that share enough labels with an anchor image based
+on a decision threshold are introduced as a positive set. This structure
+captures label dependencies by pulling positive pair embeddings together and
+pushing away negative samples that fall below the threshold. We enhance
+representation learning by incorporating a mixture density network into
+contrastive learning and generating Gaussian mixture distributions to explore
+the epistemic uncertainty of the feature encoder. We validate the effectiveness
+of our framework through experimentation with datasets from the computer vision
+and medical imaging domains. Our method outperforms the existing
+state-of-the-art methods while achieving a low computational footprint on both
+datasets. Visualization analyses also demonstrate that ProbMCL-learned
+classifiers maintain a meaningful semantic topology.",cs.CV,"['cs.CV', 'cs.LG']"
+"Towards Automatic Power Battery Detection:  New Challenge, Benchmark Dataset and Baseline",Xiaoqi Zhao · Youwei Pang · Zhenyu Chen · Qian Yu · Lihe Zhang · Hanqi Liu · Jiaming Zuo · Huchuan Lu, ,https://arxiv.org/abs/2312.02528,,2312.02528.pdf,"Towards Automatic Power Battery Detection: New Challenge, Benchmark Dataset and Baseline","We conduct a comprehensive study on a new task named power battery detection
+(PBD), which aims to localize the dense cathode and anode plates endpoints from
+X-ray images to evaluate the quality of power batteries. Existing manufacturers
+usually rely on human eye observation to complete PBD, which makes it difficult
+to balance the accuracy and efficiency of detection. To address this issue and
+drive more attention into this meaningful task, we first elaborately collect a
+dataset, called X-ray PBD, which has $1,500$ diverse X-ray images selected from
+thousands of power batteries of $5$ manufacturers, with $7$ different visual
+interference. Then, we propose a novel segmentation-based solution for PBD,
+termed multi-dimensional collaborative network (MDCNet). With the help of line
+and counting predictors, the representation of the point segmentation branch
+can be improved at both semantic and detail aspects.Besides, we design an
+effective distance-adaptive mask generation strategy, which can alleviate the
+visual challenge caused by the inconsistent distribution density of plates to
+provide MDCNet with stable supervision. Without any bells and whistles, our
+segmentation-based MDCNet consistently outperforms various other corner
+detection, crowd counting and general/tiny object detection-based solutions,
+making it a strong baseline that can help facilitate future research in PBD.
+Finally, we share some potential difficulties and works for future researches.
+The source code and datasets will be publicly available at
+\href{https://github.com/Xiaoqi-Zhao-DLUT/X-ray-PBD}{X-ray PBD}.",cs.CV,['cs.CV']
+SPU-PMD: Self-Supervised Point Cloud Upsampling via Progressive Mesh Deformation,Yanzhe Liu · Rong Chen · Yushi Li · Yixi Li · Xuehou Tan, ,,https://dl.acm.org/doi/10.1109/TPAMI.2023.3287628,,,,,nan
+Bidirectional Autoregessive Diffusion Model for Dance Generation,Canyu Zhang · Youbao Tang · NING Zhang · Ruei-Sung Lin · Mei Han · Jing Xiao · Song Wang, ,https://arxiv.org/abs/2402.04356,,2402.04356.pdf,Bidirectional Autoregressive Diffusion Model for Dance Generation,"Dance serves as a powerful medium for expressing human emotions, but the
+lifelike generation of dance is still a considerable challenge. Recently,
+diffusion models have showcased remarkable generative abilities across various
+domains. They hold promise for human motion generation due to their adaptable
+many-to-many nature. Nonetheless, current diffusion-based motion generation
+models often create entire motion sequences directly and unidirectionally,
+lacking focus on the motion with local and bidirectional enhancement. When
+choreographing high-quality dance movements, people need to take into account
+not only the musical context but also the nearby music-aligned dance motions.
+To authentically capture human behavior, we propose a Bidirectional
+Autoregressive Diffusion Model (BADM) for music-to-dance generation, where a
+bidirectional encoder is built to enforce that the generated dance is
+harmonious in both the forward and backward directions. To make the generated
+dance motion smoother, a local information decoder is built for local motion
+enhancement. The proposed framework is able to generate new motions based on
+the input conditions and nearby motions, which foresees individual motion
+slices iteratively and consolidates all predictions. To further refine the
+synchronicity between the generated dance and the beat, the beat information is
+incorporated as an input to generate better music-aligned dance movements.
+Experimental results demonstrate that the proposed model achieves
+state-of-the-art performance compared to existing unidirectional approaches on
+the prominent benchmark for music-to-dance generation.",cs.SD,"['cs.SD', 'cs.CV', 'eess.AS']"
+Enhancing Intrinsic Features for Debiasing via Investigating Class-Discerning Common Attributes in Bias-Contrastive Pair,Jeonghoon Park · Chaeyeon Chung · Jaegul Choo, ,https://arxiv.org/abs/2404.19250,,2404.19250.pdf,Enhancing Intrinsic Features for Debiasing via Investigating Class-Discerning Common Attributes in Bias-Contrastive Pair,"In the image classification task, deep neural networks frequently rely on
+bias attributes that are spuriously correlated with a target class in the
+presence of dataset bias, resulting in degraded performance when applied to
+data without bias attributes. The task of debiasing aims to compel classifiers
+to learn intrinsic attributes that inherently define a target class rather than
+focusing on bias attributes. While recent approaches mainly focus on
+emphasizing the learning of data samples without bias attributes (i.e.,
+bias-conflicting samples) compared to samples with bias attributes (i.e.,
+bias-aligned samples), they fall short of directly guiding models where to
+focus for learning intrinsic features. To address this limitation, this paper
+proposes a method that provides the model with explicit spatial guidance that
+indicates the region of intrinsic features. We first identify the intrinsic
+features by investigating the class-discerning common features between a
+bias-aligned (BA) sample and a bias-conflicting (BC) sample (i.e.,
+bias-contrastive pair). Next, we enhance the intrinsic features in the BA
+sample that are relatively under-exploited for prediction compared to the BC
+sample. To construct the bias-contrastive pair without using bias information,
+we introduce a bias-negative score that distinguishes BC samples from BA
+samples employing a biased model. The experiments demonstrate that our method
+achieves state-of-the-art performance on synthetic and real-world datasets with
+various levels of bias severity.",cs.CV,['cs.CV']
+ID-Blau: Image Deblurring by Implicit Diffusion-based reBLurring AUgmentation,Jia-Hao Wu · Fu-Jen Tsai · Yan-Tsung Peng · Charles Tsai · Chia-Wen Lin · Yen-Yu Lin,https://github.com/plusgood-steven/ID-Blau,https://arxiv.org/abs/2312.10998v1,,2312.10998v1.pdf,ID-Blau: Image Deblurring by Implicit Diffusion-based reBLurring AUgmentation,"Image deblurring aims to remove undesired blurs from an image captured in a
+dynamic scene. Much research has been dedicated to improving deblurring
+performance through model architectural designs. However, there is little work
+on data augmentation for image deblurring. Since continuous motion causes
+blurred artifacts during image exposure, we aspire to develop a groundbreaking
+blur augmentation method to generate diverse blurred images by simulating
+motion trajectories in a continuous space. This paper proposes Implicit
+Diffusion-based reBLurring AUgmentation (ID-Blau), utilizing a sharp image
+paired with a controllable blur condition map to produce a corresponding
+blurred image. We parameterize the blur patterns of a blurred image with their
+orientations and magnitudes as a pixel-wise blur condition map to simulate
+motion trajectories and implicitly represent them in a continuous space. By
+sampling diverse blur conditions, ID-Blau can generate various blurred images
+unseen in the training set. Experimental results demonstrate that ID-Blau can
+produce realistic blurred images for training and thus significantly improve
+performance for state-of-the-art deblurring models.",cs.CV,['cs.CV']
+SLICE: Stabilized LIME for Consistent Explanations for Image Classification,Revoti Prasad Bora · Kiran Raja · Philipp Terhörst · Raymond Veldhuis · Raghavendra Ramachandra, ,https://arxiv.org/abs/2403.17742,,2403.17742.pdf,Using Stratified Sampling to Improve LIME Image Explanations,"We investigate the use of a stratified sampling approach for LIME Image, a
+popular model-agnostic explainable AI method for computer vision tasks, in
+order to reduce the artifacts generated by typical Monte Carlo sampling. Such
+artifacts are due to the undersampling of the dependent variable in the
+synthetic neighborhood around the image being explained, which may result in
+inadequate explanations due to the impossibility of fitting a linear regressor
+on the sampled data. We then highlight a connection with the Shapley theory,
+where similar arguments about undersampling and sample relevance were suggested
+in the past. We derive all the formulas and adjustment factors required for an
+unbiased stratified sampling estimator. Experiments show the efficacy of the
+proposed approach.",cs.AI,['cs.AI']
+Leak and Learn: An Attacker's Cookbook to Train Using Leaked Data from Federated Learning,Joshua C. Zhao · Ahaan Dabholkar · Atul Sharma · Saurabh Bagchi, ,https://arxiv.org/abs/2403.18144,,2403.18144.pdf,Leak and Learn: An Attacker's Cookbook to Train Using Leaked Data from Federated Learning,"Federated learning is a decentralized learning paradigm introduced to
+preserve privacy of client data. Despite this, prior work has shown that an
+attacker at the server can still reconstruct the private training data using
+only the client updates. These attacks are known as data reconstruction attacks
+and fall into two major categories: gradient inversion (GI) and linear layer
+leakage attacks (LLL). However, despite demonstrating the effectiveness of
+these attacks in breaching privacy, prior work has not investigated the
+usefulness of the reconstructed data for downstream tasks. In this work, we
+explore data reconstruction attacks through the lens of training and improving
+models with leaked data. We demonstrate the effectiveness of both GI and LLL
+attacks in maliciously training models using the leaked data more accurately
+than a benign federated learning strategy. Counter-intuitively, this bump in
+training quality can occur despite limited reconstruction quality or a small
+total number of leaked images. Finally, we show the limitations of these
+attacks for downstream training, individually for GI attacks and for LLL
+attacks.",cs.CR,"['cs.CR', 'cs.CV']"
+Beyond Seen Primitive Concepts and Attribute-Object Compositional Learning,Nirat Saini · Khoi Pham · Abhinav Shrivastava, ,https://arxiv.org/html/2403.05924v1,,2403.05924v1.pdf,CSCNET: Class-Specified Cascaded Network for Compositional Zero-Shot Learning,"Attribute and object (A-O) disentanglement is a fundamental and critical
+problem for Compositional Zero-shot Learning (CZSL), whose aim is to recognize
+novel A-O compositions based on foregone knowledge. Existing methods based on
+disentangled representation learning lose sight of the contextual dependency
+between the A-O primitive pairs. Inspired by this, we propose a novel A-O
+disentangled framework for CZSL, namely Class-specified Cascaded Network
+(CSCNet). The key insight is to firstly classify one primitive and then
+specifies the predicted class as a priori for guiding another primitive
+recognition in a cascaded fashion. To this end, CSCNet constructs
+Attribute-to-Object and Object-to-Attribute cascaded branches, in addition to a
+composition branch modeling the two primitives as a whole. Notably, we devise a
+parametric classifier (ParamCls) to improve the matching between visual and
+semantic embeddings. By improving the A-O disentanglement, our framework
+achieves superior results than previous competitive methods.",cs.CV,['cs.CV']
+Improving Graph Contrastive Learning via Adaptive Positive Sampling,Jiaming Zhuo · Feiyang Qin · Can Cui · Kun Fu · Bingxin Niu · Mengzhu Wang · Yuanfang Guo · Chuan Wang · Zhen Wang · Xiaochun Cao · Liang Yang, ,,https://ieeexplore.ieee.org/document/10181235,,,,,nan
+ViewDiff: 3D-Consistent Image Generation with Text-To-Image Models,Lukas Höllein · Aljaž Božič · Norman Müller · David Novotny · Hung-Yu Tseng · Christian Richardt · Michael Zollhoefer · Matthias Nießner,https://lukashoel.github.io/ViewDiff/,https://arxiv.org/abs/2403.01807,,2403.01807.pdf,ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models,"3D asset generation is getting massive amounts of attention, inspired by the
+recent success of text-guided 2D content creation. Existing text-to-3D methods
+use pretrained text-to-image diffusion models in an optimization problem or
+fine-tune them on synthetic data, which often results in non-photorealistic 3D
+objects without backgrounds. In this paper, we present a method that leverages
+pretrained text-to-image models as a prior, and learn to generate multi-view
+images in a single denoising process from real-world data. Concretely, we
+propose to integrate 3D volume-rendering and cross-frame-attention layers into
+each block of the existing U-Net network of the text-to-image model. Moreover,
+we design an autoregressive generation that renders more 3D-consistent images
+at any viewpoint. We train our model on real-world datasets of objects and
+showcase its capabilities to generate instances with a variety of high-quality
+shapes and textures in authentic surroundings. Compared to the existing
+methods, the results generated by our method are consistent, and have favorable
+visual quality (-30% FID, -37% KID).",cs.CV,['cs.CV']
+FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation,Zijia Lu · Ehsan Elhamifar, ,https://arxiv.org/abs/2308.14900,,2308.14900.pdf,BIT: Bi-Level Temporal Modeling for Efficient Supervised Action Segmentation,"We address the task of supervised action segmentation which aims to partition
+a video into non-overlapping segments, each representing a different action.
+Recent works apply transformers to perform temporal modeling at the
+frame-level, which suffer from high computational cost and cannot well capture
+action dependencies over long temporal horizons. To address these issues, we
+propose an efficient BI-level Temporal modeling (BIT) framework that learns
+explicit action tokens to represent action segments, in parallel performs
+temporal modeling on frame and action levels, while maintaining a low
+computational cost. Our model contains (i) a frame branch that uses convolution
+to learn frame-level relationships, (ii) an action branch that uses transformer
+to learn action-level dependencies with a small set of action tokens and (iii)
+cross-attentions to allow communication between the two branches. We apply and
+extend a set-prediction objective to allow each action token to represent one
+or multiple action segments, thus can avoid learning a large number of tokens
+over long videos with many segments. Thanks to the design of our action branch,
+we can also seamlessly leverage textual transcripts of videos (when available)
+to help action segmentation by using them to initialize the action tokens. We
+evaluate our model on four video datasets (two egocentric and two third-person)
+for action segmentation with and without transcripts, showing that BIT
+significantly improves the state-of-the-art accuracy with much lower
+computational cost (30 times faster) compared to existing transformer-based
+methods.",cs.CV,['cs.CV']
+Misalignment-Robust Frequency Distribution Loss for Image Transformation,Zhangkai Ni · Juncheng Wu · Zian Wang · Wenhan Yang · Hanli Wang · Lin Ma, ,https://arxiv.org/html/2402.18192v1,,2402.18192v1.pdf,Misalignment-Robust Frequency Distribution Loss for Image Transformation,"This paper aims to address a common challenge in deep learning-based image
+transformation methods, such as image enhancement and super-resolution, which
+heavily rely on precisely aligned paired datasets with pixel-level alignments.
+However, creating precisely aligned paired images presents significant
+challenges and hinders the advancement of methods trained on such data. To
+overcome this challenge, this paper introduces a novel and simple Frequency
+Distribution Loss (FDL) for computing distribution distance within the
+frequency domain. Specifically, we transform image features into the frequency
+domain using Discrete Fourier Transformation (DFT). Subsequently, frequency
+components (amplitude and phase) are processed separately to form the FDL loss
+function. Our method is empirically proven effective as a training constraint
+due to the thoughtful utilization of global information in the frequency
+domain. Extensive experimental evaluations, focusing on image enhancement and
+super-resolution tasks, demonstrate that FDL outperforms existing
+misalignment-robust loss functions. Furthermore, we explore the potential of
+our FDL for image style transfer that relies solely on completely misaligned
+data. Our code is available at: https://github.com/eezkni/FDL",cs.CV,"['cs.CV', 'eess.IV']"
+DIEM: Decomposition-Integration Enhancing Multimodal Insights,Xinyi Jiang · Guoming Wang · Junhao Guo · Juncheng Li · Wenqiao Zhang · Rongxing Lu · Siliang Tang, ,,https://ieeexplore.ieee.org/document/10423001,,,,,nan
+Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction,Inhwan Bae · Junoh Lee · Hae-Gon Jeon,https://github.com/InhwanBae/LMTrajectory,https://arxiv.org/abs/2403.18447,,2403.18447.pdf,Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction,"Language models have demonstrated impressive ability in context understanding
+and generative performance. Inspired by the recent success of language
+foundation models, in this paper, we propose LMTraj (Language-based Multimodal
+Trajectory predictor), which recasts the trajectory prediction task into a sort
+of question-answering problem. Departing from traditional numerical regression
+models, which treat the trajectory coordinate sequence as continuous signals,
+we consider them as discrete signals like text prompts. Specially, we first
+transform an input space for the trajectory coordinate into the natural
+language space. Here, the entire time-series trajectories of pedestrians are
+converted into a text prompt, and scene images are described as text
+information through image captioning. The transformed numerical and image data
+are then wrapped into the question-answering template for use in a language
+model. Next, to guide the language model in understanding and reasoning
+high-level knowledge, such as scene context and social relationships between
+pedestrians, we introduce an auxiliary multi-task question and answering. We
+then train a numerical tokenizer with the prompt data. We encourage the
+tokenizer to separate the integer and decimal parts well, and leverage it to
+capture correlations between the consecutive numbers in the language model.
+Lastly, we train the language model using the numerical tokenizer and all of
+the question-answer prompts. Here, we propose a beam-search-based most-likely
+prediction and a temperature-based multimodal prediction to implement both
+deterministic and stochastic inferences. Applying our LMTraj, we show that the
+language-based model can be a powerful pedestrian trajectory predictor, and
+outperforms existing numerical-based predictor methods. Code is publicly
+available at https://github.com/inhwanbae/LMTrajectory .",cs.CL,"['cs.CL', 'cs.CV', 'cs.LG', 'cs.RO']"
+Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors,Yu Zhang · Songpengcheng Xia · Lei Chu · Jiarui Yang · Qi Wu · Ling Pei, ,https://arxiv.org/abs/2312.02196,,2312.02196.pdf,Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors,"This paper introduces a novel human pose estimation approach using sparse
+inertial sensors, addressing the shortcomings of previous methods reliant on
+synthetic data. It leverages a diverse array of real inertial motion capture
+data from different skeleton formats to improve motion diversity and model
+generalization. This method features two innovative components: a
+pseudo-velocity regression model for dynamic motion capture with inertial
+sensors, and a part-based model dividing the body and sensor data into three
+regions, each focusing on their unique characteristics. The approach
+demonstrates superior performance over state-of-the-art models across five
+public datasets, notably reducing pose error by 19\% on the DIP-IMU dataset,
+thus representing a significant improvement in inertial sensor-based human pose
+estimation. Our codes are available at {\url{https://github.com/dx118/dynaip}}.",cs.CV,['cs.CV']
+Learning to Remove Wrinkled Transparent Film with Polarized Prior,Jiaqi Tang · RUIZHENG WU · Xiaogang Xu · Sixing Hu · Ying-Cong Chen,https://jqt.me/_FilmRemoval_/,https://arxiv.org/abs/2403.04368v1,,2403.04368v1.pdf,Learning to Remove Wrinkled Transparent Film with Polarized Prior,"In this paper, we study a new problem, Film Removal (FR), which attempts to
+remove the interference of wrinkled transparent films and reconstruct the
+original information under films for industrial recognition systems. We first
+physically model the imaging of industrial materials covered by the film.
+Considering the specular highlight from the film can be effectively recorded by
+the polarized camera, we build a practical dataset with polarization
+information containing paired data with and without transparent film. We aim to
+remove interference from the film (specular highlights and other degradations)
+with an end-to-end framework. To locate the specular highlight, we use an angle
+estimation network to optimize the polarization angle with the minimized
+specular highlight. The image with minimized specular highlight is set as a
+prior for supporting the reconstruction network. Based on the prior and the
+polarized images, the reconstruction network can decouple all degradations from
+the film. Extensive experiments show that our framework achieves SOTA
+performance in both image reconstruction and industrial downstream tasks. Our
+code will be released at \url{https://github.com/jqtangust/FilmRemoval}.",cs.CV,['cs.CV']
+FCS: Feature Calibration and Separation for Non-Exemplar Class Incremental Learning,Qiwei Li · Yuxin Peng · Jiahuan Zhou, ,https://arxiv.org/abs/2312.12722,,2312.12722.pdf,Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning,"Non-exemplar class incremental learning aims to learn both the new and old
+tasks without accessing any training data from the past. This strict
+restriction enlarges the difficulty of alleviating catastrophic forgetting
+since all techniques can only be applied to current task data. Considering this
+challenge, we propose a novel framework of fine-grained knowledge selection and
+restoration. The conventional knowledge distillation-based methods place too
+strict constraints on the network parameters and features to prevent
+forgetting, which limits the training of new tasks. To loose this constraint,
+we proposed a novel fine-grained selective patch-level distillation to
+adaptively balance plasticity and stability. Some task-agnostic patches can be
+used to preserve the decision boundary of the old task. While some patches
+containing the important foreground are favorable for learning the new task.
+  Moreover, we employ a task-agnostic mechanism to generate more realistic
+prototypes of old tasks with the current task sample for reducing classifier
+bias for fine-grained knowledge restoration. Extensive experiments on CIFAR100,
+TinyImageNet and ImageNet-Subset demonstrate the effectiveness of our method.
+Code is available at https://github.com/scok30/vit-cil.",cs.CV,['cs.CV']
+Video-Based Human Pose Regression via Decoupled Space-Time Aggregation,Jijie He · Wenwu Yang,https://github.com/zgspose/DSTA,https://arxiv.org/abs/2403.19926,,2403.19926.pdf,Video-Based Human Pose Regression via Decoupled Space-Time Aggregation,"By leveraging temporal dependency in video sequences, multi-frame human pose
+estimation algorithms have demonstrated remarkable results in complicated
+situations, such as occlusion, motion blur, and video defocus. These algorithms
+are predominantly based on heatmaps, resulting in high computation and storage
+requirements per frame, which limits their flexibility and real-time
+application in video scenarios, particularly on edge devices. In this paper, we
+develop an efficient and effective video-based human pose regression method,
+which bypasses intermediate representations such as heatmaps and instead
+directly maps the input to the output joint coordinates. Despite the inherent
+spatial correlation among adjacent joints of the human pose, the temporal
+trajectory of each individual joint exhibits relative independence. In light of
+this, we propose a novel Decoupled Space-Time Aggregation network (DSTA) to
+separately capture the spatial contexts between adjacent joints and the
+temporal cues of each individual joint, thereby avoiding the conflation of
+spatiotemporal dimensions. Concretely, DSTA learns a dedicated feature token
+for each joint to facilitate the modeling of their spatiotemporal dependencies.
+With the proposed joint-wise local-awareness attention mechanism, our method is
+capable of efficiently and flexibly utilizing the spatial dependency of
+adjacent joints and the temporal dependency of each joint itself. Extensive
+experiments demonstrate the superiority of our method. Compared to previous
+regression-based single-frame human pose estimation methods, DSTA significantly
+enhances performance, achieving an 8.9 mAP improvement on PoseTrack2017.
+Furthermore, our approach either surpasses or is on par with the
+state-of-the-art heatmap-based multi-frame human pose estimation methods.
+Project page: https://github.com/zgspose/DSTA.",cs.CV,"['cs.CV', 'I.4.9']"
+$L_0$-Sampler: An $L_{0}$ Model Guided Volume Sampling for NeRF,Liangchen Li · Juyong Zhang, ,https://arxiv.org/abs/2311.07044,,2311.07044.pdf,$L_0$-Sampler: An $L_{0}$ Model Guided Volume Sampling for NeRF,"Since being proposed, Neural Radiance Fields (NeRF) have achieved great
+success in related tasks, mainly adopting the hierarchical volume sampling
+(HVS) strategy for volume rendering. However, the HVS of NeRF approximates
+distributions using piecewise constant functions, which provides a relatively
+rough estimation. Based on the observation that a well-trained weight function
+$w(t)$ and the $L_0$ distance between points and the surface have very high
+similarity, we propose $L_0$-Sampler by incorporating the $L_0$ model into
+$w(t)$ to guide the sampling process. Specifically, we propose to use piecewise
+exponential functions rather than piecewise constant functions for
+interpolation, which can not only approximate quasi-$L_0$ weight distributions
+along rays quite well but also can be easily implemented with few lines of code
+without additional computational burden. Stable performance improvements can be
+achieved by applying $L_0$-Sampler to NeRF and its related tasks like 3D
+reconstruction. Code is available at https://ustc3dv.github.io/L0-Sampler/ .",cs.CV,"['cs.CV', 'cs.GR']"
+3DInAction: Understanding Human Actions in 3D Point Clouds,Yizhak Ben-Shabat · Oren Shrout · Stephen Gould, ,https://arxiv.org/html/2303.06346v2,,2303.06346v2.pdf,3DInAction: Understanding Human Actions in 3D Point Clouds,"We propose a novel method for 3D point cloud action recognition.
+Understanding human actions in RGB videos has been widely studied in recent
+years, however, its 3D point cloud counterpart remains under-explored. This is
+mostly due to the inherent limitation of the point cloud data modality -- lack
+of structure, permutation invariance, and varying number of points -- which
+makes it difficult to learn a spatio-temporal representation. To address this
+limitation, we propose the 3DinAction pipeline that first estimates patches
+moving in time (t-patches) as a key building block, alongside a hierarchical
+architecture that learns an informative spatio-temporal representation. We show
+that our method achieves improved performance on existing datasets, including
+DFAUST and IKEA ASM. Code is publicly available at
+https://github.com/sitzikbs/3dincaction.",cs.CV,['cs.CV']
+Poly Kernel Inception Network for Remote Sensing Detection,Xinhao Cai · Qiuxia Lai · Yuwei Wang · Wenguan Wang · Zeren Sun · Yazhou Yao, ,https://arxiv.org/abs/2403.06258,,2403.06258.pdf,Poly Kernel Inception Network for Remote Sensing Detection,"Object detection in remote sensing images (RSIs) often suffers from several
+increasing challenges, including the large variation in object scales and the
+diverse-ranging context. Prior methods tried to address these challenges by
+expanding the spatial receptive field of the backbone, either through
+large-kernel convolution or dilated convolution. However, the former typically
+introduces considerable background noise, while the latter risks generating
+overly sparse feature representations. In this paper, we introduce the Poly
+Kernel Inception Network (PKINet) to handle the above challenges. PKINet
+employs multi-scale convolution kernels without dilation to extract object
+features of varying scales and capture local context. In addition, a Context
+Anchor Attention (CAA) module is introduced in parallel to capture long-range
+contextual information. These two components work jointly to advance the
+performance of PKINet on four challenging remote sensing detection benchmarks,
+namely DOTA-v1.0, DOTA-v1.5, HRSC2016, and DIOR-R.",cs.CV,['cs.CV']
+Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning,Zihua Zhao · Mengxi Chen · Tianjie Dai · Jiangchao Yao · Bo Han · Ya Zhang · Yanfeng Wang, ,https://arxiv.org/abs/2405.16996,,2405.16996.pdf,Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning,"Noisy correspondence that refers to mismatches in cross-modal data pairs, is
+prevalent on human-annotated or web-crawled datasets. Prior approaches to
+leverage such data mainly consider the application of uni-modal noisy label
+learning without amending the impact on both cross-modal and intra-modal
+geometrical structures in multimodal learning. Actually, we find that both
+structures are effective to discriminate noisy correspondence through
+structural differences when being well-established. Inspired by this
+observation, we introduce a Geometrical Structure Consistency (GSC) method to
+infer the true correspondence. Specifically, GSC ensures the preservation of
+geometrical structures within and between modalities, allowing for the accurate
+discrimination of noisy samples based on structural differences. Utilizing
+these inferred true correspondence labels, GSC refines the learning of
+geometrical structures by filtering out the noisy samples. Experiments across
+four cross-modal datasets confirm that GSC effectively identifies noisy samples
+and significantly outperforms the current leading methods.",cs.CV,['cs.CV']
+Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution,Guangyuan Li · Chen Rao · Juncheng Mo · Zhanjie Zhang · Wei Xing · Lei Zhao, ,https://arxiv.org/abs/2404.04785,,2404.04785.pdf,Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution,"Recently, diffusion models (DM) have been applied in magnetic resonance
+imaging (MRI) super-resolution (SR) reconstruction, exhibiting impressive
+performance, especially with regard to detailed reconstruction. However, the
+current DM-based SR reconstruction methods still face the following issues: (1)
+They require a large number of iterations to reconstruct the final image, which
+is inefficient and consumes a significant amount of computational resources.
+(2) The results reconstructed by these methods are often misaligned with the
+real high-resolution images, leading to remarkable distortion in the
+reconstructed MR images. To address the aforementioned issues, we propose an
+efficient diffusion model for multi-contrast MRI SR, named as DiffMSR.
+Specifically, we apply DM in a highly compact low-dimensional latent space to
+generate prior knowledge with high-frequency detail information. The highly
+compact latent space ensures that DM requires only a few simple iterations to
+produce accurate prior knowledge. In addition, we design the Prior-Guide Large
+Window Transformer (PLWformer) as the decoder for DM, which can extend the
+receptive field while fully utilizing the prior knowledge generated by DM to
+ensure that the reconstructed MR image remains undistorted. Extensive
+experiments on public and clinical datasets demonstrate that our DiffMSR
+outperforms state-of-the-art methods.",cs.CV,['cs.CV']
+HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models,Li Pang · Xiangyu Rui · Long Cui · Hongzhong Wang · Deyu Meng · Xiangyong Cao, ,https://arxiv.org/abs/2402.15865,,2402.15865.pdf,HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models,"Hyperspectral image (HSI) restoration aims at recovering clean images from
+degraded observations and plays a vital role in downstream tasks. Existing
+model-based methods have limitations in accurately modeling the complex image
+characteristics with handcraft priors, and deep learning-based methods suffer
+from poor generalization ability. To alleviate these issues, this paper
+proposes an unsupervised HSI restoration framework with pre-trained diffusion
+model (HIR-Diff), which restores the clean HSIs from the product of two
+low-rank components, i.e., the reduced image and the coefficient matrix.
+Specifically, the reduced image, which has a low spectral dimension, lies in
+the image field and can be inferred from our improved diffusion model where a
+new guidance function with total variation (TV) prior is designed to ensure
+that the reduced image can be well sampled. The coefficient matrix can be
+effectively pre-estimated based on singular value decomposition (SVD) and
+rank-revealing QR (RRQR) factorization. Furthermore, a novel exponential noise
+schedule is proposed to accelerate the restoration process (about 5$\times$
+acceleration for denoising) with little performance decrease. Extensive
+experimental results validate the superiority of our method in both performance
+and speed on a variety of HSI restoration tasks, including HSI denoising, noisy
+HSI super-resolution, and noisy HSI inpainting. The code is available at
+https://github.com/LiPang/HIRDiff.",cs.CV,"['cs.CV', 'eess.IV']"
+Zero-Shot Structure-Preserving Diffusion Model for  High Dynamic Range Tone Mapping,Ruoxi Zhu · Shusong Xu · Peiye Liu · Sicheng Li · Yanheng Lu · Dimin Niu · Zihao Liu · Zihao Meng · Li Zhiyong · Xinhua Chen · Yibo Fan, ,https://arxiv.org/abs/2309.16975,,2309.16975.pdf,Perceptual Tone Mapping Model for High Dynamic Range Imaging,"One of the key challenges in tone mapping is to preserve the perceptual
+quality of high dynamic range (HDR) images when mapping them to standard
+dynamic range (SDR) displays. Traditional tone mapping operators (TMOs)
+compress the luminance of HDR images without considering the surround and
+display conditions emanating into suboptimal results. Current research
+addresses this challenge by incorporating perceptual color appearance
+attributes. In this work, we propose a TMO (TMOz) that leverages CIECAM16
+perceptual attributes, i.e., brightness, colorfulness, and hue. TMOz accounts
+for the effects of both the surround and the display conditions to achieve more
+optimal colorfulness reproduction. The perceptual brightness is compressed, and
+the perceptual color scales, i.e., colorfulness and hue are derived from HDR
+images by employing CIECAM16 color adaptation equations. A psychophysical
+experiment was conducted to automate the brightness compression parameter. The
+model employs fully automatic and adaptive approach, obviating the requirement
+for manual parameter selection. TMOz was evaluated in terms of contrast,
+colorfulness and overall image quality. The objective and subjective evaluation
+methods revealed that the proposed model outperformed the state-of-the-art
+TMOs.",cs.CV,"['cs.CV', 'eess.IV']"
+VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning,Kang Chen · Xiangqian Wu,https://visual-text-qa.github.io/,,https://dl.acm.org/doi/pdf/10.1145/3581783.3612850,,,,,nan
+Composing Object Relations and Attributes for Image-Text Matching,Khoi Pham · Chuong Huynh · Ser-Nam Lim · Abhinav Shrivastava, ,,https://hmchuong.github.io/,,,,,nan
+HashPoint: Accelerated Point Searching and Sampling for Neural Rendering,Jiahao Ma · Miaomiao Liu · David Ahmedt-Aristizabal · Chuong Nguyen,https://jiahao-ma.github.io/hashpoint/,https://arxiv.org/abs/2404.14044,,2404.14044.pdf,HashPoint: Accelerated Point Searching and Sampling for Neural Rendering,"In this paper, we address the problem of efficient point searching and
+sampling for volume neural rendering. Within this realm, two typical approaches
+are employed: rasterization and ray tracing. The rasterization-based methods
+enable real-time rendering at the cost of increased memory and lower fidelity.
+In contrast, the ray-tracing-based methods yield superior quality but demand
+longer rendering time. We solve this problem by our HashPoint method combining
+these two strategies, leveraging rasterization for efficient point searching
+and sampling, and ray marching for rendering. Our method optimizes point
+searching by rasterizing points within the camera's view, organizing them in a
+hash table, and facilitating rapid searches. Notably, we accelerate the
+rendering process by adaptive sampling on the primary surface encountered by
+the ray. Our approach yields substantial speed-up for a range of
+state-of-the-art ray-tracing-based methods, maintaining equivalent or superior
+accuracy across synthetic and real test datasets. The code will be available at
+https://jiahao-ma.github.io/hashpoint/.",cs.CV,['cs.CV']
+Improving Depth Completion via Depth Feature Upsampling,Yufei Wang · Ge Zhang · Shaoqian Wang · Bo Li · Qi Liu · Le Hui · Yuchao Dai, ,https://arxiv.org/abs/2310.08956,,2310.08956.pdf,LRRU: Long-short Range Recurrent Updating Networks for Depth Completion,"Existing deep learning-based depth completion methods generally employ
+massive stacked layers to predict the dense depth map from sparse input data.
+Although such approaches greatly advance this task, their accompanied huge
+computational complexity hinders their practical applications. To accomplish
+depth completion more efficiently, we propose a novel lightweight deep network
+framework, the Long-short Range Recurrent Updating (LRRU) network. Without
+learning complex feature representations, LRRU first roughly fills the sparse
+input to obtain an initial dense depth map, and then iteratively updates it
+through learned spatially-variant kernels. Our iterative update process is
+content-adaptive and highly flexible, where the kernel weights are learned by
+jointly considering the guidance RGB images and the depth map to be updated,
+and large-to-small kernel scopes are dynamically adjusted to capture
+long-to-short range dependencies. Our initial depth map has coarse but complete
+scene depth information, which helps relieve the burden of directly regressing
+the dense depth from sparse ones, while our proposed method can effectively
+refine it to an accurate depth map with less learnable parameters and inference
+time. Experimental results demonstrate that our proposed LRRU variants achieve
+state-of-the-art performance across different parameter regimes. In particular,
+the LRRU-Base model outperforms competing approaches on the NYUv2 dataset, and
+ranks 1st on the KITTI depth completion benchmark at the time of submission.
+Project page: https://npucvr.github.io/LRRU/.",cs.CV,['cs.CV']
+Improving Out-of-Distribution Generalization in Graphs via Hierarchical Semantic Environments,Yinhua Piao · Sangseon Lee · Yijingxiu Lu · Sun Kim,https://github.com/qkrdmsghk/GOODHSE,https://arxiv.org/abs/2403.01773,,2403.01773.pdf,Improving out-of-distribution generalization in graphs via hierarchical semantic environments,"Out-of-distribution (OOD) generalization in the graph domain is challenging
+due to complex distribution shifts and a lack of environmental contexts. Recent
+methods attempt to enhance graph OOD generalization by generating flat
+environments. However, such flat environments come with inherent limitations to
+capture more complex data distributions. Considering the DrugOOD dataset, which
+contains diverse training environments (e.g., scaffold, size, etc.), flat
+contexts cannot sufficiently address its high heterogeneity. Thus, a new
+challenge is posed to generate more semantically enriched environments to
+enhance graph invariant learning for handling distribution shifts. In this
+paper, we propose a novel approach to generate hierarchical semantic
+environments for each graph. Firstly, given an input graph, we explicitly
+extract variant subgraphs from the input graph to generate proxy predictions on
+local environments. Then, stochastic attention mechanisms are employed to
+re-extract the subgraphs for regenerating global environments in a hierarchical
+manner. In addition, we introduce a new learning objective that guides our
+model to learn the diversity of environments within the same hierarchy while
+maintaining consistency across different hierarchies. This approach enables our
+model to consider the relationships between environments and facilitates robust
+graph invariant learning. Extensive experiments on real-world graph data have
+demonstrated the effectiveness of our framework. Particularly, in the
+challenging dataset DrugOOD, our method achieves up to 1.29% and 2.83%
+improvement over the best baselines on IC50 and EC50 prediction tasks,
+respectively.",cs.LG,"['cs.LG', 'cs.AI']"
+Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval,Haochen Han · Qinghua Zheng · Guang Dai · Minnan Luo · Jingdong Wang,https://github.com/hhc1997/L2RM,https://arxiv.org/abs/2403.05105,,2403.05105.pdf,Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval,"Collecting well-matched multimedia datasets is crucial for training
+cross-modal retrieval models. However, in real-world scenarios, massive
+multimodal data are harvested from the Internet, which inevitably contains
+Partially Mismatched Pairs (PMPs). Undoubtedly, such semantical irrelevant data
+will remarkably harm the cross-modal retrieval performance. Previous efforts
+tend to mitigate this problem by estimating a soft correspondence to
+down-weight the contribution of PMPs. In this paper, we aim to address this
+challenge from a new perspective: the potential semantic similarity among
+unpaired samples makes it possible to excavate useful knowledge from mismatched
+pairs. To achieve this, we propose L2RM, a general framework based on Optimal
+Transport (OT) that learns to rematch mismatched pairs. In detail, L2RM aims to
+generate refined alignments by seeking a minimal-cost transport plan across
+different modalities. To formalize the rematching idea in OT, first, we propose
+a self-supervised cost function that automatically learns from explicit
+similarity-cost mapping relation. Second, we present to model a partial OT
+problem while restricting the transport among false positives to further boost
+refined alignments. Extensive experiments on three benchmarks demonstrate our
+L2RM significantly improves the robustness against PMPs for existing models.
+The code is available at https://github.com/hhc1997/L2RM.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM']"
+Spatio-Temporal Turbulence Mitigation: A Translational Perspective,Xingguang Zhang · Nicholas M Chimitt · Yiheng Chi · Zhiyuan Mao · Stanley H. Chan, ,https://arxiv.org/abs/2401.04244,,2401.04244.pdf,Spatio-Temporal Turbulence Mitigation: A Translational Perspective,"Recovering images distorted by atmospheric turbulence is a challenging
+inverse problem due to the stochastic nature of turbulence. Although numerous
+turbulence mitigation (TM) algorithms have been proposed, their efficiency and
+generalization to real-world dynamic scenarios remain severely limited.
+Building upon the intuitions of classical TM algorithms, we present the Deep
+Atmospheric TUrbulence Mitigation network (DATUM). DATUM aims to overcome major
+challenges when transitioning from classical to deep learning approaches. By
+carefully integrating the merits of classical multi-frame TM methods into a
+deep network structure, we demonstrate that DATUM can efficiently perform
+long-range temporal aggregation using a recurrent fashion, while deformable
+attention and temporal-channel attention seamlessly facilitate pixel
+registration and lucky imaging. With additional supervision, tilt and blur
+degradation can be jointly mitigated. These inductive biases empower DATUM to
+significantly outperform existing methods while delivering a tenfold increase
+in processing speed. A large-scale training dataset, ATSyn, is presented as a
+co-invention to enable generalization in real turbulence. Our code and datasets
+are available at https://xg416.github.io/DATUM.",eess.IV,"['eess.IV', 'cs.CV']"
+Seamless Human Motion Composition with Blended Positional Encodings,German Barquero · Sergio Escalera · Cristina Palmero,https://barquerogerman.github.io/FlowMDM/,https://arxiv.org/abs/2402.15509,,2402.15509.pdf,Seamless Human Motion Composition with Blended Positional Encodings,"Conditional human motion generation is an important topic with many
+applications in virtual reality, gaming, and robotics. While prior works have
+focused on generating motion guided by text, music, or scenes, these typically
+result in isolated motions confined to short durations. Instead, we address the
+generation of long, continuous sequences guided by a series of varying textual
+descriptions. In this context, we introduce FlowMDM, the first diffusion-based
+model that generates seamless Human Motion Compositions (HMC) without any
+postprocessing or redundant denoising steps. For this, we introduce the Blended
+Positional Encodings, a technique that leverages both absolute and relative
+positional encodings in the denoising chain. More specifically, global motion
+coherence is recovered at the absolute stage, whereas smooth and realistic
+transitions are built at the relative stage. As a result, we achieve
+state-of-the-art results in terms of accuracy, realism, and smoothness on the
+Babel and HumanML3D datasets. FlowMDM excels when trained with only a single
+description per motion sequence thanks to its Pose-Centric Cross-ATtention,
+which makes it robust against varying text descriptions at inference time.
+Finally, to address the limitations of existing HMC metrics, we propose two new
+metrics: the Peak Jerk and the Area Under the Jerk, to detect abrupt
+transitions.",cs.CV,['cs.CV']
+CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation,Lingjun Zhao · Jingyu Song · Katherine Skinner,https://song-jingyu.github.io/CRKD,https://arxiv.org/abs/2403.19104,,2403.19104.pdf,CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation,"In the field of 3D object detection for autonomous driving, LiDAR-Camera (LC)
+fusion is the top-performing sensor configuration. Still, LiDAR is relatively
+high cost, which hinders adoption of this technology for consumer automobiles.
+Alternatively, camera and radar are commonly deployed on vehicles already on
+the road today, but performance of Camera-Radar (CR) fusion falls behind LC
+fusion. In this work, we propose Camera-Radar Knowledge Distillation (CRKD) to
+bridge the performance gap between LC and CR detectors with a novel
+cross-modality KD framework. We use the Bird's-Eye-View (BEV) representation as
+the shared feature space to enable effective knowledge distillation. To
+accommodate the unique cross-modality KD path, we propose four distillation
+losses to help the student learn crucial features from the teacher model. We
+present extensive evaluations on the nuScenes dataset to demonstrate the
+effectiveness of the proposed CRKD framework. The project page for CRKD is
+https://song-jingyu.github.io/CRKD.",cs.CV,"['cs.CV', 'cs.RO']"
+RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception,Ruiyang Hao · Siqi Fan · Yingru Dai · Zhenlin Zhang · Chenxi Li · YuntianWang · Haibao Yu · Wenxian Yang · Jirui Yuan · Zaiqing Nie,https://github.com/AIR-THU/DAIR-RCooper,https://arxiv.org/abs/2403.10145,,2403.10145.pdf,RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception,"The value of roadside perception, which could extend the boundaries of
+autonomous driving and traffic management, has gradually become more prominent
+and acknowledged in recent years. However, existing roadside perception
+approaches only focus on the single-infrastructure sensor system, which cannot
+realize a comprehensive understanding of a traffic area because of the limited
+sensing range and blind spots. Orienting high-quality roadside perception, we
+need Roadside Cooperative Perception (RCooper) to achieve practical
+area-coverage roadside perception for restricted traffic areas. Rcooper has its
+own domain-specific challenges, but further exploration is hindered due to the
+lack of datasets. We hence release the first real-world, large-scale RCooper
+dataset to bloom the research on practical roadside cooperative perception,
+including detection and tracking. The manually annotated dataset comprises 50k
+images and 30k point clouds, including two representative traffic scenes (i.e.,
+intersection and corridor). The constructed benchmarks prove the effectiveness
+of roadside cooperation perception and demonstrate the direction of further
+research. Codes and dataset can be accessed at:
+https://github.com/AIR-THU/DAIR-RCooper.",cs.CV,"['cs.CV', 'cs.RO', 'I.4.8; I.5.4']"
+Scene Adaptive Sparse Transformer for Event-based Object Detection,Yansong Peng · Li Hebei · Yueyi Zhang · Xiaoyan Sun · Feng Wu, ,https://arxiv.org/abs/2404.01882,,2404.01882.pdf,Scene Adaptive Sparse Transformer for Event-based Object Detection,"While recent Transformer-based approaches have shown impressive performances
+on event-based object detection tasks, their high computational costs still
+diminish the low power consumption advantage of event cameras. Image-based
+works attempt to reduce these costs by introducing sparse Transformers.
+However, they display inadequate sparsity and adaptability when applied to
+event-based object detection, since these approaches cannot balance the fine
+granularity of token-level sparsification and the efficiency of window-based
+Transformers, leading to reduced performance and efficiency. Furthermore, they
+lack scene-specific sparsity optimization, resulting in information loss and a
+lower recall rate. To overcome these limitations, we propose the Scene Adaptive
+Sparse Transformer (SAST). SAST enables window-token co-sparsification,
+significantly enhancing fault tolerance and reducing computational overhead.
+Leveraging the innovative scoring and selection modules, along with the Masked
+Sparse Window Self-Attention, SAST showcases remarkable scene-aware
+adaptability: It focuses only on important objects and dynamically optimizes
+sparsity level according to scene complexity, maintaining a remarkable balance
+between performance and computational cost. The evaluation results show that
+SAST outperforms all other dense and sparse networks in both performance and
+efficiency on two large-scale event-based object detection datasets (1Mpx and
+Gen1). Code: https://github.com/Peterande/SAST",cs.CV,['cs.CV']
+Local-consistent Transformation Learning for Rotation-invariant Point Cloud Analysis,Yiyang Chen · Lunhao Duan · Shanshan Zhao · Changxing Ding · Dacheng Tao,https://github.com/wdttt/LocoTrans,https://arxiv.org/abs/2403.11113,,2403.11113.pdf,Local-consistent Transformation Learning for Rotation-invariant Point Cloud Analysis,"Rotation invariance is an important requirement for point shape analysis. To
+achieve this, current state-of-the-art methods attempt to construct the local
+rotation-invariant representation through learning or defining the local
+reference frame (LRF). Although efficient, these LRF-based methods suffer from
+perturbation of local geometric relations, resulting in suboptimal local
+rotation invariance. To alleviate this issue, we propose a Local-consistent
+Transformation (LocoTrans) learning strategy. Specifically, we first construct
+the local-consistent reference frame (LCRF) by considering the symmetry of the
+two axes in LRF. In comparison with previous LRFs, our LCRF is able to preserve
+local geometric relationships better through performing local-consistent
+transformation. However, as the consistency only exists in local regions, the
+relative pose information is still lost in the intermediate layers of the
+network. We mitigate such a relative pose issue by developing a relative pose
+recovery (RPR) module. RPR aims to restore the relative pose between adjacent
+transformed patches. Equipped with LCRF and RPR, our LocoTrans is capable of
+learning local-consistent transformation and preserving local geometry, which
+benefits rotation invariance learning. Competitive performance under arbitrary
+rotations on both shape classification and part segmentation tasks and
+ablations can demonstrate the effectiveness of our method. Code will be
+available publicly at https://github.com/wdttt/LocoTrans.",cs.CV,['cs.CV']
+Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learning,Yun Li · Zhe Liu · Hang Chen · Lina Yao, ,https://arxiv.org/abs/2402.17251,,2402.17251.pdf,Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learning,"Compositional Zero-Shot Learning (CZSL) aims to recognize unseen
+attribute-object pairs based on a limited set of observed examples. Current
+CZSL methodologies, despite their advancements, tend to neglect the distinct
+specificity levels present in attributes. For instance, given images of sliced
+strawberries, they may fail to prioritize `Sliced-Strawberry' over a generic
+`Red-Strawberry', despite the former being more informative. They also suffer
+from ballooning search space when shifting from Close-World (CW) to Open-World
+(OW) CZSL. To address the issues, we introduce the Context-based and
+Diversity-driven Specificity learning framework for CZSL (CDS-CZSL). Our
+framework evaluates the specificity of attributes by considering the diversity
+of objects they apply to and their related context. This novel approach allows
+for more accurate predictions by emphasizing specific attribute-object pairs
+and improves composition filtering in OW-CZSL. We conduct experiments in both
+CW and OW scenarios, and our model achieves state-of-the-art results across
+three datasets.",cs.CV,['cs.CV']
+MemFlow: Optical Flow Estimation and Prediction with Memory,Qiaole Dong · Yanwei Fu,https://dqiaole.github.io/MemFlow/,https://arxiv.org/abs/2404.04808,,2404.04808.pdf,MemFlow: Optical Flow Estimation and Prediction with Memory,"Optical flow is a classical task that is important to the vision community.
+Classical optical flow estimation uses two frames as input, whilst some recent
+methods consider multiple frames to explicitly model long-range information.
+The former ones limit their ability to fully leverage temporal coherence along
+the video sequence; and the latter ones incur heavy computational overhead,
+typically not possible for real-time flow estimation. Some multi-frame-based
+approaches even necessitate unseen future frames for current estimation,
+compromising real-time applicability in safety-critical scenarios. To this end,
+we present MemFlow, a real-time method for optical flow estimation and
+prediction with memory. Our method enables memory read-out and update modules
+for aggregating historical motion information in real-time. Furthermore, we
+integrate resolution-adaptive re-scaling to accommodate diverse video
+resolutions. Besides, our approach seamlessly extends to the future prediction
+of optical flow based on past observations. Leveraging effective historical
+motion aggregation, our method outperforms VideoFlow with fewer parameters and
+faster inference speed on Sintel and KITTI-15 datasets in terms of
+generalization performance. At the time of submission, MemFlow also leads in
+performance on the 1080p Spring dataset. Codes and models will be available at:
+https://dqiaole.github.io/MemFlow/.",cs.CV,['cs.CV']
+H-ViT: A Hierarchical Vision Transformer for Deformable Image Registration,Morteza Ghahremani · Mohammad Khateri · Bailiang Jian · Benedikt Wiestler · Ehsan Adeli · Christian Wachinger, ,https://arxiv.org/abs/2306.05688,,2306.05688.pdf,ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer,"The Transformer structures have been widely used in computer vision and have
+recently made an impact in the area of medical image registration. However, the
+use of Transformer in most registration networks is straightforward. These
+networks often merely use the attention mechanism to boost the feature learning
+as the segmentation networks do, but do not sufficiently design to be adapted
+for the registration task. In this paper, we propose a novel motion
+decomposition Transformer (ModeT) to explicitly model multiple motion
+modalities by fully exploiting the intrinsic capability of the Transformer
+structure for deformation estimation. The proposed ModeT naturally transforms
+the multi-head neighborhood attention relationship into the multi-coordinate
+relationship to model multiple motion modes. Then the competitive weighting
+module (CWM) fuses multiple deformation sub-fields to generate the resulting
+deformation field. Extensive experiments on two public brain magnetic resonance
+imaging (MRI) datasets show that our method outperforms current
+state-of-the-art registration networks and Transformers, demonstrating the
+potential of our ModeT for the challenging non-rigid deformation estimation
+problem. The benchmarks and our code are publicly available at
+https://github.com/ZAX130/SmileCode.",cs.CV,['cs.CV']
+Entangled View-Epipolar Information Aggregation for Generalizable Neural Radiance Fields,Zhiyuan Min · Yawei Luo · Wei Yang · Yuesong Wang · Yi Yang,https://github.com/tatakai1/EVENeRF,https://arxiv.org/abs/2311.11845,,2311.11845.pdf,Entangled View-Epipolar Information Aggregation for Generalizable Neural Radiance Fields,"Generalizable NeRF can directly synthesize novel views across new scenes,
+eliminating the need for scene-specific retraining in vanilla NeRF. A critical
+enabling factor in these approaches is the extraction of a generalizable 3D
+representation by aggregating source-view features. In this paper, we propose
+an Entangled View-Epipolar Information Aggregation method dubbed EVE-NeRF.
+Different from existing methods that consider cross-view and along-epipolar
+information independently, EVE-NeRF conducts the view-epipolar feature
+aggregation in an entangled manner by injecting the scene-invariant appearance
+continuity and geometry consistency priors to the aggregation process. Our
+approach effectively mitigates the potential lack of inherent geometric and
+appearance constraint resulting from one-dimensional interactions, thus further
+boosting the 3D representation generalizablity. EVE-NeRF attains
+state-of-the-art performance across various evaluation scenarios. Extensive
+experiments demonstate that, compared to prevailing single-dimensional
+aggregation, the entangled network excels in the accuracy of 3D scene geometry
+and appearance reconstruction. Our code is publicly available at
+https://github.com/tatakai1/EVENeRF.",cs.CV,['cs.CV']
+Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective,Jinjing Zhao · Fangyun Wei · Chang Xu,https://github.com/ZhaoJingjing713/HPR,,,,,,,nan
+Hyperbolic Anomaly Detection,Huimin Li · Zhentao Chen · Yunhao Xu · Junlin Hu, ,https://arxiv.org/abs/2403.20236,,2403.20236.pdf,Long-Tailed Anomaly Detection with Learnable Class Names,"Anomaly detection (AD) aims to identify defective images and localize their
+defects (if any). Ideally, AD models should be able to detect defects over many
+image classes; without relying on hard-coded class names that can be
+uninformative or inconsistent across datasets; learn without anomaly
+supervision; and be robust to the long-tailed distributions of real-world
+applications. To address these challenges, we formulate the problem of
+long-tailed AD by introducing several datasets with different levels of class
+imbalance and metrics for performance evaluation. We then propose a novel
+method, LTAD, to detect defects from multiple and long-tailed classes, without
+relying on dataset class names. LTAD combines AD by reconstruction and semantic
+AD modules. AD by reconstruction is implemented with a transformer-based
+reconstruction module. Semantic AD is implemented with a binary classifier,
+which relies on learned pseudo class names and a pretrained foundation model.
+These modules are learned over two phases. Phase 1 learns the pseudo-class
+names and a variational autoencoder (VAE) for feature synthesis that augments
+the training data to combat long-tails. Phase 2 then learns the parameters of
+the reconstruction and classification modules of LTAD. Extensive experiments
+using the proposed long-tailed datasets show that LTAD substantially
+outperforms the state-of-the-art methods for most forms of dataset imbalance.
+The long-tailed dataset split is available at
+https://zenodo.org/records/10854201 .",cs.CV,['cs.CV']
+HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models,Tianrui Guan · Fuxiao Liu · Xiyang Wu · Ruiqi Xian · Zongxia Li · Xiaoyu Liu · Xijun Wang · Lichang Chen · Furong Huang · Yaser Yacoob · Dinesh Manocha · Tianyi Zhou,https://github.com/tianyi-lab/HallusionBench,https://arxiv.org/abs/2310.14566,,2310.14566.pdf,HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models,"We introduce HallusionBench, a comprehensive benchmark designed for the
+evaluation of image-context reasoning. This benchmark presents significant
+challenges to advanced large visual-language models (LVLMs), such as
+GPT-4V(Vision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, by emphasizing
+nuanced understanding and interpretation of visual data. The benchmark
+comprises 346 images paired with 1129 questions, all meticulously crafted by
+human experts. We introduce a novel structure for these visual questions
+designed to establish control groups. This structure enables us to conduct a
+quantitative analysis of the models' response tendencies, logical consistency,
+and various failure modes. In our evaluation on HallusionBench, we benchmarked
+15 different models, highlighting a 31.42% question-pair accuracy achieved by
+the state-of-the-art GPT-4V. Notably, all other evaluated models achieve
+accuracy below 16%. Moreover, our analysis not only highlights the observed
+failure modes, including language hallucination and visual illusion, but also
+deepens an understanding of these pitfalls. Our comprehensive case studies
+within HallusionBench shed light on the challenges of hallucination and
+illusion in LVLMs. Based on these insights, we suggest potential pathways for
+their future improvement. The benchmark and codebase can be accessed at
+https://github.com/tianyi-lab/HallusionBench.",cs.CV,"['cs.CV', 'cs.CL']"
+Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization,Ioanna Ntinou · Enrique Sanchez · Georgios Tzimiropoulos,https://github.com/IoannaNti/BMViT,https://arxiv.org/abs/2312.17686,,2312.17686.pdf,Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization,"Action Localization is a challenging problem that combines detection and
+recognition tasks, which are often addressed separately. State-of-the-art
+methods rely on off-the-shelf bounding box detections pre-computed at high
+resolution, and propose transformer models that focus on the classification
+task alone. Such two-stage solutions are prohibitive for real-time deployment.
+On the other hand, single-stage methods target both tasks by devoting part of
+the network (generally the backbone) to sharing the majority of the workload,
+compromising performance for speed. These methods build on adding a DETR head
+with learnable queries that after cross- and self-attention can be sent to
+corresponding MLPs for detecting a person's bounding box and action. However,
+DETR-like architectures are challenging to train and can incur in big
+complexity.
+  In this paper, we observe that \textbf{a straight bipartite matching loss can
+be applied to the output tokens of a vision transformer}. This results in a
+backbone + MLP architecture that can do both tasks without the need of an extra
+encoder-decoder head and learnable queries. We show that a single MViTv2-S
+architecture trained with bipartite matching to perform both tasks surpasses
+the same MViTv2-S when trained with RoI align on pre-computed bounding boxes.
+With a careful design of token pooling and the proposed training pipeline, our
+Bipartite-Matching Vision Transformer model, \textbf{BMViT}, achieves +3 mAP on
+AVA2.2. w.r.t. the two-stage MViTv2-S counterpart. Code is available at
+\href{https://github.com/IoannaNti/BMViT}{https://github.com/IoannaNti/BMViT}",cs.CV,['cs.CV']
+UnO: Unsupervised Occupancy Fields for Perception and Forecasting,Ben Agro · Quinlan Sykora · Sergio Casas · Thomas Gilles · Raquel Urtasun, ,https://arxiv.org/abs/2308.01471,,2308.01471.pdf,Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving,"A self-driving vehicle (SDV) must be able to perceive its surroundings and
+predict the future behavior of other traffic participants. Existing works
+either perform object detection followed by trajectory forecasting of the
+detected objects, or predict dense occupancy and flow grids for the whole
+scene. The former poses a safety concern as the number of detections needs to
+be kept low for efficiency reasons, sacrificing object recall. The latter is
+computationally expensive due to the high-dimensionality of the output grid,
+and suffers from the limited receptive field inherent to fully convolutional
+networks. Furthermore, both approaches employ many computational resources
+predicting areas or objects that might never be queried by the motion planner.
+This motivates our unified approach to perception and future prediction that
+implicitly represents occupancy and flow over time with a single neural
+network. Our method avoids unnecessary computation, as it can be directly
+queried by the motion planner at continuous spatio-temporal locations.
+Moreover, we design an architecture that overcomes the limited receptive field
+of previous explicit occupancy prediction methods by adding an efficient yet
+effective global attention mechanism. Through extensive experiments in both
+urban and highway settings, we demonstrate that our implicit model outperforms
+the current state-of-the-art. For more information, visit the project website:
+https://waabi.ai/research/implicito.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+Do Vision and Language Encoders Represent the World Similarly?,Mayug Maniparambil · Raiymbek Akshulakov · YASSER ABDELAZIZ DAHOU DJILALI · Mohamed El Amine Seddik · Sanath Narayan · Karttikeya Mangalam · Noel O&#x27;Connor,https://github.com/mayug/0-shot-llm-vision,https://arxiv.org/abs/2401.05224,,2401.05224.pdf,Do Vision and Language Encoders Represent the World Similarly?,"Aligned text-image encoders such as CLIP have become the de facto model for
+vision-language tasks. Furthermore, modality-specific encoders achieve
+impressive performances in their respective domains. This raises a central
+question: does an alignment exist between uni-modal vision and language
+encoders since they fundamentally represent the same physical world? Analyzing
+the latent spaces structure of vision and language models on image-caption
+benchmarks using the Centered Kernel Alignment (CKA), we find that the
+representation spaces of unaligned and aligned encoders are semantically
+similar. In the absence of statistical similarity in aligned encoders like
+CLIP, we show that a possible matching of unaligned encoders exists without any
+training. We frame this as a seeded graph-matching problem exploiting the
+semantic similarity between graphs and propose two methods - a Fast Quadratic
+Assignment Problem optimization, and a novel localized CKA metric-based
+matching/retrieval. We demonstrate the effectiveness of this on several
+downstream tasks including cross-lingual, cross-domain caption matching and
+image classification. Code available at github.com/mayug/0-shot-llm-vision.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+You Only Need Less Attention Each Stage in Vision Transformers,Shuoxi Zhang · Hanpeng Liu · Stephen Lin · Kun He, ,,,,,,,nan
+DemoCaricature: Democratising Caricature Generation with a Rough Sketch,Dar-Yen Chen · Ayan Kumar Bhunia · Subhadeep Koley · Aneeshan Sain · Pinaki Nath Chowdhury · Yi-Zhe Song,https://democaricature.github.io/,https://arxiv.org/abs/2312.04364v1,,2312.04364v1.pdf,DemoCaricature: Democratising Caricature Generation with a Rough Sketch,"In this paper, we democratise caricature generation, empowering individuals
+to effortlessly craft personalised caricatures with just a photo and a
+conceptual sketch. Our objective is to strike a delicate balance between
+abstraction and identity, while preserving the creativity and subjectivity
+inherent in a sketch. To achieve this, we present Explicit Rank-1 Model Editing
+alongside single-image personalisation, selectively applying nuanced edits to
+cross-attention layers for a seamless merge of identity and style.
+Additionally, we propose Random Mask Reconstruction to enhance robustness,
+directing the model to focus on distinctive identity and style features.
+Crucially, our aim is not to replace artists but to eliminate accessibility
+barriers, allowing enthusiasts to engage in the artistry.",cs.CV,['cs.CV']
+Prompt Highlighter: Interactive Control for Multi-Modal LLMs,Yuechen Zhang · Shengju Qian · Bohao Peng · Shu Liu · Jiaya Jia,https://github.com/dvlab-research/Prompt-Highlighter,https://arxiv.org/abs/2312.04302,,2312.04302.pdf,Prompt Highlighter: Interactive Control for Multi-Modal LLMs,"This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs)
+inference: explicit controllable text generation. Multi-modal LLMs empower
+multi-modality understanding with the capability of semantic generation yet
+bring less explainability and heavier reliance on prompt contents due to their
+autoregressive generative nature. While manipulating prompt formats could
+improve outputs, designing specific and precise prompts per task can be
+challenging and ineffective. To tackle this issue, we introduce a novel
+inference method, Prompt Highlighter, which enables users to highlight specific
+prompt spans to interactively control the focus during generation. Motivated by
+the classifier-free diffusion guidance, we form regular and unconditional
+context pairs based on highlighted tokens, demonstrating that the
+autoregressive generation in models can be guided in a classifier-free way.
+Notably, we find that, during inference, guiding the models with highlighted
+tokens through the attention weights leads to more desired outputs. Our
+approach is compatible with current LLMs and VLMs, achieving impressive
+customized generation results without training. Experiments confirm its
+effectiveness in focusing on input contexts and generating reliable content.
+Without tuning on LLaVA-v1.5, our method secured 70.7 in the MMBench test and
+1552.5 in MME-perception. The code is available at:
+https://github.com/dvlab-research/Prompt-Highlighter/",cs.CV,"['cs.CV', 'cs.CL']"
+Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting,Haiwei Chen · Yajie Zhao, ,https://arxiv.org/abs/2403.18186,,2403.18186.pdf,Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting,"We present a method for large-mask pluralistic image inpainting based on the
+generative framework of discrete latent codes. Our method learns latent priors,
+discretized as tokens, by only performing computations at the visible locations
+of the image. This is realized by a restrictive partial encoder that predicts
+the token label for each visible block, a bidirectional transformer that infers
+the missing labels by only looking at these tokens, and a dedicated synthesis
+network that couples the tokens with the partial image priors to generate
+coherent and pluralistic complete image even under extreme mask settings.
+Experiments on public benchmarks validate our design choices as the proposed
+method outperforms strong baselines in both visual quality and diversity
+metrics.",cs.CV,['cs.CV']
+Bayesian Differentiable Physics for Cloth Digitalization,Deshan Gong · Ningtao Mao · He Wang, ,https://arxiv.org/abs/2402.17664,,2402.17664.pdf,Bayesian Differentiable Physics for Cloth Digitalization,"We propose a new method for cloth digitalization. Deviating from existing
+methods which learn from data captured under relatively casual settings, we
+propose to learn from data captured in strictly tested measuring protocols, and
+find plausible physical parameters of the cloths. However, such data is
+currently absent, so we first propose a new dataset with accurate cloth
+measurements. Further, the data size is considerably smaller than the ones in
+current deep learning, due to the nature of the data capture process. To learn
+from small data, we propose a new Bayesian differentiable cloth model to
+estimate the complex material heterogeneity of real cloths. It can provide
+highly accurate digitalization from very limited data samples. Through
+exhaustive evaluation and comparison, we show our method is accurate in cloth
+digitalization, efficient in learning from limited data samples, and general in
+capturing material variations. Code and data are available
+https://github.com/realcrane/Bayesian-Differentiable-Physics-for-Cloth-Digitalization",cs.CV,"['cs.CV', 'F.4.8; I.6.8']"
+Few-Shot Object Detection with Foundation Models,Guangxing Han · Ser-Nam Lim, ,https://arxiv.org/abs/2312.14494,,2312.14494.pdf,Revisiting Few-Shot Object Detection with Vision-Language Models,"Few-shot object detection (FSOD) benchmarks have advanced techniques for
+detecting new categories with limited annotations. Existing benchmarks
+repurpose well-established datasets like COCO by partitioning categories into
+base and novel classes for pre-training and fine-tuning respectively. However,
+these benchmarks do not reflect how FSOD is deployed in practice. Rather than
+only pre-training on a small number of base categories, we argue that it is
+more practical to fine-tune a foundation model (e.g., a vision-language model
+(VLM) pre-trained on web-scale data) for a target domain. Surprisingly, we find
+that zero-shot inference from VLMs like GroundingDINO significantly outperforms
+the state-of-the-art (48.3 vs. 33.1 AP) on COCO. However, such zero-shot models
+can still be misaligned to target concepts of interest. For example, trailers
+on the web may be different from trailers in the context of autonomous
+vehicles. In this work, we propose Foundational FSOD, a new benchmark protocol
+that evaluates detectors pre-trained on any external datasets and fine-tuned on
+K-shots per target class. Further, we note that current FSOD benchmarks are
+actually federated datasets containing exhaustive annotations for each category
+on a subset of the data. We leverage this insight to propose simple strategies
+for fine-tuning VLMs with federated losses. We demonstrate the effectiveness of
+our approach on LVIS and nuImages, improving over prior work by 5.9 AP. Our
+code is available at https://github.com/anishmadan23/foundational_fsod",cs.CV,['cs.CV']
+MonoHair: High-Fidelity Hair Modeling from a Monocular Video,Keyu Wu · LINGCHEN YANG · Zhiyi Kuang · Yao Feng · Xutao Han · Yuefan Shen · Hongbo Fu · Kun Zhou · Youyi Zheng,https://keyuwu-cs.github.io/MonoHair/,https://arxiv.org/abs/2403.18356,,2403.18356.pdf,MonoHair: High-Fidelity Hair Modeling from a Monocular Video,"Undoubtedly, high-fidelity 3D hair is crucial for achieving realism, artistic
+expression, and immersion in computer graphics. While existing 3D hair modeling
+methods have achieved impressive performance, the challenge of achieving
+high-quality hair reconstruction persists: they either require strict capture
+conditions, making practical applications difficult, or heavily rely on learned
+prior data, obscuring fine-grained details in images. To address these
+challenges, we propose MonoHair,a generic framework to achieve high-fidelity
+hair reconstruction from a monocular video, without specific requirements for
+environments. Our approach bifurcates the hair modeling process into two main
+stages: precise exterior reconstruction and interior structure inference. The
+exterior is meticulously crafted using our Patch-based Multi-View Optimization
+(PMVO). This method strategically collects and integrates hair information from
+multiple views, independent of prior data, to produce a high-fidelity exterior
+3D line map. This map not only captures intricate details but also facilitates
+the inference of the hair's inner structure. For the interior, we employ a
+data-driven, multi-view 3D hair reconstruction method. This method utilizes 2D
+structural renderings derived from the reconstructed exterior, mirroring the
+synthetic 2D inputs used during training. This alignment effectively bridges
+the domain gap between our training data and real-world data, thereby enhancing
+the accuracy and reliability of our interior structure inference. Lastly, we
+generate a strand model and resolve the directional ambiguity by our hair
+growth algorithm. Our experiments demonstrate that our method exhibits
+robustness across diverse hairstyles and achieves state-of-the-art performance.
+For more results, please refer to our project page
+https://keyuwu-cs.github.io/MonoHair/.",cs.CV,['cs.CV']
+Solving Masked Jigsaw Puzzles with Diffusion Transformers,Jinyang Liu · Wondmgezahu Teshome · Sandesh Ghimire · Mario Sznaier · Octavia Camps, ,https://arxiv.org/abs/2404.07292,,2404.07292.pdf,Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers,"Solving image and video jigsaw puzzles poses the challenging task of
+rearranging image fragments or video frames from unordered sequences to restore
+meaningful images and video sequences. Existing approaches often hinge on
+discriminative models tasked with predicting either the absolute positions of
+puzzle elements or the permutation actions applied to the original data.
+Unfortunately, these methods face limitations in effectively solving puzzles
+with a large number of elements. In this paper, we propose JPDVT, an innovative
+approach that harnesses diffusion transformers to address this challenge.
+Specifically, we generate positional information for image patches or video
+frames, conditioned on their underlying visual content. This information is
+then employed to accurately assemble the puzzle pieces in their correct
+positions, even in scenarios involving missing pieces. Our method achieves
+state-of-the-art performance on several datasets.",cs.CV,['cs.CV']
+Shadow-Enlightened Image Outpainting,Hang Yu · Ruilin Li · Shaorong Xie · Jiayan Qiu, ,https://arxiv.org/html/2204.08563v2,,2204.08563v2.pdf,Cylin-Painting: Seamless {360\textdegree} Panoramic Image Outpainting and Beyond,"Image outpainting gains increasing attention since it can generate the
+complete scene from a partial view, providing a valuable solution to construct
+{360\textdegree} panoramic images. As image outpainting suffers from the
+intrinsic issue of unidirectional completion flow, previous methods convert the
+original problem into inpainting, which allows a bidirectional flow. However,
+we find that inpainting has its own limitations and is inferior to outpainting
+in certain situations. The question of how they may be combined for the best of
+both has as yet remained under-explored. In this paper, we provide a deep
+analysis of the differences between inpainting and outpainting, which
+essentially depends on how the source pixels contribute to the unknown regions
+under different spatial arrangements. Motivated by this analysis, we present a
+Cylin-Painting framework that involves meaningful collaborations between
+inpainting and outpainting and efficiently fuses the different arrangements,
+with a view to leveraging their complementary benefits on a seamless cylinder.
+Nevertheless, straightforwardly applying the cylinder-style convolution often
+generates visually unpleasing results as it discards important positional
+information. To address this issue, we further present a learnable positional
+embedding strategy to incorporate the missing component of positional encoding
+into the cylinder convolution, which significantly improves the panoramic
+results. It is noted that while developed for image outpainting, the proposed
+algorithm can be effectively extended to other panoramic vision tasks, such as
+object detection, depth estimation, and image super-resolution. Code will be
+made available at \url{https://github.com/KangLiao929/Cylin-Painting}.",cs.CV,['cs.CV']
+Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking,Wei Cao · Chang Luo · Biao Zhang · Matthias Nießner · Jiapeng Tang, ,https://arxiv.org/abs/2401.06614,,2401.06614.pdf,Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking,"We introduce Motion2VecSets, a 4D diffusion model for dynamic surface
+reconstruction from point cloud sequences. While existing state-of-the-art
+methods have demonstrated success in reconstructing non-rigid objects using
+neural field representations, conventional feed-forward networks encounter
+challenges with ambiguous observations from noisy, partial, or sparse point
+clouds. To address these challenges, we introduce a diffusion model that
+explicitly learns the shape and motion distribution of non-rigid objects
+through an iterative denoising process of compressed latent representations.
+The diffusion-based priors enable more plausible and probabilistic
+reconstructions when handling ambiguous inputs. We parameterize 4D dynamics
+with latent sets instead of using global latent codes. This novel 4D
+representation allows us to learn local shape and deformation patterns, leading
+to more accurate non-linear motion capture and significantly improving
+generalizability to unseen motions and identities. For more temporally-coherent
+object tracking, we synchronously denoise deformation latent sets and exchange
+information across multiple frames. To avoid computational overhead, we
+designed an interleaved space and time attention block to alternately aggregate
+deformation latents along spatial and temporal domains. Extensive comparisons
+against state-of-the-art methods demonstrate the superiority of our
+Motion2VecSets in 4D reconstruction from various imperfect observations. More
+detailed information can be found at
+https://vveicao.github.io/projects/Motion2VecSets/.",cs.CV,['cs.CV']
+Test-Time Linear Out-of-Distribution Detection,Ke Fan · Tong Liu · Xingyu Qiu · Yikai Wang · Lian Huai · Zeyu Shangguan · Shuang Gou · FENGJIAN LIU · Yuqian Fu · Yanwei Fu · Xingqun Jiang, ,https://arxiv.org/abs/2311.16420,,2311.16420.pdf,Model-free Test Time Adaptation for Out-Of-Distribution Detection,"Out-of-distribution (OOD) detection is essential for the reliability of ML
+models. Most existing methods for OOD detection learn a fixed decision
+criterion from a given in-distribution dataset and apply it universally to
+decide if a data point is OOD. Recent work~\cite{fang2022is} shows that given
+only in-distribution data, it is impossible to reliably detect OOD data without
+extra assumptions. Motivated by the theoretical result and recent exploration
+of test-time adaptation methods, we propose a Non-Parametric Test Time
+\textbf{Ada}ptation framework for \textbf{O}ut-Of-\textbf{D}istribution
+\textbf{D}etection (\abbr). Unlike conventional methods, \abbr utilizes online
+test samples for model adaptation during testing, enhancing adaptability to
+changing data distributions. The framework incorporates detected OOD instances
+into decision-making, reducing false positive rates, particularly when ID and
+OOD distributions overlap significantly. We demonstrate the effectiveness of
+\abbr through comprehensive experiments on multiple OOD detection benchmarks,
+extensive empirical studies show that \abbr significantly improves the
+performance of OOD detection over state-of-the-art methods. Specifically, \abbr
+reduces the false positive rate (FPR95) by $23.23\%$ on the CIFAR-10 benchmarks
+and $38\%$ on the ImageNet-1k benchmarks compared to the advanced methods.
+Lastly, we theoretically verify the effectiveness of \abbr.",cs.LG,"['cs.LG', 'cs.CV']"
+Spatial-Aware Regression for Keypoint Localization,Dongkai Wang · Shiliang Zhang, ,,https://dl.acm.org/doi/10.1145/3581783.3611989,,,,,nan
+Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving,JunDa Cheng · Wei Yin · Kaixuan Wang · Xiaozhi Chen · Shijie Wang · Xin Yang, ,https://arxiv.org/abs/2403.07535,,2403.07535.pdf,Adaptive Fusion of Single-View and Multi-View Depth for Autonomous Driving,"Multi-view depth estimation has achieved impressive performance over various
+benchmarks. However, almost all current multi-view systems rely on given ideal
+camera poses, which are unavailable in many real-world scenarios, such as
+autonomous driving. In this work, we propose a new robustness benchmark to
+evaluate the depth estimation system under various noisy pose settings.
+Surprisingly, we find current multi-view depth estimation methods or
+single-view and multi-view fusion methods will fail when given noisy pose
+settings. To address this challenge, we propose a single-view and multi-view
+fused depth estimation system, which adaptively integrates high-confident
+multi-view and single-view results for both robust and accurate depth
+estimations. The adaptive fusion module performs fusion by dynamically
+selecting high-confidence regions between two branches based on a wrapping
+confidence map. Thus, the system tends to choose the more reliable branch when
+facing textureless scenes, inaccurate calibration, dynamic objects, and other
+degradation or challenging conditions. Our method outperforms state-of-the-art
+multi-view and fusion methods under robustness testing. Furthermore, we achieve
+state-of-the-art performance on challenging benchmarks (KITTI and DDAD) when
+given accurate pose estimations. Project website:
+https://github.com/Junda24/AFNet/.",cs.CV,['cs.CV']
+ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection,Yichen Bai · Zongbo Han · Bing Cao · Xiaoheng Jiang · Qinghua Hu · Changqing Zhang, ,https://arxiv.org/abs/2311.15243,,2311.15243.pdf,ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection,"Out-of-distribution (OOD) detection methods often exploit auxiliary outliers
+to train model identifying OOD samples, especially discovering challenging
+outliers from auxiliary outliers dataset to improve OOD detection. However,
+they may still face limitations in effectively distinguishing between the most
+challenging OOD samples that are much like in-distribution (ID) data, i.e.,
+\idlike samples. To this end, we propose a novel OOD detection framework that
+discovers \idlike outliers using CLIP \cite{DBLP:conf/icml/RadfordKHRGASAM21}
+from the vicinity space of the ID samples, thus helping to identify these most
+challenging OOD samples. Then a prompt learning framework is proposed that
+utilizes the identified \idlike outliers to further leverage the capabilities
+of CLIP for OOD detection. Benefiting from the powerful CLIP, we only need a
+small number of ID samples to learn the prompts of the model without exposing
+other auxiliary outlier datasets. By focusing on the most challenging \idlike
+OOD samples and elegantly exploiting the capabilities of CLIP, our method
+achieves superior few-shot learning performance on various real-world image
+datasets (e.g., in 4-shot OOD detection on the ImageNet-1k dataset, our method
+reduces the average FPR95 by 12.16\% and improves the average AUROC by 2.76\%,
+compared to state-of-the-art methods). Code is available at
+https://github.com/ycfate/ID-like.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis,Feng Liang · Bichen Wu · Jialiang Wang · Licheng Yu · Kunpeng Li · Yinan Zhao · Ishan Misra · Jia-Bin Huang · Peizhao Zhang · Peter Vajda · Diana Marculescu, ,https://arxiv.org/abs/2312.17681,,2312.17681.pdf,FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis,"Diffusion models have transformed the image-to-image (I2I) synthesis and are
+now permeating into videos. However, the advancement of video-to-video (V2V)
+synthesis has been hampered by the challenge of maintaining temporal
+consistency across video frames. This paper proposes a consistent V2V synthesis
+framework by jointly leveraging spatial conditions and temporal optical flow
+clues within the source video. Contrary to prior methods that strictly adhere
+to optical flow, our approach harnesses its benefits while handling the
+imperfection in flow estimation. We encode the optical flow via warping from
+the first frame and serve it as a supplementary reference in the diffusion
+model. This enables our model for video synthesis by editing the first frame
+with any prevalent I2I models and then propagating edits to successive frames.
+Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
+FlowVid works seamlessly with existing I2I models, facilitating various
+modifications, including stylization, object swaps, and local edits. (2)
+Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
+takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
+Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
+FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
+(10.2%), and TokenFlow (40.4%).",cs.CV,"['cs.CV', 'cs.MM']"
+Prompt-enhanced Multiple Instance Learning for Weakly Supervised Anomaly Detection,Junxi Chen · Liang Li · Li Su · Zheng-Jun Zha · Qingming Huang, ,https://arxiv.org/abs/2306.14451,,2306.14451.pdf,Learning Prompt-Enhanced Context Features for Weakly-Supervised Video Anomaly Detection,"Video anomaly detection under weak supervision presents significant
+challenges, particularly due to the lack of frame-level annotations during
+training. While prior research has utilized graph convolution networks and
+self-attention mechanisms alongside multiple instance learning (MIL)-based
+classification loss to model temporal relations and learn discriminative
+features, these methods often employ multi-branch architectures to capture
+local and global dependencies separately, resulting in increased parameters and
+computational costs. Moreover, the coarse-grained interclass separability
+provided by the binary constraint of MIL-based loss neglects the fine-grained
+discriminability within anomalous classes. In response, this paper introduces a
+weakly supervised anomaly detection framework that focuses on efficient context
+modeling and enhanced semantic discriminability. We present a Temporal Context
+Aggregation (TCA) module that captures comprehensive contextual information by
+reusing the similarity matrix and implementing adaptive fusion. Additionally,
+we propose a Prompt-Enhanced Learning (PEL) module that integrates semantic
+priors using knowledge-based prompts to boost the discriminative capacity of
+context features while ensuring separability between anomaly sub-classes.
+Extensive experiments validate the effectiveness of our method's components,
+demonstrating competitive performance with reduced parameters and computational
+effort on three challenging benchmarks: UCF-Crime, XD-Violence, and
+ShanghaiTech datasets. Notably, our approach significantly improves the
+detection accuracy of certain anomaly sub-classes, underscoring its practical
+value and efficacy. Our code is available at:
+https://github.com/yujiangpu20/PEL4VAD.",cs.CV,['cs.CV']
+Efficient Dataset Distillation via Minimax Diffusion,Jianyang Gu · Saeed Vahidian · Vyacheslav Kungurtsev · Haonan Wang · Wei Jiang · Yang You · Yiran Chen,https://github.com/vimar-gu/MinimaxDiffusion,https://arxiv.org/abs/2311.15529v1,,2311.15529v1.pdf,Efficient Dataset Distillation via Minimax Diffusion,"Dataset distillation reduces the storage and computational consumption of
+training a network by generating a small surrogate dataset that encapsulates
+rich information of the original large-scale one. However, previous
+distillation methods heavily rely on the sample-wise iterative optimization
+scheme. As the images-per-class (IPC) setting or image resolution grows larger,
+the necessary computation will demand overwhelming time and resources. In this
+work, we intend to incorporate generative diffusion techniques for computing
+the surrogate dataset. Observing that key factors for constructing an effective
+surrogate dataset are representativeness and diversity, we design additional
+minimax criteria in the generative training to enhance these facets for the
+generated images of diffusion models. We present a theoretical model of the
+process as hierarchical diffusion control demonstrating the flexibility of the
+diffusion process to target these criteria without jeopardizing the
+faithfulness of the sample to the desired distribution. The proposed method
+achieves state-of-the-art validation performance while demanding much less
+computational resources. Under the 100-IPC setting on ImageWoof, our method
+requires less than one-twentieth the distillation time of previous methods, yet
+yields even better performance. Source code available in
+https://github.com/vimar-gu/MinimaxDiffusion.",cs.CV,['cs.CV']
+State Space Models for Event Cameras,Nikola Zubic · Mathias Gehrig · Davide Scaramuzza,https://github.com/uzh-rpg/ssms_event_cameras,https://arxiv.org/abs/2402.15584,,2402.15584.pdf,State Space Models for Event Cameras,"Today, state-of-the-art deep neural networks that process event-camera data
+first convert a temporal window of events into dense, grid-like input
+representations. As such, they exhibit poor generalizability when deployed at
+higher inference frequencies (i.e., smaller temporal windows) than the ones
+they were trained on. We address this challenge by introducing state-space
+models (SSMs) with learnable timescale parameters to event-based vision. This
+design adapts to varying frequencies without the need to retrain the network at
+different frequencies. Additionally, we investigate two strategies to
+counteract aliasing effects when deploying the model at higher frequencies. We
+comprehensively evaluate our approach against existing methods based on RNN and
+Transformer architectures across various benchmarks, including Gen1 and 1 Mpx
+event camera datasets. Our results demonstrate that SSM-based models train 33%
+faster and also exhibit minimal performance degradation when tested at higher
+frequencies than the training input. Traditional RNN and Transformer models
+exhibit performance drops of more than 20 mAP, with SSMs having a drop of 3.76
+mAP, highlighting the effectiveness of SSMs in event-based vision tasks.",cs.CV,"['cs.CV', 'cs.LG']"
+ReCoRe: Regularized Contrastive Representation Learning of World Model,"Rudra P,K. Poudel · Harit Pandya · Stephan Liwicki · Roberto Cipolla",https://www.toshiba.eu/pages/eu/Cambridge-Research-Laboratory/world_models,https://arxiv.org/abs/2312.09056v1,,2312.09056v1.pdf,ReCoRe: Regularized Contrastive Representation Learning of World Model,"While recent model-free Reinforcement Learning (RL) methods have demonstrated
+human-level effectiveness in gaming environments, their success in everyday
+tasks like visual navigation has been limited, particularly under significant
+appearance variations. This limitation arises from (i) poor sample efficiency
+and (ii) over-fitting to training scenarios. To address these challenges, we
+present a world model that learns invariant features using (i) contrastive
+unsupervised learning and (ii) an intervention-invariant regularizer. Learning
+an explicit representation of the world dynamics i.e. a world model, improves
+sample efficiency while contrastive learning implicitly enforces learning of
+invariant features, which improves generalization. However, the naive
+integration of contrastive loss to world models fails due to a lack of
+supervisory signals to the visual encoder, as world-model-based RL methods
+independently optimize representation learning and agent policy. To overcome
+this issue, we propose an intervention-invariant regularizer in the form of an
+auxiliary task such as depth prediction, image denoising, etc., that explicitly
+enforces invariance to style-interventions. Our method outperforms current
+state-of-the-art model-based and model-free RL methods and significantly on
+out-of-distribution point navigation task evaluated on the iGibson benchmark.
+We further demonstrate that our approach, with only visual observations,
+outperforms recent language-guided foundation models for point navigation,
+which is essential for deployment on robots with limited computation
+capabilities. Finally, we demonstrate that our proposed model excels at the
+sim-to-real transfer of its perception module on Gibson benchmark.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV', 'cs.RO', 'stat.ML']"
+Semantic Human Mesh Reconstruction with Textures,xiaoyu zhan · Jianxin Yang · Yuanqi Li · Jie Guo · Yanwen Guo · Wenping Wang,https://zhanxy.xyz/projects/shert,https://arxiv.org/abs/2403.02561,,2403.02561.pdf,Semantic Human Mesh Reconstruction with Textures,"The field of 3D detailed human mesh reconstruction has made significant
+progress in recent years. However, current methods still face challenges when
+used in industrial applications due to unstable results, low-quality meshes,
+and a lack of UV unwrapping and skinning weights. In this paper, we present
+SHERT, a novel pipeline that can reconstruct semantic human meshes with
+textures and high-precision details. SHERT applies semantic- and normal-based
+sampling between the detailed surface (e.g. mesh and SDF) and the corresponding
+SMPL-X model to obtain a partially sampled semantic mesh and then generates the
+complete semantic mesh by our specifically designed self-supervised completion
+and refinement networks. Using the complete semantic mesh as a basis, we employ
+a texture diffusion model to create human textures that are driven by both
+images and texts. Our reconstructed meshes have stable UV unwrapping,
+high-quality triangle meshes, and consistent semantic information. The given
+SMPL-X model provides semantic information and shape priors, allowing SHERT to
+perform well even with incorrect and incomplete inputs. The semantic
+information also makes it easy to substitute and animate different body parts
+such as the face, body, and hands. Quantitative and qualitative experiments
+demonstrate that SHERT is capable of producing high-fidelity and robust
+semantic meshes that outperform state-of-the-art methods.",cs.CV,['cs.CV']
+Infrared Small Target Detection with Scale and Location Sensitivity,Qiankun Liu · Rui Liu · Bolun Zheng · Hongkui Wang · Ying Fu, ,https://arxiv.org/abs/2403.19366,,2403.19366.pdf,Infrared Small Target Detection with Scale and Location Sensitivity,"Recently, infrared small target detection (IRSTD) has been dominated by
+deep-learning-based methods. However, these methods mainly focus on the design
+of complex model structures to extract discriminative features, leaving the
+loss functions for IRSTD under-explored. For example, the widely used
+Intersection over Union (IoU) and Dice losses lack sensitivity to the scales
+and locations of targets, limiting the detection performance of detectors. In
+this paper, we focus on boosting detection performance with a more effective
+loss but a simpler model structure. Specifically, we first propose a novel
+Scale and Location Sensitive (SLS) loss to handle the limitations of existing
+losses: 1) for scale sensitivity, we compute a weight for the IoU loss based on
+target scales to help the detector distinguish targets with different scales:
+2) for location sensitivity, we introduce a penalty term based on the center
+points of targets to help the detector localize targets more precisely. Then,
+we design a simple Multi-Scale Head to the plain U-Net (MSHNet). By applying
+SLS loss to each scale of the predictions, our MSHNet outperforms existing
+state-of-the-art methods by a large margin. In addition, the detection
+performance of existing detectors can be further improved when trained with our
+SLS loss, demonstrating the effectiveness and generalization of our SLS loss.
+The code is available at https://github.com/ying-fu/MSHNet.",cs.CV,['cs.CV']
+Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships,Sebastian Koch · Narunas Vaskevicius · Mirco Colosi · Pedro Hermosilla · Timo Ropinski,https://kochsebastian.com/open3dsg,https://arxiv.org/abs/2402.12259,,2402.12259.pdf,Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships,"Current approaches for 3D scene graph prediction rely on labeled datasets to
+train models for a fixed set of known object classes and relationship
+categories. We present Open3DSG, an alternative approach to learn 3D scene
+graph prediction in an open world without requiring labeled scene graph data.
+We co-embed the features from a 3D scene graph prediction backbone with the
+feature space of powerful open world 2D vision language foundation models. This
+enables us to predict 3D scene graphs from 3D point clouds in a zero-shot
+manner by querying object classes from an open vocabulary and predicting the
+inter-object relationships from a grounded LLM with scene graph features and
+queried object classes as context. Open3DSG is the first 3D point cloud method
+to predict not only explicit open-vocabulary object classes, but also open-set
+relationships that are not limited to a predefined label set, making it
+possible to express rare as well as specific objects and relationships in the
+predicted 3D scene graph. Our experiments show that Open3DSG is effective at
+predicting arbitrary object classes as well as their complex inter-object
+relationships describing spatial, supportive, semantic and comparative
+relationships.",cs.CV,['cs.CV']
+SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation,Bin Xie · Jiale Cao · Jin Xie · Fahad Shahbaz Khan · Yanwei Pang,https://github.com/xb534/SED,https://arxiv.org/abs/2311.15537,,2311.15537.pdf,SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation,"Open-vocabulary semantic segmentation strives to distinguish pixels into
+different semantic groups from an open set of categories. Most existing methods
+explore utilizing pre-trained vision-language models, in which the key is to
+adopt the image-level model for pixel-level segmentation task. In this paper,
+we propose a simple encoder-decoder, named SED, for open-vocabulary semantic
+segmentation, which comprises a hierarchical encoder-based cost map generation
+and a gradual fusion decoder with category early rejection. The hierarchical
+encoder-based cost map generation employs hierarchical backbone, instead of
+plain transformer, to predict pixel-level image-text cost map. Compared to
+plain transformer, hierarchical backbone better captures local spatial
+information and has linear computational complexity with respect to input size.
+Our gradual fusion decoder employs a top-down structure to combine cost map and
+the feature maps of different backbone levels for segmentation. To accelerate
+inference speed, we introduce a category early rejection scheme in the decoder
+that rejects many no-existing categories at the early layer of decoder,
+resulting in at most 4.7 times acceleration without accuracy degradation.
+Experiments are performed on multiple open-vocabulary semantic segmentation
+datasets, which demonstrates the efficacy of our SED method. When using
+ConvNeXt-B, our SED method achieves mIoU score of 31.6\% on ADE20K with 150
+categories at 82 millisecond ($ms$) per image on a single A6000. We will
+release it at \url{https://github.com/xb534/SED.git}.",cs.CV,['cs.CV']
+"Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editing",Boqiang Zhang · Hongtao Xie · Zuan Gao · Yuxin Wang, ,https://arxiv.org/abs/2405.12724,,2405.12724.pdf,RemoCap: Disentangled Representation Learning for Motion Capture,"Reconstructing 3D human bodies from realistic motion sequences remains a
+challenge due to pervasive and complex occlusions. Current methods struggle to
+capture the dynamics of occluded body parts, leading to model penetration and
+distorted motion. RemoCap leverages Spatial Disentanglement (SD) and Motion
+Disentanglement (MD) to overcome these limitations. SD addresses occlusion
+interference between the target human body and surrounding objects. It achieves
+this by disentangling target features along the dimension axis. By aligning
+features based on their spatial positions in each dimension, SD isolates the
+target object's response within a global window, enabling accurate capture
+despite occlusions. The MD module employs a channel-wise temporal shuffling
+strategy to simulate diverse scene dynamics. This process effectively
+disentangles motion features, allowing RemoCap to reconstruct occluded parts
+with greater fidelity. Furthermore, this paper introduces a sequence velocity
+loss that promotes temporal coherence. This loss constrains inter-frame
+velocity errors, ensuring the predicted motion exhibits realistic consistency.
+Extensive comparisons with state-of-the-art (SOTA) methods on benchmark
+datasets demonstrate RemoCap's superior performance in 3D human body
+reconstruction. On the 3DPW dataset, RemoCap surpasses all competitors,
+achieving the best results in MPVPE (81.9), MPJPE (72.7), and PA-MPJPE (44.1)
+metrics. Codes are available at https://wanghongsheng01.github.io/RemoCap/.",cs.CV,['cs.CV']
+SpecNeRF: Gaussian Directional Encoding for Specular Reflections,Li Ma · Vasu Agrawal · Haithem Turki · Changil Kim · Chen Gao · Pedro V. Sander · Michael Zollhoefer · Christian Richardt, ,https://arxiv.org/abs/2312.13102,,2312.13102.pdf,SpecNeRF: Gaussian Directional Encoding for Specular Reflections,"Neural radiance fields have achieved remarkable performance in modeling the
+appearance of 3D scenes. However, existing approaches still struggle with the
+view-dependent appearance of glossy surfaces, especially under complex lighting
+of indoor environments. Unlike existing methods, which typically assume distant
+lighting like an environment map, we propose a learnable Gaussian directional
+encoding to better model the view-dependent effects under near-field lighting
+conditions. Importantly, our new directional encoding captures the
+spatially-varying nature of near-field lighting and emulates the behavior of
+prefiltered environment maps. As a result, it enables the efficient evaluation
+of preconvolved specular color at any 3D location with varying roughness
+coefficients. We further introduce a data-driven geometry prior that helps
+alleviate the shape radiance ambiguity in reflection modeling. We show that our
+Gaussian directional encoding and geometry prior significantly improve the
+modeling of challenging specular reflections in neural radiance fields, which
+helps decompose appearance into more physically meaningful components.",cs.CV,['cs.CV']
+Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation,Jiaming Liu · Ran Xu · Senqiao Yang · Renrui Zhang · Qizhe Zhang · Zehui Chen · Yandong Guo · Shanghang Zhang,https://sites.google.com/view/continual-mae/home,https://arxiv.org/abs/2312.12480,,2312.12480.pdf,Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation,"Continual Test-Time Adaptation (CTTA) is proposed to migrate a source
+pre-trained model to continually changing target distributions, addressing
+real-world dynamism. Existing CTTA methods mainly rely on entropy minimization
+or teacher-student pseudo-labeling schemes for knowledge extraction in
+unlabeled target domains. However, dynamic data distributions cause
+miscalibrated predictions and noisy pseudo-labels in existing self-supervised
+learning methods, hindering the effective mitigation of error accumulation and
+catastrophic forgetting problems during the continual adaptation process. To
+tackle these issues, we propose a continual self-supervised method, Adaptive
+Distribution Masked Autoencoders (ADMA), which enhances the extraction of
+target domain knowledge while mitigating the accumulation of distribution
+shifts. Specifically, we propose a Distribution-aware Masking (DaM) mechanism
+to adaptively sample masked positions, followed by establishing consistency
+constraints between the masked target samples and the original target samples.
+Additionally, for masked tokens, we utilize an efficient decoder to reconstruct
+a hand-crafted feature descriptor (e.g., Histograms of Oriented Gradients),
+leveraging its invariant properties to boost task-relevant representations.
+Through conducting extensive experiments on four widely recognized benchmarks,
+our proposed method attains state-of-the-art performance in both classification
+and segmentation CTTA tasks. Our project page:
+https://sites.google.com/view/continual-mae/home.",cs.CV,['cs.CV']
+3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow,Felix Taubner · Prashant Raina · Mathieu Tuli · Eu Wern Teh · Chul Lee · Jinmiao Huang,https://felixtaubner.github.io/flowface,https://arxiv.org/abs/2404.09819,,2404.09819.pdf,3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow,"When working with 3D facial data, improving fidelity and avoiding the uncanny
+valley effect is critically dependent on accurate 3D facial performance
+capture. Because such methods are expensive and due to the widespread
+availability of 2D videos, recent methods have focused on how to perform
+monocular 3D face tracking. However, these methods often fall short in
+capturing precise facial movements due to limitations in their network
+architecture, training, and evaluation processes. Addressing these challenges,
+we propose a novel face tracker, FlowFace, that introduces an innovative 2D
+alignment network for dense per-vertex alignment. Unlike prior work, FlowFace
+is trained on high-quality 3D scan annotations rather than weak supervision or
+synthetic data. Our 3D model fitting module jointly fits a 3D face model from
+one or many observations, integrating existing neutral shape priors for
+enhanced identity and expression disentanglement and per-vertex deformations
+for detailed facial feature reconstruction. Additionally, we propose a novel
+metric and benchmark for assessing tracking accuracy. Our method exhibits
+superior performance on both custom and publicly available benchmarks. We
+further validate the effectiveness of our tracker by generating high-quality 3D
+data from 2D videos, which leads to performance gains on downstream tasks.",cs.CV,['cs.CV']
+Deep Single Image Camera Calibration by Heatmap Regression to Recover Fisheye Images Under Manhattan World Assumption,Nobuhiko Wakai · Satoshi Sato · Yasunori Ishii · Takayoshi Yamashita, ,,https://paperswithcode.com/search?q=author:Yasunori+Ishii,,,,,nan
+Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation,Hongwei Yan · Liyuan Wang · Kaisheng Ma · Yi Zhong, ,https://arxiv.org/abs/2404.00417,,2404.00417.pdf,Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation,"To accommodate real-world dynamics, artificial intelligence systems need to
+cope with sequentially arriving content in an online manner. Beyond regular
+Continual Learning (CL) attempting to address catastrophic forgetting with
+offline training of each task, Online Continual Learning (OCL) is a more
+challenging yet realistic setting that performs CL in a one-pass data stream.
+Current OCL methods primarily rely on memory replay of old training samples.
+However, a notable gap from CL to OCL stems from the additional
+overfitting-underfitting dilemma associated with the use of rehearsal buffers:
+the inadequate learning of new training samples (underfitting) and the repeated
+learning of a few old training samples (overfitting). To this end, we introduce
+a novel approach, Multi-level Online Sequential Experts (MOSE), which
+cultivates the model as stacked sub-experts, integrating multi-level
+supervision and reverse self-distillation. Supervision signals across multiple
+stages facilitate appropriate convergence of the new task while gathering
+various strengths from experts by knowledge distillation mitigates the
+performance decline of old tasks. MOSE demonstrates remarkable efficacy in
+learning new samples and preserving past knowledge through multi-level experts,
+thereby significantly advancing OCL performance over state-of-the-art baselines
+(e.g., up to 7.3% on Split CIFAR-100 and 6.1% on Split Tiny-ImageNet).",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains,George Eskandar, ,https://arxiv.org/abs/2402.17562v1,,2402.17562v1.pdf,An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains,"3D Object Detectors (3D-OD) are crucial for understanding the environment in
+many robotic tasks, especially autonomous driving. Including 3D information via
+Lidar sensors improves accuracy greatly. However, such detectors perform poorly
+on domains they were not trained on, i.e. different locations, sensors,
+weather, etc., limiting their reliability in safety-critical applications.
+There exist methods to adapt 3D-ODs to these domains; however, these methods
+treat 3D-ODs as a black box, neglecting underlying architectural decisions and
+source-domain training strategies. Instead, we dive deep into the details of
+3D-ODs, focusing our efforts on fundamental factors that influence robustness
+prior to domain adaptation.
+  We systematically investigate four design choices (and the interplay between
+them) often overlooked in 3D-OD robustness and domain adaptation: architecture,
+voxel encoding, data augmentations, and anchor strategies. We assess their
+impact on the robustness of nine state-of-the-art 3D-ODs across six benchmarks
+encompassing three types of domain gaps - sensor type, weather, and location.
+  Our main findings are: (1) transformer backbones with local point features
+are more robust than 3D CNNs, (2) test-time anchor size adjustment is crucial
+for adaptation across geographical locations, significantly boosting scores
+without retraining, (3) source-domain augmentations allow the model to
+generalize to low-resolution sensors, and (4) surprisingly, robustness to bad
+weather is improved when training directly on more clean weather data than on
+training with bad weather data. We outline our main conclusions and findings to
+provide practical guidance on developing more robust 3D-ODs.",cs.CV,['cs.CV']
+FreeKD: Knowledge Distillation via Semantic Frequency Prompt,Yuan Zhang · Tao Huang · Jiaming Liu · Tao Jiang · Kuan Cheng · Shanghang Zhang, ,https://arxiv.org/abs/2311.12079,,2311.12079.pdf,FreeKD: Knowledge Distillation via Semantic Frequency Prompt,"Knowledge distillation (KD) has been applied to various tasks successfully,
+and mainstream methods typically boost the student model via spatial imitation
+losses. However, the consecutive downsamplings induced in the spatial domain of
+teacher model is a type of corruption, hindering the student from analyzing
+what specific information needs to be imitated, which results in accuracy
+degradation. To better understand the underlying pattern of corrupted feature
+maps, we shift our attention to the frequency domain. During frequency
+distillation, we encounter a new challenge: the low-frequency bands convey
+general but minimal context, while the high are more informative but also
+introduce noise. Not each pixel within the frequency bands contributes equally
+to the performance. To address the above problem: (1) We propose the Frequency
+Prompt plugged into the teacher model, absorbing the semantic frequency context
+during finetuning. (2) During the distillation period, a pixel-wise frequency
+mask is generated via Frequency Prompt, to localize those pixel of interests
+(PoIs) in various frequency bands. Additionally, we employ a position-aware
+relational frequency loss for dense prediction tasks, delivering a high-order
+spatial enhancement to the student model. We dub our Frequency Knowledge
+Distillation method as FreeKD, which determines the optimal localization and
+extent for the frequency distillation. Extensive experiments demonstrate that
+FreeKD not only outperforms spatial-based distillation methods consistently on
+dense prediction tasks (e.g., FreeKD brings 3.8 AP gains for RepPoints-R50 on
+COCO2017 and 4.55 mIoU gains for PSPNet-R18 on Cityscapes), but also conveys
+more robustness to the student. Notably, we also validate the generalization of
+our approach on large-scale vision models (e.g., DINO and SAM).",cs.CV,['cs.CV']
+Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation,Jihyun Kim · Changjae Oh · Hoseok Do · Soohyun Kim · Kwanghoon Sohn, ,https://arxiv.org/abs/2405.04356,,2405.04356.pdf,Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation,"We present a new multi-modal face image generation method that converts a
+text prompt and a visual input, such as a semantic mask or scribble map, into a
+photo-realistic face image. To do this, we combine the strengths of Generative
+Adversarial networks (GANs) and diffusion models (DMs) by employing the
+multi-modal features in the DM into the latent space of the pre-trained GANs.
+We present a simple mapping and a style modulation network to link two models
+and convert meaningful representations in feature maps and attention maps into
+latent codes. With GAN inversion, the estimated latent codes can be used to
+generate 2D or 3D-aware facial images. We further present a multi-step training
+strategy that reflects textual and structural representations into the
+generated image. Our proposed network produces realistic 2D, multi-view, and
+stylized face images, which align well with inputs. We validate our method by
+using pre-trained 2D and 3D GANs, and our results outperform existing methods.
+Our project page is available at
+https://github.com/1211sh/Diffusion-driven_GAN-Inversion/.",cs.CV,['cs.CV']
+Probing Synergistic High-Order Interaction in Infrared and Visible Image Fusion,Naishan Zheng · Man Zhou · Jie Huang · Junming Hou · Haoying Li · Yuan Xu · Feng Zhao, ,,https://ieeexplore.ieee.org/document/10539339,,,,,nan
+Bridging the Gap Between End-to-End and Two-Step Text Spotting,Mingxin Huang · Hongliang Li · Yuliang Liu · Xiang Bai · Lianwen Jin, ,https://arxiv.org/abs/2404.04624,,2404.04624.pdf,Bridging the Gap Between End-to-End and Two-Step Text Spotting,"Modularity plays a crucial role in the development and maintenance of complex
+systems. While end-to-end text spotting efficiently mitigates the issues of
+error accumulation and sub-optimal performance seen in traditional two-step
+methodologies, the two-step methods continue to be favored in many competitions
+and practical settings due to their superior modularity. In this paper, we
+introduce Bridging Text Spotting, a novel approach that resolves the error
+accumulation and suboptimal performance issues in two-step methods while
+retaining modularity. To achieve this, we adopt a well-trained detector and
+recognizer that are developed and trained independently and then lock their
+parameters to preserve their already acquired capabilities. Subsequently, we
+introduce a Bridge that connects the locked detector and recognizer through a
+zero-initialized neural network. This zero-initialized neural network,
+initialized with weights set to zeros, ensures seamless integration of the
+large receptive field features in detection into the locked recognizer.
+Furthermore, since the fixed detector and recognizer cannot naturally acquire
+end-to-end optimization features, we adopt the Adapter to facilitate their
+efficient learning of these features. We demonstrate the effectiveness of the
+proposed method through extensive experiments: Connecting the latest detector
+and recognizer through Bridging Text Spotting, we achieved an accuracy of 83.3%
+on Total-Text, 69.8% on CTW1500, and 89.5% on ICDAR 2015. The code is available
+at https://github.com/mxin262/Bridging-Text-Spotting.",cs.CV,['cs.CV']
+OED: Towards One-stage End-to-End Dynamic Scene Graph Generation,Guan Wang · Zhimin Li · Qingchao Chen · Yang Liu, ,https://arxiv.org/abs/2405.16925,,2405.16925.pdf,OED: Towards One-stage End-to-End Dynamic Scene Graph Generation,"Dynamic Scene Graph Generation (DSGG) focuses on identifying visual
+relationships within the spatial-temporal domain of videos. Conventional
+approaches often employ multi-stage pipelines, which typically consist of
+object detection, temporal association, and multi-relation classification.
+However, these methods exhibit inherent limitations due to the separation of
+multiple stages, and independent optimization of these sub-problems may yield
+sub-optimal solutions. To remedy these limitations, we propose a one-stage
+end-to-end framework, termed OED, which streamlines the DSGG pipeline. This
+framework reformulates the task as a set prediction problem and leverages
+pair-wise features to represent each subject-object pair within the scene
+graph. Moreover, another challenge of DSGG is capturing temporal dependencies,
+we introduce a Progressively Refined Module (PRM) for aggregating temporal
+context without the constraints of additional trackers or handcrafted
+trajectories, enabling end-to-end optimization of the network. Extensive
+experiments conducted on the Action Genome benchmark demonstrate the
+effectiveness of our design. The code and models are available at
+\url{https://github.com/guanw-pku/OED}.",cs.CV,['cs.CV']
+Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding,Zhiheng Cheng · Qingyue Wei · Hongru Zhu · Yan Wang · Liangqiong Qu · Wei Shao · Yuyin Zhou, ,https://arxiv.org/abs/2403.18271,,2403.18271.pdf,Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding,"The Segment Anything Model (SAM) has garnered significant attention for its
+versatile segmentation abilities and intuitive prompt-based interface. However,
+its application in medical imaging presents challenges, requiring either
+substantial training costs and extensive medical datasets for full model
+fine-tuning or high-quality prompts for optimal performance. This paper
+introduces H-SAM: a prompt-free adaptation of SAM tailored for efficient
+fine-tuning of medical images via a two-stage hierarchical decoding procedure.
+In the initial stage, H-SAM employs SAM's original decoder to generate a prior
+probabilistic mask, guiding a more intricate decoding process in the second
+stage. Specifically, we propose two key designs: 1) A class-balanced,
+mask-guided self-attention mechanism addressing the unbalanced label
+distribution, enhancing image embedding; 2) A learnable mask cross-attention
+mechanism spatially modulating the interplay among different image regions
+based on the prior mask. Moreover, the inclusion of a hierarchical pixel
+decoder in H-SAM enhances its proficiency in capturing fine-grained and
+localized details. This approach enables SAM to effectively integrate learned
+medical priors, facilitating enhanced adaptation for medical image segmentation
+with limited samples. Our H-SAM demonstrates a 4.78% improvement in average
+Dice compared to existing prompt-free SAM variants for multi-organ segmentation
+using only 10% of 2D slices. Notably, without using any unlabeled data, H-SAM
+even outperforms state-of-the-art semi-supervised models relying on extensive
+unlabeled training data across various medical datasets. Our code is available
+at https://github.com/Cccccczh404/H-SAM.",cs.CV,['cs.CV']
+Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers,Jinxia Xie · Bineng Zhong · Zhiyi Mo · Shengping Zhang · Liangtao Shi · Shuxiang Song · Rongrong Ji, ,https://arxiv.org/abs/2403.10574,,2403.10574.pdf,Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers,"The rich spatio-temporal information is crucial to capture the complicated
+target appearance variations in visual tracking. However, most top-performing
+tracking algorithms rely on many hand-crafted components for spatio-temporal
+information aggregation. Consequently, the spatio-temporal information is far
+away from being fully explored. To alleviate this issue, we propose an adaptive
+tracker with spatio-temporal transformers (named AQATrack), which adopts simple
+autoregressive queries to effectively learn spatio-temporal information without
+many hand-designed components. Firstly, we introduce a set of learnable and
+autoregressive queries to capture the instantaneous target appearance changes
+in a sliding window fashion. Then, we design a novel attention mechanism for
+the interaction of existing queries to generate a new query in current frame.
+Finally, based on the initial target template and learnt autoregressive
+queries, a spatio-temporal information fusion module (STM) is designed for
+spatiotemporal formation aggregation to locate a target object. Benefiting from
+the STM, we can effectively combine the static appearance and instantaneous
+changes to guide robust tracking. Extensive experiments show that our method
+significantly improves the tracker's performance on six popular tracking
+benchmarks: LaSOT, LaSOText, TrackingNet, GOT-10k, TNL2K, and UAV123.",cs.CV,['cs.CV']
+GRAM: Global Reasoning for Multi-Page VQA,Itshak Blau · Sharon Fogel · Roi Ronen · Alona Golts · Shahar Tsiper · Elad Ben Avraham · Aviad Aberdam · Roy Ganz · Ron Litman, ,https://arxiv.org/abs/2401.03411,,2401.03411.pdf,GRAM: Global Reasoning for Multi-Page VQA,"The increasing use of transformer-based large language models brings forward
+the challenge of processing long sequences. In document visual question
+answering (DocVQA), leading methods focus on the single-page setting, while
+documents can span hundreds of pages. We present GRAM, a method that seamlessly
+extends pre-trained single-page models to the multi-page setting, without
+requiring computationally-heavy pretraining. To do so, we leverage a
+single-page encoder for local page-level understanding, and enhance it with
+document-level designated layers and learnable tokens, facilitating the flow of
+information across pages for global reasoning. To enforce our model to utilize
+the newly introduced document tokens, we propose a tailored bias adaptation
+method. For additional computational savings during decoding, we introduce an
+optional compression stage using our compression-transformer
+(C-Former),reducing the encoded sequence length, thereby allowing a tradeoff
+between quality and latency. Extensive experiments showcase GRAM's
+state-of-the-art performance on the benchmarks for multi-page DocVQA,
+demonstrating the effectiveness of our approach.",cs.CL,"['cs.CL', 'cs.CV']"
+Attribute-Guided Pedestrian Retrieval: Bridging Person Re-ID with Internal Attribute Variability,Yan Huang · Zhang Zhang · Qiang Wu · yi zhong · Liang Wang, ,,https://www.youtube.com/watch?v=5xrCUp_gdwg,,,,,nan
+AnyScene: Customized Image Synthesis with Composited Foreground,Ruidong Chen · Lanjun Wang · Weizhi Nie · Yongdong Zhang · An-An Liu, ,https://ar5iv.labs.arxiv.org/html/2302.09778,,2302.09778.pdf,Composer: Creative and Controllable Image Synthesis with Composable Conditions,"Recent large-scale generative models learned on big data are capable of
+synthesizing incredible images yet suffer from limited controllability. This
+work offers a new generation paradigm that allows flexible control of the
+output image, such as spatial layout and palette, while maintaining the
+synthesis quality and model creativity. With compositionality as the core idea,
+we first decompose an image into representative factors, and then train a
+diffusion model with all these factors as the conditions to recompose the
+input. At the inference stage, the rich intermediate representations work as
+composable elements, leading to a huge design space (i.e., exponentially
+proportional to the number of decomposed factors) for customizable content
+creation. It is noteworthy that our approach, which we call Composer, supports
+various levels of conditions, such as text description as the global
+information, depth map and sketch as the local guidance, color histogram for
+low-level details, etc. Besides improving controllability, we confirm that
+Composer serves as a general framework and facilitates a wide range of
+classical generative tasks without retraining. Code and models will be made
+available.",cs.CV,"['cs.CV', 'cs.GR']"
+Multiway Point Cloud Mosaicking with Diffusion and Global Optimization,Shengze Jin · Iro Armeni · Marc Pollefeys · Daniel Barath, ,https://arxiv.org/abs/2404.00429,,2404.00429.pdf,Multiway Point Cloud Mosaicking with Diffusion and Global Optimization,"We introduce a novel framework for multiway point cloud mosaicking (named
+Wednesday), designed to co-align sets of partially overlapping point clouds --
+typically obtained from 3D scanners or moving RGB-D cameras -- into a unified
+coordinate system. At the core of our approach is ODIN, a learned pairwise
+registration algorithm that iteratively identifies overlaps and refines
+attention scores, employing a diffusion-based process for denoising pairwise
+correlation matrices to enhance matching accuracy. Further steps include
+constructing a pose graph from all point clouds, performing rotation averaging,
+a novel robust algorithm for re-estimating translations optimally in terms of
+consensus maximization and translation optimization. Finally, the point cloud
+rotations and positions are optimized jointly by a diffusion-based approach.
+Tested on four diverse, large-scale datasets, our method achieves
+state-of-the-art pairwise and multiway registration results by a large margin
+on all benchmarks. Our code and models are available at
+https://github.com/jinsz/Multiway-Point-Cloud-Mosaicking-with-Diffusion-and-Global-Optimization.",cs.CV,['cs.CV']
+Dexterous Grasp Transformer,Guo-Hao Xu · Yi-Lin Wei · Dian Zheng · Xiao-Ming Wu · Wei-Shi Zheng, ,https://arxiv.org/abs/2404.18135,,2404.18135.pdf,Dexterous Grasp Transformer,"In this work, we propose a novel discriminative framework for dexterous grasp
+generation, named Dexterous Grasp TRansformer (DGTR), capable of predicting a
+diverse set of feasible grasp poses by processing the object point cloud with
+only one forward pass. We formulate dexterous grasp generation as a set
+prediction task and design a transformer-based grasping model for it. However,
+we identify that this set prediction paradigm encounters several optimization
+challenges in the field of dexterous grasping and results in restricted
+performance. To address these issues, we propose progressive strategies for
+both the training and testing phases. First, the dynamic-static matching
+training (DSMT) strategy is presented to enhance the optimization stability
+during the training phase. Second, we introduce the adversarial-balanced
+test-time adaptation (AB-TTA) with a pair of adversarial losses to improve
+grasping quality during the testing phase. Experimental results on the
+DexGraspNet dataset demonstrate the capability of DGTR to predict dexterous
+grasp poses with both high quality and diversity. Notably, while keeping high
+quality, the diversity of grasp poses predicted by DGTR significantly
+outperforms previous works in multiple metrics without any data pre-processing.
+Codes are available at https://github.com/iSEE-Laboratory/DGTR .",cs.RO,['cs.RO']
+MoCha-Stereo: Motif Channel Attention Network for Stereo Matching,Ziyang Chen · Wei Long · He Yao · Yongjun Zhang · Bingshu Wang · Yongbin Qin · Jia Wu,https://github.com/ZYangChen/MoCha-Stereo,https://arxiv.org/abs/2404.06842,,2404.06842.pdf,MoCha-Stereo: Motif Channel Attention Network for Stereo Matching,"Learning-based stereo matching techniques have made significant progress.
+However, existing methods inevitably lose geometrical structure information
+during the feature channel generation process, resulting in edge detail
+mismatches. In this paper, the Motif Cha}nnel Attention Stereo Matching Network
+(MoCha-Stereo) is designed to address this problem. We provide the Motif
+Channel Correlation Volume (MCCV) to determine more accurate edge matching
+costs. MCCV is achieved by projecting motif channels, which capture common
+geometric structures in feature channels, onto feature maps and cost volumes.
+In addition, edge variations in %potential feature channels of the
+reconstruction error map also affect details matching, we propose the
+Reconstruction Error Motif Penalty (REMP) module to further refine the
+full-resolution disparity estimation. REMP integrates the frequency information
+of typical channel features from the reconstruction error. MoCha-Stereo ranks
+1st on the KITTI-2015 and KITTI-2012 Reflective leaderboards. Our structure
+also shows excellent performance in Multi-View Stereo. Code is avaliable at
+https://github.com/ZYangChen/MoCha-Stereo.",cs.CV,['cs.CV']
+Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis,Zhan Li · Zhang Chen · Zhong Li · Yi Xu,https://oppo-us-research.github.io/SpacetimeGaussians-website/,https://arxiv.org/abs/2312.16812,,2312.16812.pdf,Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis,"Novel view synthesis of dynamic scenes has been an intriguing yet challenging
+problem. Despite recent advancements, simultaneously achieving high-resolution
+photorealistic results, real-time rendering, and compact storage remains a
+formidable task. To address these challenges, we propose Spacetime Gaussian
+Feature Splatting as a novel dynamic scene representation, composed of three
+pivotal components. First, we formulate expressive Spacetime Gaussians by
+enhancing 3D Gaussians with temporal opacity and parametric motion/rotation.
+This enables Spacetime Gaussians to capture static, dynamic, as well as
+transient content within a scene. Second, we introduce splatted feature
+rendering, which replaces spherical harmonics with neural features. These
+features facilitate the modeling of view- and time-dependent appearance while
+maintaining small size. Third, we leverage the guidance of training error and
+coarse depth to sample new Gaussians in areas that are challenging to converge
+with existing pipelines. Experiments on several established real-world datasets
+demonstrate that our method achieves state-of-the-art rendering quality and
+speed, while retaining compact storage. At 8K resolution, our lite-version
+model can render at 60 FPS on an Nvidia RTX 4090 GPU. Our code is available at
+https://github.com/oppo-us-research/SpacetimeGaussians.",cs.CV,"['cs.CV', 'cs.GR']"
+MoReVQA: Exploring Modular Reasoning Models for Video Question Answering,Juhong Min · Shyamal Buch · Arsha Nagrani · Minsu Cho · Cordelia Schmid, ,https://arxiv.org/abs/2404.06511,,2404.06511.pdf,MoReVQA: Exploring Modular Reasoning Models for Video Question Answering,"This paper addresses the task of video question answering (videoQA) via a
+decomposed multi-stage, modular reasoning framework. Previous modular methods
+have shown promise with a single planning stage ungrounded in visual content.
+However, through a simple and effective baseline, we find that such systems can
+lead to brittle behavior in practice for challenging videoQA settings. Thus,
+unlike traditional single-stage planning methods, we propose a multi-stage
+system consisting of an event parser, a grounding stage, and a final reasoning
+stage in conjunction with an external memory. All stages are training-free, and
+performed using few-shot prompting of large models, creating interpretable
+intermediate outputs at each stage. By decomposing the underlying planning and
+task complexity, our method, MoReVQA, improves over prior work on standard
+videoQA benchmarks (NExT-QA, iVQA, EgoSchema, ActivityNet-QA) with
+state-of-the-art results, and extensions to related tasks (grounded videoQA,
+paragraph captioning).",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+DiffAvatar: Simulation-Ready Garment Optimization with Differentiable Simulation,Yifei Li · Hsiaoyu Chen · Egor Larionov · Nikolaos Sarafianos · Wojciech Matusik · Tuur Stuyck, ,https://arxiv.org/abs/2311.12194,,2311.12194.pdf,DiffAvatar: Simulation-Ready Garment Optimization with Differentiable Simulation,"The realism of digital avatars is crucial in enabling telepresence
+applications with self-expression and customization. While physical simulations
+can produce realistic motions for clothed humans, they require high-quality
+garment assets with associated physical parameters for cloth simulations.
+However, manually creating these assets and calibrating their parameters is
+labor-intensive and requires specialized expertise. Current methods focus on
+reconstructing geometry, but don't generate complete assets for physics-based
+applications. To address this gap, we propose \papername,~a novel approach that
+performs body and garment co-optimization using differentiable simulation. By
+integrating physical simulation into the optimization loop and accounting for
+the complex nonlinear behavior of cloth and its intricate interaction with the
+body, our framework recovers body and garment geometry and extracts important
+material parameters in a physically plausible way. Our experiments demonstrate
+that our approach generates realistic clothing and body shape suitable for
+downstream applications. We provide additional insights and results on our
+webpage: https://people.csail.mit.edu/liyifei/publication/diffavatar/",cs.CV,['cs.CV']
+SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers,Ioannis Kakogeorgiou · Spyros Gidaris · Konstantinos Karantzalos · Nikos Komodakis, ,https://arxiv.org/abs/2312.00648,,2312.00648.pdf,SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers,"Unsupervised object-centric learning aims to decompose scenes into
+interpretable object entities, termed slots. Slot-based auto-encoders stand out
+as a prominent method for this task. Within them, crucial aspects include
+guiding the encoder to generate object-specific slots and ensuring the decoder
+utilizes them during reconstruction. This work introduces two novel techniques,
+(i) an attention-based self-training approach, which distills superior
+slot-based attention masks from the decoder to the encoder, enhancing object
+segmentation, and (ii) an innovative patch-order permutation strategy for
+autoregressive transformers that strengthens the role of slot vectors in
+reconstruction. The effectiveness of these strategies is showcased
+experimentally. The combined approach significantly surpasses prior slot-based
+autoencoder methods in unsupervised object segmentation, especially with
+complex real-world images. We provide the implementation code at
+https://github.com/gkakogeorgiou/spot .",cs.CV,['cs.CV']
+Color Shift Estimation-and-Correction for Image Enhancement,Yiyu Li · Ke Xu · Gerhard Hancke · Rynson W.H. Lau, ,https://arxiv.org/abs/2405.17725,,2405.17725.pdf,Color Shift Estimation-and-Correction for Image Enhancement,"Images captured under sub-optimal illumination conditions may contain both
+over- and under-exposures. Current approaches mainly focus on adjusting image
+brightness, which may exacerbate the color tone distortion in under-exposed
+areas and fail to restore accurate colors in over-exposed regions. We observe
+that over- and under-exposed regions display opposite color tone distribution
+shifts with respect to each other, which may not be easily normalized in joint
+modeling as they usually do not have ``normal-exposed'' regions/pixels as
+reference. In this paper, we propose a novel method to enhance images with both
+over- and under-exposures by learning to estimate and correct such color
+shifts. Specifically, we first derive the color feature maps of the brightened
+and darkened versions of the input image via a UNet-based network, followed by
+a pseudo-normal feature generator to produce pseudo-normal color feature maps.
+We then propose a novel COlor Shift Estimation (COSE) module to estimate the
+color shifts between the derived brightened (or darkened) color feature maps
+and the pseudo-normal color feature maps. The COSE module corrects the
+estimated color shifts of the over- and under-exposed regions separately. We
+further propose a novel COlor MOdulation (COMO) module to modulate the
+separately corrected colors in the over- and under-exposed regions to produce
+the enhanced image. Comprehensive experiments show that our method outperforms
+existing approaches. Project webpage: https://github.com/yiyulics/CSEC.",cs.CV,['cs.CV']
+Human Gaussian Splatting : Real-time Rendering of Animatable Avatars,Arthur Moreau · Jifei Song · Helisa Dhamo · Richard Shaw · Yiren Zhou · Eduardo Pérez-Pellitero,https://perezpellitero.github.io/projects/hugs/index.html,https://arxiv.org/abs/2311.17113,,2311.17113.pdf,Human Gaussian Splatting: Real-time Rendering of Animatable Avatars,"This work addresses the problem of real-time rendering of photorealistic
+human body avatars learned from multi-view videos. While the classical
+approaches to model and render virtual humans generally use a textured mesh,
+recent research has developed neural body representations that achieve
+impressive visual quality. However, these models are difficult to render in
+real-time and their quality degrades when the character is animated with body
+poses different than the training observations. We propose an animatable human
+model based on 3D Gaussian Splatting, that has recently emerged as a very
+efficient alternative to neural radiance fields. The body is represented by a
+set of gaussian primitives in a canonical space which is deformed with a coarse
+to fine approach that combines forward skinning and local non-rigid refinement.
+We describe how to learn our Human Gaussian Splatting (HuGS) model in an
+end-to-end fashion from multi-view observations, and evaluate it against the
+state-of-the-art approaches for novel pose synthesis of clothed body. Our
+method achieves 1.5 dB PSNR improvement over the state-of-the-art on THuman4
+dataset while being able to render in real-time (80 fps for 512x512
+resolution).",cs.CV,"['cs.CV', 'cs.GR']"
+Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations,Rui Zhao · Ruiqin Xiong · Jing Zhao · Jian Zhang · Xiaopeng Fan · Zhaofei Yu · Tiejun Huang, ,https://ar5iv.labs.arxiv.org/html/2303.11684,,2303.11684.pdf,SpikeCV: Open a Continuous Computer Vision Era,"SpikeCV is a new open-source computer vision platform for the spike camera,
+which is a neuromorphic visual sensor that has developed rapidly in recent
+years. In the spike camera, each pixel position directly accumulates the light
+intensity and asynchronously fires spikes. The output binary spikes can reach a
+frequency of 40,000 Hz. As a new type of visual expression, spike sequence has
+high spatiotemporal completeness and preserves the continuous visual
+information of the external world. Taking advantage of the low latency and high
+dynamic range of the spike camera, many spike-based algorithms have made
+significant progress, such as high-quality imaging and ultra-high-speed target
+detection.
+  To build up a community ecology for the spike vision to facilitate more users
+to take advantage of the spike camera, SpikeCV provides a variety of
+ultra-high-speed scene datasets, hardware interfaces, and an easy-to-use
+modules library. SpikeCV focuses on encapsulation for spike data,
+standardization for dataset interfaces, modularization for vision tasks, and
+real-time applications for challenging scenes. With the advent of the
+open-source Python ecosystem, modules of SpikeCV can be used as a Python
+library to fulfilled most of the numerical analysis needs of researchers. We
+demonstrate the efficiency of the SpikeCV on offline inference and real-time
+applications. The project repository address are
+\url{https://openi.pcl.ac.cn/Cordium/SpikeCV} and
+\url{https://github.com/Zyj061/SpikeCV",cs.CV,['cs.CV']
+DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans,Akash Sengupta · Thiemo Alldieck · NIKOS KOLOTOUROS · Enric Corona · Andrei Zanfir · Cristian Sminchisescu,https://akashsengupta1997.github.io/diffhuman/,https://arxiv.org/abs/2404.00485,,2404.00485.pdf,DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans,"We present DiffHuman, a probabilistic method for photorealistic 3D human
+reconstruction from a single RGB image. Despite the ill-posed nature of this
+problem, most methods are deterministic and output a single solution, often
+resulting in a lack of geometric detail and blurriness in unseen or uncertain
+regions. In contrast, DiffHuman predicts a probability distribution over 3D
+reconstructions conditioned on an input 2D image, which allows us to sample
+multiple detailed 3D avatars that are consistent with the image. DiffHuman is
+implemented as a conditional diffusion model that denoises pixel-aligned 2D
+observations of an underlying 3D shape representation. During inference, we may
+sample 3D avatars by iteratively denoising 2D renders of the predicted 3D
+representation. Furthermore, we introduce a generator neural network that
+approximates rendering with considerably reduced runtime (55x speed up),
+resulting in a novel dual-branch diffusion framework. Our experiments show that
+DiffHuman can produce diverse and detailed reconstructions for the parts of the
+person that are unseen or uncertain in the input image, while remaining
+competitive with the state-of-the-art when reconstructing visible surfaces.",cs.CV,['cs.CV']
+LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulation,Ke Guo · Zhenwei Miao · Wei Jing · Weiwei Liu · Weizi Li · Dayang Hao · Jia Pan,https://sites.google.com/view/lasil,https://arxiv.org/abs/2403.17601,,2403.17601.pdf,LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulation,"Microscopic traffic simulation plays a crucial role in transportation
+engineering by providing insights into individual vehicle behavior and overall
+traffic flow. However, creating a realistic simulator that accurately
+replicates human driving behaviors in various traffic conditions presents
+significant challenges. Traditional simulators relying on heuristic models
+often fail to deliver accurate simulations due to the complexity of real-world
+traffic environments. Due to the covariate shift issue, existing imitation
+learning-based simulators often fail to generate stable long-term simulations.
+In this paper, we propose a novel approach called learner-aware supervised
+imitation learning to address the covariate shift problem in multi-agent
+imitation learning. By leveraging a variational autoencoder simultaneously
+modeling the expert and learner state distribution, our approach augments
+expert states such that the augmented state is aware of learner state
+distribution. Our method, applied to urban traffic simulation, demonstrates
+significant improvements over existing state-of-the-art baselines in both
+short-term microscopic and long-term macroscopic realism when evaluated on the
+real-world dataset pNEUMA.",cs.AI,"['cs.AI', 'cs.LG']"
+Sparse Semi-Detr: Sparse Learnable Queries for Semi-Supervised Object Detection,Tahira Shehzadi · Khurram Azeem Hashmi · Didier Stricker · Muhammad Zeshan Afzal, ,https://arxiv.org/abs/2404.01819,,2404.01819.pdf,Sparse Semi-DETR: Sparse Learnable Queries for Semi-Supervised Object Detection,"In this paper, we address the limitations of the DETR-based semi-supervised
+object detection (SSOD) framework, particularly focusing on the challenges
+posed by the quality of object queries. In DETR-based SSOD, the one-to-one
+assignment strategy provides inaccurate pseudo-labels, while the one-to-many
+assignments strategy leads to overlapping predictions. These issues compromise
+training efficiency and degrade model performance, especially in detecting
+small or occluded objects. We introduce Sparse Semi-DETR, a novel
+transformer-based, end-to-end semi-supervised object detection solution to
+overcome these challenges. Sparse Semi-DETR incorporates a Query Refinement
+Module to enhance the quality of object queries, significantly improving
+detection capabilities for small and partially obscured objects. Additionally,
+we integrate a Reliable Pseudo-Label Filtering Module that selectively filters
+high-quality pseudo-labels, thereby enhancing detection accuracy and
+consistency. On the MS-COCO and Pascal VOC object detection benchmarks, Sparse
+Semi-DETR achieves a significant improvement over current state-of-the-art
+methods that highlight Sparse Semi-DETR's effectiveness in semi-supervised
+object detection, particularly in challenging scenarios involving small or
+partially obscured objects.",cs.CV,['cs.CV']
+CityDreamer: Compositional Generative Model of Unbounded 3D Cities,Haozhe Xie · Zhaoxi Chen · Fangzhou Hong · Ziwei Liu,https://www.infinitescript.com/project/city-dreamer,https://arxiv.org/abs/2309.00610,,2309.00610.pdf,CityDreamer: Compositional Generative Model of Unbounded 3D Cities,"3D city generation is a desirable yet challenging task, since humans are more
+sensitive to structural distortions in urban environments. Additionally,
+generating 3D cities is more complex than 3D natural scenes since buildings, as
+objects of the same class, exhibit a wider range of appearances compared to the
+relatively consistent appearance of objects like trees in natural scenes. To
+address these challenges, we propose \textbf{CityDreamer}, a compositional
+generative model designed specifically for unbounded 3D cities. Our key insight
+is that 3D city generation should be a composition of different types of neural
+fields: 1) various building instances, and 2) background stuff, such as roads
+and green lands. Specifically, we adopt the bird's eye view scene
+representation and employ a volumetric render for both instance-oriented and
+stuff-oriented neural fields. The generative hash grid and periodic positional
+embedding are tailored as scene parameterization to suit the distinct
+characteristics of building instances and background stuff. Furthermore, we
+contribute a suite of CityGen Datasets, including OSM and GoogleEarth, which
+comprises a vast amount of real-world city imagery to enhance the realism of
+the generated 3D cities both in their layouts and appearances. CityDreamer
+achieves state-of-the-art performance not only in generating realistic 3D
+cities but also in localized editing within the generated cities.",cs.CV,['cs.CV']
+"One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications",Mengyao Lyu · Yuhong Yang · Haiwen Hong · Hui Chen · Xuan Jin · Yuan He · Hui Xue · Jungong Han · Guiguang Ding,https://lyumengyao.github.io/projects/spm,https://arxiv.org/abs/2312.16145,,2312.16145.pdf,"One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications","The prevalent use of commercial and open-source diffusion models (DMs) for
+text-to-image generation prompts risk mitigation to prevent undesired
+behaviors. Existing concept erasing methods in academia are all based on full
+parameter or specification-based fine-tuning, from which we observe the
+following issues: 1) Generation alternation towards erosion: Parameter drift
+during target elimination causes alternations and potential deformations across
+all generations, even eroding other concepts at varying degrees, which is more
+evident with multi-concept erased; 2) Transfer inability & deployment
+inefficiency: Previous model-specific erasure impedes the flexible combination
+of concepts and the training-free transfer towards other models, resulting in
+linear cost growth as the deployment scenarios increase. To achieve
+non-invasive, precise, customizable, and transferable elimination, we ground
+our erasing framework on one-dimensional adapters to erase multiple concepts
+from most DMs at once across versatile erasing applications. The
+concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
+learn targeted erasing, and meantime the alteration and erosion phenomenon is
+effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
+obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
+specific re-tuning, enabling timely and efficient adaptation to diverse
+scenarios. During generation, our Facilitated Transport mechanism dynamically
+regulates the permeability of each SPM to respond to different input prompts,
+further minimizing the impact on other concepts. Quantitative and qualitative
+results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
+the superior erasing of SPM. Our code and pre-tuned SPMs are available on the
+project page https://lyumengyao.github.io/projects/spm.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention,Ju-Hyeon Nam · Nur Suriza Syazwany · Su Jung Kim · Sang-Chul Lee,https://skawngus1111.github.io/MADGNet_project/,https://arxiv.org/abs/2405.06284,,2405.06284.pdf,Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention,"Generalizability in deep neural networks plays a pivotal role in medical
+image segmentation. However, deep learning-based medical image analyses tend to
+overlook the importance of frequency variance, which is critical element for
+achieving a model that is both modality-agnostic and domain-generalizable.
+Additionally, various models fail to account for the potential information loss
+that can arise from multi-task learning under deep supervision, a factor that
+can impair the model representation ability. To address these challenges, we
+propose a Modality-agnostic Domain Generalizable Network (MADGNet) for medical
+image segmentation, which comprises two key components: a Multi-Frequency in
+Multi-Scale Attention (MFMSA) block and Ensemble Sub-Decoding Module (E-SDM).
+The MFMSA block refines the process of spatial feature extraction, particularly
+in capturing boundary features, by incorporating multi-frequency and
+multi-scale features, thereby offering informative cues for tissue outline and
+anatomical structures. Moreover, we propose E-SDM to mitigate information loss
+in multi-task learning with deep supervision, especially during substantial
+upsampling from low resolution. We evaluate the segmentation performance of
+MADGNet across six modalities and fifteen datasets. Through extensive
+experiments, we demonstrate that MADGNet consistently outperforms
+state-of-the-art models across various modalities, showcasing superior
+segmentation performance. This affirms MADGNet as a robust solution for medical
+image segmentation that excels in diverse imaging scenarios. Our MADGNet code
+is available in GitHub Link.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG']"
+CoG-DQA: Chain-of-Guiding Learning with Large Language Models for Diagram Question Answering,Shaowei Wang · Lingling Zhang · Longji Zhu · Tao Qin · Kim-Hui Yap · Xinyu Zhang · Jun Liu, ,https://arxiv.org/abs/2312.17269,,2312.17269.pdf,Conversational Question Answering with Reformulations over Knowledge Graph,"Conversational question answering (convQA) over knowledge graphs (KGs)
+involves answering multi-turn natural language questions about information
+contained in a KG. State-of-the-art methods of ConvQA often struggle with
+inexplicit question-answer pairs. These inputs are easy for human beings to
+understand given a conversation history, but hard for a machine to interpret,
+which can degrade ConvQA performance. To address this problem, we propose a
+reinforcement learning (RL) based model, CornNet, which utilizes question
+reformulations generated by large language models (LLMs) to improve ConvQA
+performance. CornNet adopts a teacher-student architecture where a teacher
+model learns question representations using human writing reformulations, and a
+student model to mimic the teacher model's output via reformulations generated
+by LLMs. The learned question representation is then used by an RL model to
+locate the correct answer in a KG. Extensive experimental results show that
+CornNet outperforms state-of-the-art convQA models.",cs.CL,"['cs.CL', 'cs.AI']"
+Lane2Seq: Towards Unified Lane Detection via Sequence Generation,Kunyang Zhou,https://zkyseu.github.io/lane2seq.github.io/,https://arxiv.org/abs/2402.17172,,2402.17172.pdf,Lane2Seq: Towards Unified Lane Detection via Sequence Generation,"In this paper, we present a novel sequence generation-based framework for
+lane detection, called Lane2Seq. It unifies various lane detection formats by
+casting lane detection as a sequence generation task. This is different from
+previous lane detection methods, which depend on well-designed task-specific
+head networks and corresponding loss functions. Lane2Seq only adopts a plain
+transformer-based encoder-decoder architecture with a simple cross-entropy
+loss. Additionally, we propose a new multi-format model tuning based on
+reinforcement learning to incorporate the task-specific knowledge into
+Lane2Seq. Experimental results demonstrate that such a simple sequence
+generation paradigm not only unifies lane detection but also achieves
+competitive performance on benchmarks. For example, Lane2Seq gets 97.95\% and
+97.42\% F1 score on Tusimple and LLAMAS datasets, establishing a new
+state-of-the-art result for two benchmarks.",cs.CV,['cs.CV']
+Scaling Up Dynamic 3D Human-Scene Interaction Modelling,Nan Jiang · Zhiyuan Zhang · Hongjie Li · Xiaoxuan Ma · Zan Wang · Yixin Chen · Tengyu Liu · Yixin Zhu · Siyuan Huang, ,https://arxiv.org/abs/2403.08629,,2403.08629.pdf,Scaling Up Dynamic Human-Scene Interaction Modeling,"Confronting the challenges of data scarcity and advanced motion synthesis in
+human-scene interaction modeling, we introduce the TRUMANS dataset alongside a
+novel HSI motion synthesis method. TRUMANS stands as the most comprehensive
+motion-captured HSI dataset currently available, encompassing over 15 hours of
+human interactions across 100 indoor scenes. It intricately captures whole-body
+human motions and part-level object dynamics, focusing on the realism of
+contact. This dataset is further scaled up by transforming physical
+environments into exact virtual models and applying extensive augmentations to
+appearance and motion for both humans and objects while maintaining interaction
+fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model
+that efficiently generates HSI sequences of any length, taking into account
+both scene context and intended actions. In experiments, our approach shows
+remarkable zero-shot generalizability on a range of 3D scene datasets (e.g.,
+PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic
+original motion-captured sequences, as confirmed by quantitative experiments
+and human studies.",cs.CV,['cs.CV']
+QUADify: Extracting Meshes with Pixel-level Details and Materials from Images,Maximilian Frühauf · Hayko Riemenschneider · Markus Gross · Christopher Schroers,https://maxfruehauf.com/publications/fruehauf2024quadify/drs_project_page/,,https://www.youtube.com/watch?v=n8M9c9yKGMk,,,,,nan
+BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning,Ruyang Liu · Chen Li · Yixiao Ge · Thomas H. Li · Ying Shan · Ge Li, ,http://export.arxiv.org/abs/2309.15785,,2309.15785.pdf,One For All: Video Conversation is Feasible Without Video Instruction Tuning,"The recent progress in Large Language Models (LLM) has spurred various
+advancements in image-language conversation agents, while how to build a
+proficient video-based dialogue system is still under exploration. Considering
+the extensive scale of LLM and visual backbone, minimal GPU memory is left for
+facilitating effective temporal modeling, which is crucial for comprehending
+and providing feedback on videos. To this end, we propose Branching Temporal
+Adapter (BT-Adapter), a novel method for extending image-language pretrained
+models into the video domain. Specifically, BT-Adapter serves as a plug-and-use
+temporal modeling branch alongside the pretrained visual encoder, which is
+tuned while keeping the backbone frozen. Just pretrained once, BT-Adapter can
+be seamlessly integrated into all image conversation models using this version
+of CLIP, enabling video conversations without the need for video instructions.
+Besides, we develop a unique asymmetric token masking strategy inside the
+branch with tailor-made training tasks for BT-Adapter, facilitating faster
+convergence and better results. Thanks to BT-Adapter, we are able to empower
+existing multimodal dialogue models with strong video understanding
+capabilities without incurring excessive GPU costs. Without bells and whistles,
+BT-Adapter achieves (1) state-of-the-art zero-shot results on various video
+tasks using thousands of fewer GPU hours. (2) better performance than current
+video chatbots without any video instruction tuning. (3) state-of-the-art
+results of video chatting using video instruction tuning, outperforming
+previous SOTAs by a large margin.",cs.CV,['cs.CV']
+Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection,Ting Lei · Shaofeng Yin · Yang Liu, ,https://arxiv.org/abs/2404.06194,,2404.06194.pdf,Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection,"Open-vocabulary human-object interaction (HOI) detection, which is concerned
+with the problem of detecting novel HOIs guided by natural language, is crucial
+for understanding human-centric scenes. However, prior zero-shot HOI detectors
+often employ the same levels of feature maps to model HOIs with varying
+distances, leading to suboptimal performance in scenes containing human-object
+pairs with a wide range of distances. In addition, these detectors primarily
+rely on category names and overlook the rich contextual information that
+language can provide, which is essential for capturing open vocabulary concepts
+that are typically rare and not well-represented by category names alone. In
+this paper, we introduce a novel end-to-end open vocabulary HOI detection
+framework with conditional multi-level decoding and fine-grained semantic
+enhancement (CMD-SE), harnessing the potential of Visual-Language Models
+(VLMs). Specifically, we propose to model human-object pairs with different
+distances with different levels of feature maps by incorporating a soft
+constraint during the bipartite matching process. Furthermore, by leveraging
+large language models (LLMs) such as GPT models, we exploit their extensive
+world knowledge to generate descriptions of human body part states for various
+interactions. Then we integrate the generalizable and fine-grained semantics of
+human body parts to improve interaction recognition. Experimental results on
+two datasets, SWIG-HOI and HICO-DET, demonstrate that our proposed method
+achieves state-of-the-art results in open vocabulary HOI detection. The code
+and models are available at https://github.com/ltttpku/CMD-SE-release.",cs.CV,['cs.CV']
+SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction,Zechuan Zhang · Zongxin Yang · Yi Yang,https://river-zhang.github.io/SIFU-projectpage/,https://arxiv.org/abs/2312.06704,,2312.06704.pdf,SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction,"Creating high-quality 3D models of clothed humans from single images for
+real-world applications is crucial. Despite recent advancements, accurately
+reconstructing humans in complex poses or with loose clothing from in-the-wild
+images, along with predicting textures for unseen areas, remains a significant
+challenge. A key limitation of previous methods is their insufficient prior
+guidance in transitioning from 2D to 3D and in texture prediction. In response,
+we introduce SIFU (Side-view Conditioned Implicit Function for Real-world
+Usable Clothed Human Reconstruction), a novel approach combining a Side-view
+Decoupling Transformer with a 3D Consistent Texture Refinement pipeline.SIFU
+employs a cross-attention mechanism within the transformer, using SMPL-X
+normals as queries to effectively decouple side-view features in the process of
+mapping 2D features to 3D. This method not only improves the precision of the
+3D models but also their robustness, especially when SMPL-X estimates are not
+perfect. Our texture refinement process leverages text-to-image diffusion-based
+prior to generate realistic and consistent textures for invisible views.
+Through extensive experiments, SIFU surpasses SOTA methods in both geometry and
+texture reconstruction, showcasing enhanced robustness in complex scenarios and
+achieving an unprecedented Chamfer and P2S measurement. Our approach extends to
+practical applications such as 3D printing and scene building, demonstrating
+its broad utility in real-world scenarios. Project page
+https://river-zhang.github.io/SIFU-projectpage/ .",cs.CV,['cs.CV']
+Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images,Chaoqin Huang · Aofan Jiang · Jinghao Feng · Ya Zhang · Xinchao Wang · Yanfeng Wang, ,https://arxiv.org/abs/2403.12570,,2403.12570.pdf,Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images,"Recent advancements in large-scale visual-language pre-trained models have
+led to significant progress in zero-/few-shot anomaly detection within natural
+image domains. However, the substantial domain divergence between natural and
+medical images limits the effectiveness of these methodologies in medical
+anomaly detection. This paper introduces a novel lightweight multi-level
+adaptation and comparison framework to repurpose the CLIP model for medical
+anomaly detection. Our approach integrates multiple residual adapters into the
+pre-trained visual encoder, enabling a stepwise enhancement of visual features
+across different levels. This multi-level adaptation is guided by multi-level,
+pixel-wise visual-language feature alignment loss functions, which recalibrate
+the model's focus from object semantics in natural imagery to anomaly
+identification in medical images. The adapted features exhibit improved
+generalization across various medical data types, even in zero-shot scenarios
+where the model encounters unseen medical modalities and anatomical regions
+during training. Our experiments on medical anomaly detection benchmarks
+demonstrate that our method significantly surpasses current state-of-the-art
+models, with an average AUC improvement of 6.24% and 7.33% for anomaly
+classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot
+and few-shot settings, respectively. Source code is available at:
+https://github.com/MediaBrain-SJTU/MVFA-AD",cs.CV,['cs.CV']
+Panacea: Panoramic and Controllable Video Generation for Autonomous Driving,Yuqing Wen · Yucheng Zhao · Yingfei Liu · Fan Jia · Yanhui Wang · Chong Luo · Chi Zhang · Tiancai Wang · Xiaoyan Sun · Xiangyu Zhang, ,https://arxiv.org/abs/2311.16813,,2311.16813.pdf,Panacea: Panoramic and Controllable Video Generation for Autonomous Driving,"The field of autonomous driving increasingly demands high-quality annotated
+training data. In this paper, we propose Panacea, an innovative approach to
+generate panoramic and controllable videos in driving scenarios, capable of
+yielding an unlimited numbers of diverse, annotated samples pivotal for
+autonomous driving advancements. Panacea addresses two critical challenges:
+'Consistency' and 'Controllability.' Consistency ensures temporal and
+cross-view coherence, while Controllability ensures the alignment of generated
+content with corresponding annotations. Our approach integrates a novel 4D
+attention and a two-stage generation pipeline to maintain coherence,
+supplemented by the ControlNet framework for meticulous control by the
+Bird's-Eye-View (BEV) layouts. Extensive qualitative and quantitative
+evaluations of Panacea on the nuScenes dataset prove its effectiveness in
+generating high-quality multi-view driving-scene videos. This work notably
+propels the field of autonomous driving by effectively augmenting the training
+dataset used for advanced BEV perception techniques.",cs.CV,['cs.CV']
+No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation,Xiangyang Zhu · Renrui Zhang · Bowei He · Ziyu Guo · Jiaming Liu · Han Xiao · Chaoyou Fu · Hao Dong · Peng Gao, ,https://arxiv.org/abs/2404.04050,,2404.04050.pdf,No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation,"To reduce the reliance on large-scale datasets, recent works in 3D
+segmentation resort to few-shot learning. Current 3D few-shot segmentation
+methods first pre-train models on 'seen' classes, and then evaluate their
+generalization performance on 'unseen' classes. However, the prior pre-training
+stage not only introduces excessive time overhead but also incurs a significant
+domain gap on 'unseen' classes. To tackle these issues, we propose a
+Non-parametric Network for few-shot 3D Segmentation, Seg-NN, and its Parametric
+variant, Seg-PN. Without training, Seg-NN extracts dense representations by
+hand-crafted filters and achieves comparable performance to existing parametric
+models. Due to the elimination of pre-training, Seg-NN can alleviate the domain
+gap issue and save a substantial amount of time. Based on Seg-NN, Seg-PN only
+requires training a lightweight QUEry-Support Transferring (QUEST) module,
+which enhances the interaction between the support set and query set.
+Experiments suggest that Seg-PN outperforms previous state-of-the-art method by
++4.19% and +7.71% mIoU on S3DIS and ScanNet datasets respectively, while
+reducing training time by -90%, indicating its effectiveness and efficiency.",cs.CV,['cs.CV']
+MoST: Motion Style Transformer between Diverse Action Contents,Boeun Kim · Jungho Kim · Hyung Jin Chang · Jin Young Choi,https://boeun-kim.github.io/page-MoST/,https://arxiv.org/abs/2403.06225,,2403.06225.pdf,MoST: Motion Style Transformer between Diverse Action Contents,"While existing motion style transfer methods are effective between two
+motions with identical content, their performance significantly diminishes when
+transferring style between motions with different contents. This challenge lies
+in the lack of clear separation between content and style of a motion. To
+tackle this challenge, we propose a novel motion style transformer that
+effectively disentangles style from content and generates a plausible motion
+with transferred style from a source motion. Our distinctive approach to
+achieving the goal of disentanglement is twofold: (1) a new architecture for
+motion style transformer with `part-attentive style modulator across body
+parts' and `Siamese encoders that encode style and content features
+separately'; (2) style disentanglement loss. Our method outperforms existing
+methods and demonstrates exceptionally high quality, particularly in motion
+pairs with different contents, without the need for heuristic post-processing.
+Codes are available at https://github.com/Boeun-Kim/MoST.",cs.CV,"['cs.CV', 'cs.AI']"
+Learned Lossless Image Compression based on Bit Plane Slicing,Zhe Zhang · Huairui Wang · Zhenzhong Chen · Shan Liu, ,https://arxiv.org/abs/2308.13287,,2308.13287.pdf,Efficient Learned Lossless JPEG Recompression,"JPEG is one of the most popular image compression methods. It is beneficial
+to compress those existing JPEG files without introducing additional
+distortion. In this paper, we propose a deep learning based method to further
+compress JPEG images losslessly. Specifically, we propose a Multi-Level
+Parallel Conditional Modeling (ML-PCM) architecture, which enables parallel
+decoding in different granularities. First, luma and chroma are processed
+independently to allow parallel coding. Second, we propose pipeline parallel
+context model (PPCM) and compressed checkerboard context model (CCCM) for the
+effective conditional modeling and efficient decoding within luma and chroma
+components. Our method has much lower latency while achieves better compression
+ratio compared with previous SOTA. After proper software optimization, we can
+obtain a good throughput of 57 FPS for 1080P images on NVIDIA T4 GPU.
+Furthermore, combined with quantization, our approach can also act as a lossy
+JPEG codec which has obvious advantage over SOTA lossy compression methods in
+high bit rate (bpp$>0.9$).",eess.IV,['eess.IV']
+Structure-Aware Sparse-View X-ray 3D Reconstruction,Yuanhao Cai · Jiahao Wang · Alan L. Yuille · Zongwei Zhou · Angtian Wang,https://github.com/caiyuanhao1998/SAX-NeRF,https://arxiv.org/abs/2311.10959,,2311.10959.pdf,Structure-Aware Sparse-View X-ray 3D Reconstruction,"X-ray, known for its ability to reveal internal structures of objects, is
+expected to provide richer information for 3D reconstruction than visible
+light. Yet, existing neural radiance fields (NeRF) algorithms overlook this
+important nature of X-ray, leading to their limitations in capturing structural
+contents of imaged objects. In this paper, we propose a framework,
+Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view
+X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer
+(Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal
+structures of objects in 3D space by modeling the dependencies within each line
+segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray
+sampling strategy to extract contextual and geometric information in 2D
+projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray
+applications. Experiments on X3D show that SAX-NeRF surpasses previous
+NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT
+reconstruction. Code, models, and data are released at
+https://github.com/caiyuanhao1998/SAX-NeRF",eess.IV,"['eess.IV', 'cs.CV']"
+Point Cloud Pre-training with Diffusion Models,xiao zheng · Xiaoshui Huang · Guofeng Mei · Zhaoyang Lyu · Yuenan Hou · Wanli Ouyang · Bo Dai · Yongshun Gong, ,https://arxiv.org/abs/2311.14960,,2311.14960.pdf,Point Cloud Pre-training with Diffusion Models,"Pre-training a model and then fine-tuning it on downstream tasks has
+demonstrated significant success in the 2D image and NLP domains. However, due
+to the unordered and non-uniform density characteristics of point clouds, it is
+non-trivial to explore the prior knowledge of point clouds and pre-train a
+point cloud backbone. In this paper, we propose a novel pre-training method
+called Point cloud Diffusion pre-training (PointDif). We consider the point
+cloud pre-training task as a conditional point-to-point generation problem and
+introduce a conditional point generator. This generator aggregates the features
+extracted by the backbone and employs them as the condition to guide the
+point-to-point recovery from the noisy point cloud, thereby assisting the
+backbone in capturing both local and global geometric priors as well as the
+global point density distribution of the object. We also present a recurrent
+uniform sampling optimization strategy, which enables the model to uniformly
+recover from various noise levels and learn from balanced supervision. Our
+PointDif achieves substantial improvement across various real-world datasets
+for diverse downstream tasks such as classification, segmentation and
+detection. Specifically, PointDif attains 70.0% mIoU on S3DIS Area 5 for the
+segmentation task and achieves an average improvement of 2.4% on ScanObjectNN
+for the classification task compared to TAP. Furthermore, our pre-training
+framework can be flexibly applied to diverse point cloud backbones and bring
+considerable gains.",cs.CV,['cs.CV']
+DiffLoc: Diffusion Model for Outdoor LiDAR Localization,Wen Li · Yuyang Yang · Shangshu Yu · Guosheng Hu · Chenglu Wen · Ming Cheng · Cheng Wang, ,,https://www.youtube.com/watch?v=sSW9nHQR0nc,,,,,nan
+Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation,Shenshen Bu · Taiji Li · Zhiming Dai · Yuedong Yang,https://github.com/hnjzbss/EKAGen,https://arxiv.org/abs/2311.00399,,2311.00399.pdf,Enhanced Knowledge Injection for Radiology Report Generation,"Automatic generation of radiology reports holds crucial clinical value, as it
+can alleviate substantial workload on radiologists and remind less experienced
+ones of potential anomalies. Despite the remarkable performance of various
+image captioning methods in the natural image field, generating accurate
+reports for medical images still faces challenges, i.e., disparities in visual
+and textual data, and lack of accurate domain knowledge. To address these
+issues, we propose an enhanced knowledge injection framework, which utilizes
+two branches to extract different types of knowledge. The Weighted Concept
+Knowledge (WCK) branch is responsible for introducing clinical medical concepts
+weighted by TF-IDF scores. The Multimodal Retrieval Knowledge (MRK) branch
+extracts triplets from similar reports, emphasizing crucial clinical
+information related to entity positions and existence. By integrating this
+finer-grained and well-structured knowledge with the current image, we are able
+to leverage the multi-source knowledge gain to ultimately facilitate more
+accurate report generation. Extensive experiments have been conducted on two
+public benchmarks, demonstrating that our method achieves superior performance
+over other state-of-the-art methods. Ablation studies further validate the
+effectiveness of two extracted knowledge sources.",cs.CV,"['cs.CV', 'cs.CL']"
+Enhancing Multimodal Cooperation via Sample-level Modality Valuation,Yake Wei · Ruoxuan Feng · Zihe Wang · Di Hu, ,https://arxiv.org/html/2309.06255v3,,2309.06255v3.pdf,Enhancing Multimodal Cooperation via Fine-grained Modality Valuation,"One primary topic of multimodal learning is to jointly incorporate
+heterogeneous information from different modalities. However, most models often
+suffer from unsatisfactory multimodal cooperation, which cannot jointly utilize
+all modalities well. Some methods are proposed to identify and enhance the
+worse learnt modality, but they are often hard to provide the fine-grained
+observation of multimodal cooperation at sample-level with theoretical support.
+Hence, it is essential to reasonably observe and improve the fine-grained
+cooperation between modalities, especially when facing realistic scenarios
+where the modality discrepancy could vary across different samples. To this
+end, we introduce a sample-level modality valuation metric to evaluate the
+contribution of each modality for each sample. Via modality valuation, we
+observe that modality discrepancy indeed could be different at sample-level,
+beyond the global contribution discrepancy at dataset-level. We further analyze
+this issue and improve cooperation between modalities at sample-level by
+enhancing the discriminative ability of low-contributing modalities in a
+targeted manner. Overall, our methods reasonably observe the fine-grained
+uni-modal contribution and achieve considerable improvement. The source code
+and dataset are available at
+\url{https://github.com/GeWu-Lab/Valuate-and-Enhance-Multimodal-Cooperation}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+A Pedestrian is Worth One Prompt: Towards Language Guidance Person Re-Identification,Zexian Yang · Dayan Wu · Chenming Wu · Zheng Lin · JingziGU · Weiping Wang, ,,https://ieeexplore.ieee.org/document/10301577,,,,,nan
+Data-Efficient Multimodal Fusion on a Single GPU,Noël Vouitsis · Zhaoyan Liu · Satya Krishna Gorti · Valentin Villecroze · Jesse C. Cresswell · Guangwei Yu · Gabriel Loaiza-Ganem · Maksims Volkovs, ,https://arxiv.org/abs/2312.10144,,2312.10144.pdf,Data-Efficient Multimodal Fusion on a Single GPU,"The goal of multimodal alignment is to learn a single latent space that is
+shared between multimodal inputs. The most powerful models in this space have
+been trained using massive datasets of paired inputs and large-scale
+computational resources, making them prohibitively expensive to train in many
+practical scenarios. We surmise that existing unimodal encoders pre-trained on
+large amounts of unimodal data should provide an effective bootstrap to create
+multimodal models from unimodal ones at much lower costs. We therefore propose
+FuseMix, a multimodal augmentation scheme that operates on the latent spaces of
+arbitrary pre-trained unimodal encoders. Using FuseMix for multimodal
+alignment, we achieve competitive performance -- and in certain cases
+outperform state-of-the art methods -- in both image-text and audio-text
+retrieval, with orders of magnitude less compute and data: for example, we
+outperform CLIP on the Flickr30K text-to-image retrieval task with $\sim \!
+600\times$ fewer GPU days and $\sim \! 80\times$ fewer image-text pairs.
+Additionally, we show how our method can be applied to convert pre-trained
+text-to-image generative models into audio-to-image ones. Code is available at:
+https://github.com/layer6ai-labs/fusemix.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+SimDA: Simple Diffusion Adapter for Efficient Video Generation,Zhen Xing · Qi Dai · Han Hu · Zuxuan Wu · Yu-Gang Jiang, ,https://arxiv.org/abs/2308.09710,,2308.09710.pdf,SimDA: Simple Diffusion Adapter for Efficient Video Generation,"The recent wave of AI-generated content has witnessed the great development
+and success of Text-to-Image (T2I) technologies. By contrast, Text-to-Video
+(T2V) still falls short of expectations though attracting increasing interests.
+Existing works either train from scratch or adapt large T2I model to videos,
+both of which are computation and resource expensive. In this work, we propose
+a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B
+parameters of a strong T2I model, adapting it to video generation in a
+parameter-efficient way. In particular, we turn the T2I model for T2V by
+designing light-weight spatial and temporal adapters for transfer learning.
+Besides, we change the original spatial attention to the proposed Latent-Shift
+Attention (LSA) for temporal consistency. With similar model architecture, we
+further train a video super-resolution model to generate high-definition
+(1024x1024) videos. In addition to T2V generation in the wild, SimDA could also
+be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our
+method could minimize the training effort with extremely few tunable parameters
+for model adaptation.",cs.CV,"['cs.CV', 'cs.AI']"
+Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation,Tianshui Chen · Jianman Lin · Zhijing Yang · Chunmei Qing · Liang Lin, ,https://arxiv.org/abs/2401.11085,,2401.11085.pdf,Adaptive Global-Local Representation Learning and Selection for Cross-Domain Facial Expression Recognition,"Domain shift poses a significant challenge in Cross-Domain Facial Expression
+Recognition (CD-FER) due to the distribution variation across different
+domains. Current works mainly focus on learning domain-invariant features
+through global feature adaptation, while neglecting the transferability of
+local features. Additionally, these methods lack discriminative supervision
+during training on target datasets, resulting in deteriorated feature
+representation in target domain. To address these limitations, we propose an
+Adaptive Global-Local Representation Learning and Selection (AGLRLS) framework.
+The framework incorporates global-local adversarial adaptation and
+semantic-aware pseudo label generation to enhance the learning of
+domain-invariant and discriminative feature during training. Meanwhile, a
+global-local prediction consistency learning is introduced to improve
+classification results during inference. Specifically, the framework consists
+of separate global-local adversarial learning modules that learn
+domain-invariant global and local features independently. We also design a
+semantic-aware pseudo label generation module, which computes semantic labels
+based on global and local features. Moreover, a novel dynamic threshold
+strategy is employed to learn the optimal thresholds by leveraging independent
+prediction of global and local features, ensuring filtering out the unreliable
+pseudo labels while retaining reliable ones. These labels are utilized for
+model optimization through the adversarial learning process in an end-to-end
+manner. During inference, a global-local prediction consistency module is
+developed to automatically learn an optimal result from multiple predictions.
+We conduct comprehensive experiments and analysis based on a fair evaluation
+benchmark. The results demonstrate that the proposed framework outperforms the
+current competing methods by a substantial margin.",cs.CV,"['cs.CV', 'cs.AI']"
+Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow,Hanyu Zhou · Yi Chang · Zhiwei Shi,https://hyzhouboy.github.io/,https://arxiv.org/abs/2403.07432,,2403.07432.pdf,Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow,"Single RGB or LiDAR is the mainstream sensor for the challenging scene flow,
+which relies heavily on visual features to match motion features. Compared with
+single modality, existing methods adopt a fusion strategy to directly fuse the
+cross-modal complementary knowledge in motion space. However, these direct
+fusion methods may suffer the modality gap due to the visual intrinsic
+heterogeneous nature between RGB and LiDAR, thus deteriorating motion features.
+We discover that event has the homogeneous nature with RGB and LiDAR in both
+visual and motion spaces. In this work, we bring the event as a bridge between
+RGB and LiDAR, and propose a novel hierarchical visual-motion fusion framework
+for scene flow, which explores a homogeneous space to fuse the cross-modal
+complementary knowledge for physical interpretation. In visual fusion, we
+discover that event has a complementarity (relative v.s. absolute) in luminance
+space with RGB for high dynamic imaging, and has a complementarity (local
+boundary v.s. global shape) in scene structure space with LiDAR for structure
+integrity. In motion fusion, we figure out that RGB, event and LiDAR are
+complementary (spatial-dense, temporal-dense v.s. spatiotemporal-sparse) to
+each other in correlation space, which motivates us to fuse their motion
+correlations for motion continuity. The proposed hierarchical fusion can
+explicitly fuse the multimodal knowledge to progressively improve scene flow
+from visual space to motion space. Extensive experiments have been performed to
+verify the superiority of the proposed method.",cs.CV,['cs.CV']
+Efficient Vision-Language Pre-training by Cluster Masking,Zihao Wei · Zixuan Pan · Andrew Owens, ,https://arxiv.org/abs/2405.08815,,2405.08815.pdf,Efficient Vision-Language Pre-training by Cluster Masking,"We propose a simple strategy for masking image patches during visual-language
+contrastive learning that improves the quality of the learned representations
+and the training speed. During each iteration of training, we randomly mask
+clusters of visually similar image patches, as measured by their raw pixel
+intensities. This provides an extra learning signal, beyond the contrastive
+training itself, since it forces a model to predict words for masked visual
+structures solely from context. It also speeds up training by reducing the
+amount of data used in each image. We evaluate the effectiveness of our model
+by pre-training on a number of benchmarks, finding that it outperforms other
+masking strategies, such as FLIP, on the quality of the learned representation.",cs.CV,['cs.CV']
+Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction,Cheng Sun · Wei-En Tai · Yu-Lin Shih · Kuan-Wei Chen · Yong-Jing Syu · Kent Selwyn The · Yu-Chiang Frank Wang · Hwann-Tzong Chen, ,https://arxiv.org/abs/2311.18695v1,,2311.18695v1.pdf,Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction,"State-of-the-art single-view 360-degree room layout reconstruction methods
+formulate the problem as a high-level 1D (per-column) regression task. On the
+other hand, traditional low-level 2D layout segmentation is simpler to learn
+and can represent occluded regions, but it requires complex post-processing for
+the targeting layout polygon and sacrifices accuracy. We present Seg2Reg to
+render 1D layout depth regression from the 2D segmentation map in a
+differentiable and occlusion-aware way, marrying the merits of both sides.
+Specifically, our model predicts floor-plan density for the input
+equirectangular 360-degree image. Formulating the 2D layout representation as a
+density field enables us to employ `flattened' volume rendering to form 1D
+layout depth regression. In addition, we propose a novel 3D warping
+augmentation on layout to improve generalization. Finally, we re-implement
+recent room layout reconstruction methods into our codebase for benchmarking
+and explore modern backbones and training techniques to serve as the strong
+baseline. Our model significantly outperforms previous arts. The code will be
+made available upon publication.",cs.CV,"['cs.CV', 'cs.LG']"
+Patch2Self2: Self-supervised Denoising on Coresets via Matrix Sketching,Shreyas Fadnavis · Agniva Chowdhury · Joshua Batson · Petros Drineas · Eleftherios Garyfallidis, ,,,,,,,nan
+X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition,Shuofeng Sun · Yongming Rao · Jiwen Lu · Haibin Yan,https://github.com/sunshuofeng/X-3D,https://arxiv.org/abs/2404.15010,,2404.15010.pdf,X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition,"Numerous prior studies predominantly emphasize constructing relation vectors
+for individual neighborhood points and generating dynamic kernels for each
+vector and embedding these into high-dimensional spaces to capture implicit
+local structures. However, we contend that such implicit high-dimensional
+structure modeling approch inadequately represents the local geometric
+structure of point clouds due to the absence of explicit structural
+information. Hence, we introduce X-3D, an explicit 3D structure modeling
+approach. X-3D functions by capturing the explicit local structural information
+within the input 3D space and employing it to produce dynamic kernels with
+shared weights for all neighborhood points within the current local region.
+This modeling approach introduces effective geometric prior and significantly
+diminishes the disparity between the local structure of the embedding space and
+the original input point cloud, thereby improving the extraction of local
+features. Experiments show that our method can be used on a variety of methods
+and achieves state-of-the-art performance on segmentation, classification,
+detection tasks with lower extra computational cost, such as \textbf{90.7\%} on
+ScanObjectNN for classification, \textbf{79.2\%} on S3DIS 6 fold and
+\textbf{74.3\%} on S3DIS Area 5 for segmentation, \textbf{76.3\%} on ScanNetV2
+for segmentation and \textbf{64.5\%} mAP , \textbf{46.9\%} mAP on SUN RGB-D and
+\textbf{69.0\%} mAP , \textbf{51.1\%} mAP on ScanNetV2 . Our code is available
+at
+\href{https://github.com/sunshuofeng/X-3D}{https://github.com/sunshuofeng/X-3D}.",cs.CV,['cs.CV']
+NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation,Sicheng Li · Hao Li · Yiyi Liao · Lu Yu,https://jasonlsc.github.io/nerfcodec_homepage/,https://arxiv.org/abs/2404.02185,,2404.02185.pdf,NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation,"The emergence of Neural Radiance Fields (NeRF) has greatly impacted 3D scene
+modeling and novel-view synthesis. As a kind of visual media for 3D scene
+representation, compression with high rate-distortion performance is an eternal
+target. Motivated by advances in neural compression and neural field
+representation, we propose NeRFCodec, an end-to-end NeRF compression framework
+that integrates non-linear transform, quantization, and entropy coding for
+memory-efficient scene representation. Since training a non-linear transform
+directly on a large scale of NeRF feature planes is impractical, we discover
+that pre-trained neural 2D image codec can be utilized for compressing the
+features when adding content-specific parameters. Specifically, we reuse neural
+2D image codec but modify its encoder and decoder heads, while keeping the
+other parts of the pre-trained decoder frozen. This allows us to train the full
+pipeline via supervision of rendering loss and entropy loss, yielding the
+rate-distortion balance by updating the content-specific parameters. At test
+time, the bitstreams containing latent code, feature decoder head, and other
+side information are transmitted for communication. Experimental results
+demonstrate our method outperforms existing NeRF compression methods, enabling
+high-quality novel view synthesis with a memory budget of 0.5 MB.",cs.CV,"['cs.CV', 'cs.GR', 'eess.IV']"
+Desigen: A Pipeline for Controllable Design Template Generation,Haohan Weng · Danqing Huang · YU QIAO · Hu Zheng · Chin-Yew Lin · Tong Zhang · C. L. Philip Chen, ,https://arxiv.org/html/2403.09093v1,,2403.09093v1.pdf,Desigen: A Pipeline for Controllable Design Template Generation,"Templates serve as a good starting point to implement a design (e.g., banner,
+slide) but it takes great effort from designers to manually create. In this
+paper, we present Desigen, an automatic template creation pipeline which
+generates background images as well as harmonious layout elements over the
+background. Different from natural images, a background image should preserve
+enough non-salient space for the overlaying layout elements. To equip existing
+advanced diffusion-based models with stronger spatial control, we propose two
+simple but effective techniques to constrain the saliency distribution and
+reduce the attention weight in desired regions during the background generation
+process. Then conditioned on the background, we synthesize the layout with a
+Transformer-based autoregressive generator. To achieve a more harmonious
+composition, we propose an iterative inference strategy to adjust the
+synthesized background and layout in multiple rounds. We constructed a design
+dataset with more than 40k advertisement banners to verify our approach.
+Extensive experiments demonstrate that the proposed pipeline generates
+high-quality templates comparable to human designers. More than a single-page
+design, we further show an application of presentation generation that outputs
+a set of theme-consistent slides. The data and code are available at
+https://whaohan.github.io/desigen.",cs.CV,['cs.CV']
+"Sparse views, Near light: A practical paradigm for uncalibrated point-light photometric stereo",Mohammed Brahimi · Bjoern Haefner · Zhenzhang Ye · Bastian Goldluecke · Daniel Cremers, ,https://arxiv.org/abs/2404.00098,,2404.00098.pdf,"Sparse Views, Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo","Neural approaches have shown a significant progress on camera-based
+reconstruction. But they require either a fairly dense sampling of the viewing
+sphere, or pre-training on an existing dataset, thereby limiting their
+generalizability. In contrast, photometric stereo (PS) approaches have shown
+great potential for achieving high-quality reconstruction under sparse
+viewpoints. Yet, they are impractical because they typically require tedious
+laboratory conditions, are restricted to dark rooms, and often multi-staged,
+making them subject to accumulated errors. To address these shortcomings, we
+propose an end-to-end uncalibrated multi-view PS framework for reconstructing
+high-resolution shapes acquired from sparse viewpoints in a real-world
+environment. We relax the dark room assumption, and allow a combination of
+static ambient lighting and dynamic near LED lighting, thereby enabling easy
+data capture outside the lab. Experimental validation confirms that it
+outperforms existing baseline approaches in the regime of sparse viewpoints by
+a large margin. This allows to bring high-accuracy 3D reconstruction from the
+dark room to the real world, while maintaining a reasonable data capture
+complexity.",cs.CV,['cs.CV']
+LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example,Soyeon Yoon · Kwan Yun · Kwanggyoon Seo · Sihun Cha · Jung Eun Yoo · Junyong Noh,https://kwanyun.github.io/lego/,https://arxiv.org/abs/2403.15227,,2403.15227.pdf,LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example,"Recent advances in 3D face stylization have made significant strides in few
+to zero-shot settings. However, the degree of stylization achieved by existing
+methods is often not sufficient for practical applications because they are
+mostly based on statistical 3D Morphable Models (3DMM) with limited variations.
+To this end, we propose a method that can produce a highly stylized 3D face
+model with desired topology. Our methods train a surface deformation network
+with 3DMM and translate its domain to the target style using a paired exemplar.
+The network achieves stylization of the 3D face mesh by mimicking the style of
+the target using a differentiable renderer and directional CLIP losses.
+Additionally, during the inference process, we utilize a Mesh Agnostic Encoder
+(MAGE) that takes deformation target, a mesh of diverse topologies as input to
+the stylization process and encodes its shape into our latent space. The
+resulting stylized face model can be animated by commonly used 3DMM blend
+shapes. A set of quantitative and qualitative evaluations demonstrate that our
+method can produce highly stylized face meshes according to a given style and
+output them in a desired topology. We also demonstrate example applications of
+our method including image-based stylized avatar generation, linear
+interpolation of geometric styles, and facial animation of stylized avatars.",cs.CV,"['cs.CV', 'cs.GR', '68T45', 'I.4.9']"
+Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models,Chang Liu · Haoning Wu · Yujie Zhong · Xiaoyun Zhang · Yanfeng Wang · Weidi Xie,https://haoningwu3639.github.io/StoryGen_Webpage/,https://ar5iv.labs.arxiv.org/html/2312.03884,,2312.03884.pdf,WonderJourney: Going from Anywhere to Everywhere,"We introduce WonderJourney, a modularized framework for perpetual 3D scene
+generation. Unlike prior work on view generation that focuses on a single type
+of scenes, we start at any user-provided location (by a text description or an
+image) and generate a journey through a long sequence of diverse yet coherently
+connected 3D scenes. We leverage an LLM to generate textual descriptions of the
+scenes in this journey, a text-driven point cloud generation pipeline to make a
+compelling and coherent sequence of 3D scenes, and a large VLM to verify the
+generated scenes. We show compelling, diverse visual results across various
+scene types and styles, forming imaginary ""wonderjourneys"". Project website:
+https://kovenyu.com/WonderJourney/",cs.CV,"['cs.CV', 'cs.GR']"
+CSTA: CNN-based Spatiotemporal Attention for Video Summarization,Jaewon Son · Jaehun Park · Kwangsu Kim,https://github.com/thswodnjs3/CSTA,https://arxiv.org/abs/2405.11905,,2405.11905.pdf,CSTA: CNN-based Spatiotemporal Attention for Video Summarization,"Video summarization aims to generate a concise representation of a video,
+capturing its essential content and key moments while reducing its overall
+length. Although several methods employ attention mechanisms to handle
+long-term dependencies, they often fail to capture the visual significance
+inherent in frames. To address this limitation, we propose a CNN-based
+SpatioTemporal Attention (CSTA) method that stacks each feature of frames from
+a single video to form image-like frame representations and applies 2D CNN to
+these frame features. Our methodology relies on CNN to comprehend the inter and
+intra-frame relations and to find crucial attributes in videos by exploiting
+its ability to learn absolute positions within images. In contrast to previous
+work compromising efficiency by designing additional modules to focus on
+spatial importance, CSTA requires minimal computational overhead as it uses CNN
+as a sliding window. Extensive experiments on two benchmark datasets (SumMe and
+TVSum) demonstrate that our proposed approach achieves state-of-the-art
+performance with fewer MACs compared to previous methods. Codes are available
+at https://github.com/thswodnjs3/CSTA.",cs.CV,['cs.CV']
+LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP,Yunshi HUANG · Fereshteh Shakeri · Jose Dolz · Malik Boudiaf · Houda Bahig · Ismail Ben Ayed, ,https://arxiv.org/abs/2404.02285,,2404.02285.pdf,LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP,"In a recent, strongly emergent literature on few-shot CLIP adaptation, Linear
+Probe (LP) has been often reported as a weak baseline. This has motivated
+intensive research building convoluted prompt learning or feature adaptation
+strategies. In this work, we propose and examine from convex-optimization
+perspectives a generalization of the standard LP baseline, in which the linear
+classifier weights are learnable functions of the text embedding, with
+class-wise multipliers blending image and text knowledge. As our objective
+function depends on two types of variables, i.e., the class visual prototypes
+and the learnable blending parameters, we propose a computationally efficient
+block coordinate Majorize-Minimize (MM) descent algorithm. In our full-batch MM
+optimizer, which we coin LP++, step sizes are implicit, unlike standard
+gradient descent practices where learning rates are intensively searched over
+validation sets. By examining the mathematical properties of our loss (e.g.,
+Lipschitz gradient continuity), we build majorizing functions yielding
+data-driven learning rates and derive approximations of the loss's minima,
+which provide data-informed initialization of the variables. Our image-language
+objective function, along with these non-trivial optimization insights and
+ingredients, yields, surprisingly, highly competitive few-shot CLIP
+performances. Furthermore, LP++ operates in black-box, relaxes intensive
+validation searches for the optimization hyper-parameters, and runs
+orders-of-magnitudes faster than state-of-the-art few-shot CLIP adaptation
+methods. Our code is available at:
+\url{https://github.com/FereshteShakeri/FewShot-CLIP-Strong-Baseline.git}.",cs.CV,['cs.CV']
+EarthLoc: Astronaut Photography Localization by Indexing Earth from Space,Gabriele Berton · Alex Stoken · Barbara Caputo · Carlo Masone,https://github.com/gmberton/EarthLoc,https://arxiv.org/abs/2403.06758,,2403.06758.pdf,EarthLoc: Astronaut Photography Localization by Indexing Earth from Space,"Astronaut photography, spanning six decades of human spaceflight, presents a
+unique Earth observations dataset with immense value for both scientific
+research and disaster response. Despite its significance, accurately localizing
+the geographical extent of these images, crucial for effective utilization,
+poses substantial challenges. Current manual localization efforts are
+time-consuming, motivating the need for automated solutions. We propose a novel
+approach - leveraging image retrieval - to address this challenge efficiently.
+We introduce innovative training techniques, including Year-Wise Data
+Augmentation and a Neutral-Aware Multi-Similarity Loss, which contribute to the
+development of a high-performance model, EarthLoc. We develop six evaluation
+datasets and perform a comprehensive benchmark comparing EarthLoc to existing
+methods, showcasing its superior efficiency and accuracy. Our approach marks a
+significant advancement in automating the localization of astronaut
+photography, which will help bridge a critical gap in Earth observations data.
+Code and datasets are available at https://github.com/gmberton/EarthLoc",cs.CV,['cs.CV']
+Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis,Jiawen Li · Yuxuan Chen · Hongbo Chu · Sun Qiehe · Tian Guan · Anjia Han · Yonghong He, ,https://arxiv.org/abs/2403.07719,,2403.07719.pdf,Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis,"Histopathological whole slide images (WSIs) classification has become a
+foundation task in medical microscopic imaging processing. Prevailing
+approaches involve learning WSIs as instance-bag representations, emphasizing
+significant instances but struggling to capture the interactions between
+instances. Additionally, conventional graph representation methods utilize
+explicit spatial positions to construct topological structures but restrict the
+flexible interaction capabilities between instances at arbitrary locations,
+particularly when spatially distant. In response, we propose a novel dynamic
+graph representation algorithm that conceptualizes WSIs as a form of the
+knowledge graph structure. Specifically, we dynamically construct neighbors and
+directed edge embeddings based on the head and tail relationships between
+instances. Then, we devise a knowledge-aware attention mechanism that can
+update the head node features by learning the joint attention score of each
+neighbor and edge. Finally, we obtain a graph-level embedding through the
+global pooling process of the updated head, serving as an implicit
+representation for the WSI classification. Our end-to-end graph representation
+learning approach has outperformed the state-of-the-art WSI analysis methods on
+three TCGA benchmark datasets and in-house test sets. Our code is available at
+https://github.com/WonderLandxD/WiKG.",cs.CV,['cs.CV']
+Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary,Leheng Zhang · Yawei Li · Xingyu Zhou · Xiaorui Zhao · Shuhang Gu, ,https://arxiv.org/abs/2401.08209,,2401.08209.pdf,Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary,"Single Image Super-Resolution is a classic computer vision problem that
+involves estimating high-resolution (HR) images from low-resolution (LR) ones.
+Although deep neural networks (DNNs), especially Transformers for
+super-resolution, have seen significant advancements in recent years,
+challenges still remain, particularly in limited receptive field caused by
+window-based self-attention. To address these issues, we introduce a group of
+auxiliary Adaptive Token Dictionary to SR Transformer and establish an ATD-SR
+method. The introduced token dictionary could learn prior information from
+training data and adapt the learned prior to specific testing image through an
+adaptive refinement step. The refinement strategy could not only provide global
+information to all input tokens but also group image tokens into categories.
+Based on category partitions, we further propose a category-based
+self-attention mechanism designed to leverage distant but similar tokens for
+enhancing input features. The experimental results show that our method
+achieves the best performance on various single image super-resolution
+benchmarks.",cs.CV,['cs.CV']
+Learning to Select Views for Efficient Multi-View Understanding,Yunzhong Hou · Stephen Gould · Liang Zheng, ,,https://openreview.net/forum?id=mzWQ2hOKNX,,,,,nan
+MedBN: Robust Test-Time Adaptation against Malicious Test Samples,Hyejin Park · Jeongyeon Hwang · Sunung Mun · Sangdon Park · Jungseul Ok,http://hyejin-s.github.io/medbn,https://arxiv.org/abs/2403.19326,,2403.19326.pdf,MedBN: Robust Test-Time Adaptation against Malicious Test Samples,"Test-time adaptation (TTA) has emerged as a promising solution to address
+performance decay due to unforeseen distribution shifts between training and
+test data. While recent TTA methods excel in adapting to test data variations,
+such adaptability exposes a model to vulnerability against malicious examples,
+an aspect that has received limited attention. Previous studies have uncovered
+security vulnerabilities within TTA even when a small proportion of the test
+batch is maliciously manipulated. In response to the emerging threat, we
+propose median batch normalization (MedBN), leveraging the robustness of the
+median for statistics estimation within the batch normalization layer during
+test-time inference. Our method is algorithm-agnostic, thus allowing seamless
+integration with existing TTA frameworks. Our experimental results on benchmark
+datasets, including CIFAR10-C, CIFAR100-C and ImageNet-C, consistently
+demonstrate that MedBN outperforms existing approaches in maintaining robust
+performance across different attack scenarios, encompassing both instant and
+cumulative attacks. Through extensive experiments, we show that our approach
+sustains the performance even in the absence of attacks, achieving a practical
+balance between robustness and performance.",cs.LG,"['cs.LG', 'cs.CR', 'cs.CV']"
+Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis,Tianci Bi · Xiaoyi Zhang · Zhizheng Zhang · Wenxuan Xie · Cuiling Lan · Yan Lu · Nanning Zheng, ,https://arxiv.org/abs/2405.07481,,2405.07481.pdf,Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis,"Significant progress has been made in scene text detection models since the
+rise of deep learning, but scene text layout analysis, which aims to group
+detected text instances as paragraphs, has not kept pace. Previous works either
+treated text detection and grouping using separate models, or train a model
+from scratch while using a unified one. All of them have not yet made full use
+of the already well-trained text detectors and easily obtainable detection
+datasets. In this paper, we present Text Grouping Adapter (TGA), a module that
+can enable the utilization of various pre-trained text detectors to learn
+layout analysis, allowing us to adopt a well-trained text detector right off
+the shelf or just fine-tune it efficiently. Designed to be compatible with
+various text detector architectures, TGA takes detected text regions and image
+features as universal inputs to assemble text instance features. To capture
+broader contextual information for layout analysis, we propose to predict text
+group masks from text instance features by one-to-many assignment. Our
+comprehensive experiments demonstrate that, even with frozen pre-trained
+models, incorporating our TGA into various pre-trained text detectors and text
+spotters can achieve superior layout analysis performance, simultaneously
+inheriting generalized text detection ability from pre-training. In the case of
+full parameter fine-tuning, we can further improve layout analysis performance.",cs.CV,['cs.CV']
+Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement,Jinyoung Jun · Jae-Han Lee · Chang-Su Kim, ,https://arxiv.org/abs/2404.19294,,2404.19294.pdf,Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement,"The main function of depth completion is to compensate for an insufficient
+and unpredictable number of sparse depth measurements of hardware sensors.
+However, existing research on depth completion assumes that the sparsity -- the
+number of points or LiDAR lines -- is fixed for training and testing. Hence,
+the completion performance drops severely when the number of sparse depths
+changes significantly. To address this issue, we propose the sparsity-adaptive
+depth refinement (SDR) framework, which refines monocular depth estimates using
+sparse depth points. For SDR, we propose the masked spatial propagation network
+(MSPN) to perform SDR with a varying number of sparse depths effectively by
+gradually propagating sparse depth information throughout the entire depth map.
+Experimental results demonstrate that MPSN achieves state-of-the-art
+performance on both SDR and conventional depth completion scenarios.",cs.CV,['cs.CV']
+Depth-aware Test-Time Training for Zero-shot Video Object Segmentation,Weihuang Liu · Xi Shen · Haolun Li · Xiuli Bi · Bo Liu · Chi-Man Pun · Xiaodong Cun,https://nifangbaage.github.io/DATTT/,https://arxiv.org/abs/2403.04258,,2403.04258.pdf,Depth-aware Test-Time Training for Zero-shot Video Object Segmentation,"Zero-shot Video Object Segmentation (ZSVOS) aims at segmenting the primary
+moving object without any human annotations. Mainstream solutions mainly focus
+on learning a single model on large-scale video datasets, which struggle to
+generalize to unseen videos. In this work, we introduce a test-time training
+(TTT) strategy to address the problem. Our key insight is to enforce the model
+to predict consistent depth during the TTT process. In detail, we first train a
+single network to perform both segmentation and depth prediction tasks. This
+can be effectively learned with our specifically designed depth modulation
+layer. Then, for the TTT process, the model is updated by predicting consistent
+depth maps for the same frame under different data augmentations. In addition,
+we explore different TTT weight updating strategies. Our empirical results
+suggest that the momentum-based weight initialization and looping-based
+training scheme lead to more stable improvements. Experiments show that the
+proposed method achieves clear improvements on ZSVOS. Our proposed video TTT
+strategy provides significant superiority over state-of-the-art TTT methods.
+Our code is available at: https://nifangbaage.github.io/DATTT.",cs.CV,['cs.CV']
+Viewpoint-Aware Visual Grounding in 3D Scenes,Xiangxi Shi · Zhonghua Wu · Stefan Lee, ,https://arxiv.org/abs/2403.03077,,2403.03077.pdf,MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding,"3D visual grounding involves matching natural language descriptions with
+their corresponding objects in 3D spaces. Existing methods often face
+challenges with accuracy in object recognition and struggle in interpreting
+complex linguistic queries, particularly with descriptions that involve
+multiple anchors or are view-dependent. In response, we present the MiKASA
+(Multi-Key-Anchor Scene-Aware) Transformer. Our novel end-to-end trained model
+integrates a self-attention-based scene-aware object encoder and an original
+multi-key-anchor technique, enhancing object recognition accuracy and the
+understanding of spatial relationships. Furthermore, MiKASA improves the
+explainability of decision-making, facilitating error diagnosis. Our model
+achieves the highest overall accuracy in the Referit3D challenge for both the
+Sr3D and Nr3D datasets, particularly excelling by a large margin in categories
+that require viewpoint-dependent descriptions.",cs.CV,['cs.CV']
+Training Vision Transformers for Semi-Supervised Semantic Segmentation,Xinting Hu · Li Jiang · Bernt Schiele, ,,https://github.com/JoyHuYY1412/S4Former,,,,,nan
+DeconfuseTrack: Dealing with Confusion for Multi-Object Tracking,Cheng Huang · Shoudong Han · Mengyu He · Wenbo Zheng · Yuhao Wei, ,https://arxiv.org/abs/2403.02767,,2403.02767.pdf,DeconfuseTrack:Dealing with Confusion for Multi-Object Tracking,"Accurate data association is crucial in reducing confusion, such as ID
+switches and assignment errors, in multi-object tracking (MOT). However,
+existing advanced methods often overlook the diversity among trajectories and
+the ambiguity and conflicts present in motion and appearance cues, leading to
+confusion among detections, trajectories, and associations when performing
+simple global data association. To address this issue, we propose a simple,
+versatile, and highly interpretable data association approach called Decomposed
+Data Association (DDA). DDA decomposes the traditional association problem into
+multiple sub-problems using a series of non-learning-based modules and
+selectively addresses the confusion in each sub-problem by incorporating
+targeted exploitation of new cues. Additionally, we introduce Occlusion-aware
+Non-Maximum Suppression (ONMS) to retain more occluded detections, thereby
+increasing opportunities for association with trajectories and indirectly
+reducing the confusion caused by missed detections. Finally, based on DDA and
+ONMS, we design a powerful multi-object tracker named DeconfuseTrack,
+specifically focused on resolving confusion in MOT. Extensive experiments
+conducted on the MOT17 and MOT20 datasets demonstrate that our proposed DDA and
+ONMS significantly enhance the performance of several popular trackers.
+Moreover, DeconfuseTrack achieves state-of-the-art performance on the MOT17 and
+MOT20 test sets, significantly outperforms the baseline tracker ByteTrack in
+metrics such as HOTA, IDF1, AssA. This validates that our tracking design
+effectively reduces confusion caused by simple global association.",cs.CV,['cs.CV']
+SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis,Ziqiao Peng · Wentao Hu · Yue Shi · Xiangyu Zhu · Xiaomei Zhang · Hao Zhao · Jun He · Hongyan Liu · Zhaoxin Fan,https://ziqiaopeng.github.io/synctalk/,https://arxiv.org/html/2311.17590v2,,2311.17590v2.pdf,SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis,"Achieving high synchronization in the synthesis of realistic, speech-driven
+talking head videos presents a significant challenge. Traditional Generative
+Adversarial Networks (GAN) struggle to maintain consistent facial identity,
+while Neural Radiance Fields (NeRF) methods, although they can address this
+issue, often produce mismatched lip movements, inadequate facial expressions,
+and unstable head poses. A lifelike talking head requires synchronized
+coordination of subject identity, lip movements, facial expressions, and head
+poses. The absence of these synchronizations is a fundamental flaw, leading to
+unrealistic and artificial outcomes. To address the critical issue of
+synchronization, identified as the ""devil"" in creating realistic talking heads,
+we introduce SyncTalk. This NeRF-based method effectively maintains subject
+identity, enhancing synchronization and realism in talking head synthesis.
+SyncTalk employs a Face-Sync Controller to align lip movements with speech and
+innovatively uses a 3D facial blendshape model to capture accurate facial
+expressions. Our Head-Sync Stabilizer optimizes head poses, achieving more
+natural head movements. The Portrait-Sync Generator restores hair details and
+blends the generated head with the torso for a seamless visual experience.
+Extensive experiments and user studies demonstrate that SyncTalk outperforms
+state-of-the-art methods in synchronization and realism. We recommend watching
+the supplementary video: https://ziqiaopeng.github.io/synctalk",cs.CV,['cs.CV']
+Efficient Test-Time Adaptation of Vision-Language Models,Adilbek Karmanov · Dayan Guan · Shijian Lu · Abdulmotaleb El Saddik · Eric P. Xing,https://kdiaaa.github.io/tda/,https://arxiv.org/abs/2403.18293,,2403.18293.pdf,Efficient Test-Time Adaptation of Vision-Language Models,"Test-time adaptation with pre-trained vision-language models has attracted
+increasing attention for tackling distribution shifts during the test time.
+Though prior studies have achieved very promising performance, they involve
+intensive computation which is severely unaligned with test-time adaptation. We
+design TDA, a training-free dynamic adapter that enables effective and
+efficient test-time adaptation with vision-language models. TDA works with a
+lightweight key-value cache that maintains a dynamic queue with few-shot pseudo
+labels as values and the corresponding test-sample features as keys. Leveraging
+the key-value cache, TDA allows adapting to test data gradually via progressive
+pseudo label refinement which is super-efficient without incurring any
+backpropagation. In addition, we introduce negative pseudo labeling that
+alleviates the adverse impact of pseudo label noises by assigning pseudo labels
+to certain negative classes when the model is uncertain about its pseudo label
+predictions. Extensive experiments over two benchmarks demonstrate TDA's
+superior effectiveness and efficiency as compared with the state-of-the-art.
+The code has been released in \url{https://kdiaaa.github.io/tda/}.",cs.CV,['cs.CV']
+Systematic comparison of semi-supervised and self-supervised learning for medical image classification,Zhe Huang · Ruijie Jiang · Shuchin Aeron · Michael C. Hughes, ,https://arxiv.org/abs/2307.08919v2,,2307.08919v2.pdf,Systematic comparison of semi-supervised and self-supervised learning for medical image classification,"In many medical image classification problems, labeled data is scarce while
+unlabeled data is more available. Semi-supervised learning and self-supervised
+learning are two different research directions that can improve accuracy by
+learning from extra unlabeled data. Recent methods from both directions have
+reported significant gains on traditional benchmarks. Yet past benchmarks do
+not focus on medical tasks and rarely compare self- and semi- methods together
+on equal footing. Furthermore, past benchmarks often handle hyperparameter
+tuning suboptimally. First, they may not tune hyperparameters at all, leading
+to underfitting. Second, when tuning does occur, it often unrealistically uses
+a labeled validation set much larger than the train set. Both cases make
+previously published rankings of methods difficult to translate to practical
+settings. This study contributes a systematic evaluation of self- and semi-
+methods with a unified experimental protocol intended to guide a practitioner
+with scarce overall labeled data and a limited compute budget. We answer two
+key questions: Can hyperparameter tuning be effective with realistic-sized
+validation sets? If so, when all methods are tuned well, which self- or
+semi-supervised methods reach the best accuracy? Our study compares 13
+representative semi- and self-supervised methods to strong labeled-set-only
+baselines on 4 medical datasets. From 20000+ total GPU hours of computation, we
+provide valuable best practices to resource-constrained, results-focused
+practitioners.",cs.CV,"['cs.CV', 'cs.LG']"
+Grounded Text-to-Image Synthesis with Attention Refocusing,Quynh Phung · Songwei Ge · Jia-Bin Huang, ,https://arxiv.org/abs/2306.05427,,2306.05427.pdf,Grounded Text-to-Image Synthesis with Attention Refocusing,"Driven by the scalable diffusion models trained on large-scale datasets,
+text-to-image synthesis methods have shown compelling results. However, these
+models still fail to precisely follow the text prompt involving multiple
+objects, attributes, or spatial compositions. In this paper, we reveal the
+potential causes in the diffusion model's cross-attention and self-attention
+layers. We propose two novel losses to refocus attention maps according to a
+given spatial layout during sampling. Creating the layouts manually requires
+additional effort and can be tedious. Therefore, we explore using large
+language models (LLM) to produce these layouts for our method. We conduct
+extensive experiments on the DrawBench, HRS, and TIFA benchmarks to evaluate
+our proposed method. We show that our proposed attention refocusing effectively
+improves the controllability of existing approaches.",cs.CV,['cs.CV']
+"Flexible Biometrics Recognition: Bridging the Multimodality Gap through Attention, Alignment and Prompt Tuning",Leslie Ching Ow Tiong · Dick Sigmund · Chen-Hui Chan · Andrew Beng Jin Teoh,https://github.com/MIS-DevWorks/FBR,,https://mdpi-res.com/d_attachment/sensors/sensors-23-06006/article_deploy/sensors-23-06006.pdf?version=1687952937,,,,,nan
+Universal Semi-Supervised Domain Adaptation by Mitigating Common-Class Bias,Wenyu Zhang · Qingmu Liu · Felix Ong · Mohamed Ragab · Chuan-Sheng Foo, ,https://arxiv.org/abs/2403.11234,,2403.11234.pdf,Universal Semi-Supervised Domain Adaptation by Mitigating Common-Class Bias,"Domain adaptation is a critical task in machine learning that aims to improve
+model performance on a target domain by leveraging knowledge from a related
+source domain. In this work, we introduce Universal Semi-Supervised Domain
+Adaptation (UniSSDA), a practical yet challenging setting where the target
+domain is partially labeled, and the source and target label space may not
+strictly match. UniSSDA is at the intersection of Universal Domain Adaptation
+(UniDA) and Semi-Supervised Domain Adaptation (SSDA): the UniDA setting does
+not allow for fine-grained categorization of target private classes not
+represented in the source domain, while SSDA focuses on the restricted
+closed-set setting where source and target label spaces match exactly. Existing
+UniDA and SSDA methods are susceptible to common-class bias in UniSSDA
+settings, where models overfit to data distributions of classes common to both
+domains at the expense of private classes. We propose a new prior-guided
+pseudo-label refinement strategy to reduce the reinforcement of common-class
+bias due to pseudo-labeling, a common label propagation strategy in domain
+adaptation. We demonstrate the effectiveness of the proposed strategy on
+benchmark datasets Office-Home, DomainNet, and VisDA. The proposed strategy
+attains the best performance across UniSSDA adaptation settings and establishes
+a new baseline for UniSSDA.",cs.CV,['cs.CV']
+OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation,Bohao Peng · Xiaoyang Wu · Li Jiang · Yukang Chen · Hengshuang Zhao · Zhuotao Tian · Jiaya Jia, ,https://arxiv.org/abs/2403.14418,,2403.14418.pdf,OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation,"The booming of 3D recognition in the 2020s began with the introduction of
+point cloud transformers. They quickly overwhelmed sparse CNNs and became
+state-of-the-art models, especially in 3D semantic segmentation. However,
+sparse CNNs are still valuable networks, due to their efficiency treasure, and
+ease of application. In this work, we reexamine the design distinctions and
+test the limits of what a sparse CNN can achieve. We discover that the key
+credit to the performance difference is adaptivity. Specifically, we propose
+two key components, i.e., adaptive receptive fields (spatially) and adaptive
+relation, to bridge the gap. This exploration led to the creation of
+Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a
+lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal
+computational cost. Without any self-attention modules, OA-CNNs favorably
+surpass point transformers in terms of accuracy in both indoor and outdoor
+scenes, with much less latency and memory cost. Notably, it achieves 76.1%,
+78.9%, and 70.6% mIoU on ScanNet v2, nuScenes, and SemanticKITTI validation
+benchmarks respectively, while maintaining at most 5x better speed than
+transformer counterparts. This revelation highlights the potential of pure
+sparse CNNs to outperform transformer-related networks.",cs.CV,['cs.CV']
+Unified Language-driven Zero-shot Domain Adaptation,Senqiao Yang · Zhuotao Tian · Li Jiang · Jiaya Jia, ,https://arxiv.org/abs/2404.07155,,2404.07155.pdf,Unified Language-driven Zero-shot Domain Adaptation,"This paper introduces Unified Language-driven Zero-shot Domain Adaptation
+(ULDA), a novel task setting that enables a single model to adapt to diverse
+target domains without explicit domain-ID knowledge. We identify the
+constraints in the existing language-driven zero-shot domain adaptation task,
+particularly the requirement for domain IDs and domain-specific models, which
+may restrict flexibility and scalability. To overcome these issues, we propose
+a new framework for ULDA, consisting of Hierarchical Context Alignment (HCA),
+Domain Consistent Representation Learning (DCRL), and Text-Driven Rectifier
+(TDR). These components work synergistically to align simulated features with
+target text across multiple visual levels, retain semantic correlations between
+different regional representations, and rectify biases between simulated and
+real target visual features, respectively. Our extensive empirical evaluations
+demonstrate that this framework achieves competitive performance in both
+settings, surpassing even the model that requires domain-ID, showcasing its
+superiority and generalization ability. The proposed method is not only
+effective but also maintains practicality and efficiency, as it does not
+introduce additional computational costs during inference. Our project page is
+https://senqiaoyang.com/project/ULDA .",cs.CV,['cs.CV']
+Delving into the Trajectory Long-tail Distribution for Muti-object Tracking,Sijia Chen · En Yu · Jinyang Li · Wenbing Tao, ,https://arxiv.org/abs/2403.04700,,2403.04700.pdf,Delving into the Trajectory Long-tail Distribution for Muti-object Tracking,"Multiple Object Tracking (MOT) is a critical area within computer vision,
+with a broad spectrum of practical implementations. Current research has
+primarily focused on the development of tracking algorithms and enhancement of
+post-processing techniques. Yet, there has been a lack of thorough examination
+concerning the nature of tracking data it self. In this study, we pioneer an
+exploration into the distribution patterns of tracking data and identify a
+pronounced long-tail distribution issue within existing MOT datasets. We note a
+significant imbalance in the distribution of trajectory lengths across
+different pedestrians, a phenomenon we refer to as ``pedestrians trajectory
+long-tail distribution''. Addressing this challenge, we introduce a bespoke
+strategy designed to mitigate the effects of this skewed distribution.
+Specifically, we propose two data augmentation strategies, including Stationary
+Camera View Data Augmentation (SVA) and Dynamic Camera View Data Augmentation
+(DVA) , designed for viewpoint states and the Group Softmax (GS) module for
+Re-ID. SVA is to backtrack and predict the pedestrian trajectory of tail
+classes, and DVA is to use diffusion model to change the background of the
+scene. GS divides the pedestrians into unrelated groups and performs softmax
+operation on each group individually. Our proposed strategies can be integrated
+into numerous existing tracking systems, and extensive experimentation
+validates the efficacy of our method in reducing the influence of long-tail
+distribution on multi-object tracking performance. The code is available at
+https://github.com/chen-si-jia/Trajectory-Long-tail-Distribution-for-MOT.",cs.CV,['cs.CV']
+HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes,Yichen Yao · Zimo Jiang · YUJING SUN · Zhencai Zhu · Xinge Zhu · Runnan Chen · Yuexin Ma, ,https://arxiv.org/abs/2403.02769,,2403.02769.pdf,HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes,"Human-centric 3D scene understanding has recently drawn increasing attention,
+driven by its critical impact on robotics. However, human-centric real-life
+scenarios are extremely diverse and complicated, and humans have intricate
+motions and interactions. With limited labeled data, supervised methods are
+difficult to generalize to general scenarios, hindering real-life applications.
+Mimicking human intelligence, we propose an unsupervised 3D detection method
+for human-centric scenarios by transferring the knowledge from synthetic human
+instances to real scenes. To bridge the gap between the distinct data
+representations and feature distributions of synthetic models and real point
+clouds, we introduce novel modules for effective instance-to-scene
+representation transfer and synthetic-to-real feature alignment. Remarkably,
+our method exhibits superior performance compared to current state-of-the-art
+techniques, achieving 87.8% improvement in mAP and closely approaching the
+performance of fully supervised methods (62.15 mAP vs. 69.02 mAP) on HuCenLife
+Dataset.",cs.CV,['cs.CV']
+VAREN: Very Accurate and Realistic Equine Network,Silvia Zuffi · Ylva Mellbin · Ci Li · Markus Höschle · Hedvig Kjellström · Senya Polikovsky · Elin Hernlund · Michael J. Black,https://varen.is.tue.mpg.de/,,https://www.kth.se/is/rpl/rpl-news/accepted-publications-march-1.1339092,,,,,nan
+Mask Grounding for Referring Image Segmentation,Yong Xien Chng · Henry Zheng · Yizeng Han · Xuchong QIU · Gao Huang,https://yxchng.github.io/projects/mask-grounding/,https://arxiv.org/abs/2312.12198,,2312.12198.pdf,Mask Grounding for Referring Image Segmentation,"Referring Image Segmentation (RIS) is a challenging task that requires an
+algorithm to segment objects referred by free-form language expressions.
+Despite significant progress in recent years, most state-of-the-art (SOTA)
+methods still suffer from considerable language-image modality gap at the pixel
+and word level. These methods generally 1) rely on sentence-level language
+features for language-image alignment and 2) lack explicit training supervision
+for fine-grained visual grounding. Consequently, they exhibit weak object-level
+correspondence between visual and language features. Without well-grounded
+features, prior methods struggle to understand complex expressions that require
+strong reasoning over relationships among multiple objects, especially when
+dealing with rarely used or ambiguous clauses. To tackle this challenge, we
+introduce a novel Mask Grounding auxiliary task that significantly improves
+visual grounding within language features, by explicitly teaching the model to
+learn fine-grained correspondence between masked textual tokens and their
+matching visual objects. Mask Grounding can be directly used on prior RIS
+methods and consistently bring improvements. Furthermore, to holistically
+address the modality gap, we also design a cross-modal alignment loss and an
+accompanying alignment module. These additions work synergistically with Mask
+Grounding. With all these techniques, our comprehensive approach culminates in
+MagNet (Mask-grounded Network), an architecture that significantly outperforms
+prior arts on three key benchmarks (RefCOCO, RefCOCO+ and G-Ref), demonstrating
+our method's effectiveness in addressing current limitations of RIS algorithms.
+Our code and pre-trained weights will be released.",cs.CV,['cs.CV']
+MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model,Kaiyu Song · Hanjiang Lai · Yan Pan · Jian Yin, ,https://arxiv.org/abs/2312.04802,,2312.04802.pdf,MimicDiffusion: Purifying Adversarial Perturbation via Mimicking Clean Diffusion Model,"Deep neural networks (DNNs) are vulnerable to adversarial perturbation, where
+an imperceptible perturbation is added to the image that can fool the DNNs.
+Diffusion-based adversarial purification focuses on using the diffusion model
+to generate a clean image against such adversarial attacks. Unfortunately, the
+generative process of the diffusion model is also inevitably affected by
+adversarial perturbation since the diffusion model is also a deep network where
+its input has adversarial perturbation. In this work, we propose
+MimicDiffusion, a new diffusion-based adversarial purification technique, that
+directly approximates the generative process of the diffusion model with the
+clean image as input. Concretely, we analyze the differences between the guided
+terms using the clean image and the adversarial sample. After that, we first
+implement MimicDiffusion based on Manhattan distance. Then, we propose two
+guidance to purify the adversarial perturbation and approximate the clean
+diffusion model. Extensive experiments on three image datasets including
+CIFAR-10, CIFAR-100, and ImageNet with three classifier backbones including
+WideResNet-70-16, WideResNet-28-10, and ResNet50 demonstrate that
+MimicDiffusion significantly performs better than the state-of-the-art
+baselines. On CIFAR-10, CIFAR-100, and ImageNet, it achieves 92.67\%, 61.35\%,
+and 61.53\% average robust accuracy, which are 18.49\%, 13.23\%, and 17.64\%
+higher, respectively. The code is available in the supplementary material.",cs.CV,['cs.CV']
+Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing,Hyelin Nam · Gihyun Kwon · Geon Yeong Park · Jong Chul Ye,https://hyelinnam.github.io/CDS/,https://arxiv.org/abs/2311.18608,,2311.18608.pdf,Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing,"With the remarkable advent of text-to-image diffusion models, image editing
+methods have become more diverse and continue to evolve. A promising recent
+approach in this realm is Delta Denoising Score (DDS) - an image editing
+technique based on Score Distillation Sampling (SDS) framework that leverages
+the rich generative prior of text-to-image diffusion models. However, relying
+solely on the difference between scoring functions is insufficient for
+preserving specific structural elements from the original image, a crucial
+aspect of image editing. To address this, here we present an embarrassingly
+simple yet very powerful modification of DDS, called Contrastive Denoising
+Score (CDS), for latent diffusion models (LDM). Inspired by the similarities
+and differences between DDS and the contrastive learning for unpaired
+image-to-image translation(CUT), we introduce a straightforward approach using
+CUT loss within the DDS framework. Rather than employing auxiliary networks as
+in the original CUT approach, we leverage the intermediate features of LDM,
+specifically those from the self-attention layers, which possesses rich spatial
+information. Our approach enables zero-shot image-to-image translation and
+neural radiance field (NeRF) editing, achieving structural correspondence
+between the input and output while maintaining content controllability.
+Qualitative results and comparisons demonstrates the effectiveness of our
+proposed method. Project page: https://hyelinnam.github.io/CDS/",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modeling,Jiawei Shi · Hui Deng · Yuchao Dai,https://npucvr.github.io/TSM-NRSfM/,https://arxiv.org/abs/2405.04309,,2405.04309.pdf,Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modeling,"Even though Non-rigid Structure-from-Motion (NRSfM) has been extensively
+studied and great progress has been made, there are still key challenges that
+hinder their broad real-world applications: 1) the inherent motion/rotation
+ambiguity requires either explicit camera motion recovery with extra constraint
+or complex Procrustean Alignment; 2) existing low-rank modeling of the global
+shape can over-penalize drastic deformations in the 3D shape sequence. This
+paper proposes to resolve the above issues from a spatial-temporal modeling
+perspective. First, we propose a novel Temporally-smooth Procrustean Alignment
+module that estimates 3D deforming shapes and adjusts the camera motion by
+aligning the 3D shape sequence consecutively. Our new alignment module remedies
+the requirement of complex reference 3D shape during alignment, which is more
+conductive to non-isotropic deformation modeling. Second, we propose a
+spatial-weighted approach to enforce the low-rank constraint adaptively at
+different locations to accommodate drastic spatially-variant deformation
+reconstruction better. Our modeling outperform existing low-rank based methods,
+and extensive experiments across different datasets validate the effectiveness
+of our method.",cs.CV,['cs.CV']
+Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization,Yujia Liu · Chenxi Yang · Dingquan Li · Jianhao Ding · Tingting Jiang, ,https://arxiv.org/abs/2403.11397,,2403.11397.pdf,Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization,"The task of No-Reference Image Quality Assessment (NR-IQA) is to estimate the
+quality score of an input image without additional information. NR-IQA models
+play a crucial role in the media industry, aiding in performance evaluation and
+optimization guidance. However, these models are found to be vulnerable to
+adversarial attacks, which introduce imperceptible perturbations to input
+images, resulting in significant changes in predicted scores. In this paper, we
+propose a defense method to improve the stability in predicted scores when
+attacked by small perturbations, thus enhancing the adversarial robustness of
+NR-IQA models. To be specific, we present theoretical evidence showing that the
+magnitude of score changes is related to the $\ell_1$ norm of the model's
+gradient with respect to the input image. Building upon this theoretical
+foundation, we propose a norm regularization training strategy aimed at
+reducing the $\ell_1$ norm of the gradient, thereby boosting the robustness of
+NR-IQA models. Experiments conducted on four NR-IQA baseline models demonstrate
+the effectiveness of our strategy in reducing score changes in the presence of
+adversarial attacks. To the best of our knowledge, this work marks the first
+attempt to defend against adversarial attacks on NR-IQA models. Our study
+offers valuable insights into the adversarial robustness of NR-IQA models and
+provides a foundation for future research in this area.",cs.CV,"['cs.CV', 'eess.IV']"
+A Unified Framework for Human-centric Point Cloud Video Understanding,Yiteng Xu · Kecheng Ye · xiao han · yiming ren · Xinge Zhu · Yuexin Ma, ,https://arxiv.org/abs/2403.20031,,2403.20031.pdf,A Unified Framework for Human-centric Point Cloud Video Understanding,"Human-centric Point Cloud Video Understanding (PVU) is an emerging field
+focused on extracting and interpreting human-related features from sequences of
+human point clouds, further advancing downstream human-centric tasks and
+applications. Previous works usually focus on tackling one specific task and
+rely on huge labeled data, which has poor generalization capability.
+Considering that human has specific characteristics, including the structural
+semantics of human body and the dynamics of human motions, we propose a unified
+framework to make full use of the prior knowledge and explore the inherent
+features in the data itself for generalized human-centric point cloud video
+understanding. Extensive experiments demonstrate that our method achieves
+state-of-the-art performance on various human-related tasks, including action
+recognition and 3D pose estimation. All datasets and code will be released
+soon.",cs.CV,['cs.CV']
+iKUN: Speak to Trackers without Retraining,Yunhao Du · Cheng Lei · Zhicheng Zhao · Fei Su,https://github.com/dyhBUPT/iKUN,https://arxiv.org/abs/2312.16245,,2312.16245.pdf,iKUN: Speak to Trackers without Retraining,"Referring multi-object tracking (RMOT) aims to track multiple objects based
+on input textual descriptions. Previous works realize it by simply integrating
+an extra textual module into the multi-object tracker. However, they typically
+need to retrain the entire framework and have difficulties in optimization. In
+this work, we propose an insertable Knowledge Unification Network, termed iKUN,
+to enable communication with off-the-shelf trackers in a plug-and-play manner.
+Concretely, a knowledge unification module (KUM) is designed to adaptively
+extract visual features based on textual guidance. Meanwhile, to improve the
+localization accuracy, we present a neural version of Kalman filter (NKF) to
+dynamically adjust process noise and observation noise based on the current
+motion status. Moreover, to address the problem of open-set long-tail
+distribution of textual descriptions, a test-time similarity calibration method
+is proposed to refine the confidence score with pseudo frequency. Extensive
+experiments on Refer-KITTI dataset verify the effectiveness of our framework.
+Finally, to speed up the development of RMOT, we also contribute a more
+challenging dataset, Refer-Dance, by extending public DanceTrack dataset with
+motion and dressing descriptions. The codes and dataset are available at
+https://github.com/dyhBUPT/iKUN.",cs.CV,['cs.CV']
+Accelerating Diffusion Sampling with Optimized Time Steps,Shuchen Xue · Zhaoqiang Liu · Fei Chen · Shifeng Zhang · Tianyang Hu · Enze Xie · Zhenguo Li, ,https://arxiv.org/abs/2402.17376,,2402.17376.pdf,Accelerating Diffusion Sampling with Optimized Time Steps,"Diffusion probabilistic models (DPMs) have shown remarkable performance in
+high-resolution image synthesis, but their sampling efficiency is still to be
+desired due to the typically large number of sampling steps. Recent
+advancements in high-order numerical ODE solvers for DPMs have enabled the
+generation of high-quality images with much fewer sampling steps. While this is
+a significant development, most sampling methods still employ uniform time
+steps, which is not optimal when using a small number of steps. To address this
+issue, we propose a general framework for designing an optimization problem
+that seeks more appropriate time steps for a specific numerical ODE solver for
+DPMs. This optimization problem aims to minimize the distance between the
+ground-truth solution to the ODE and an approximate solution corresponding to
+the numerical solver. It can be efficiently solved using the constrained trust
+region method, taking less than $15$ seconds. Our extensive experiments on both
+unconditional and conditional sampling using pixel- and latent-space DPMs
+demonstrate that, when combined with the state-of-the-art sampling method
+UniPC, our optimized time steps significantly improve image generation
+performance in terms of FID scores for datasets such as CIFAR-10 and ImageNet,
+compared to using uniform time steps.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+DemoFusion: Democratising High-Resolution Image Generation With No $$$,Ruoyi DU · Dongliang Chang · Timothy Hospedales · Yi-Zhe Song · Zhanyu Ma,https://ruoyidu.github.io/demofusion/demofusion.html,https://arxiv.org/abs/2311.16973,,2311.16973.pdf,DemoFusion: Democratising High-Resolution Image Generation With No $$$,"High-resolution image generation with Generative Artificial Intelligence
+(GenAI) has immense potential but, due to the enormous capital investment
+required for training, it is increasingly centralised to a few large
+corporations, and hidden behind paywalls. This paper aims to democratise
+high-resolution GenAI by advancing the frontier of high-resolution generation
+while remaining accessible to a broad audience. We demonstrate that existing
+Latent Diffusion Models (LDMs) possess untapped potential for higher-resolution
+image generation. Our novel DemoFusion framework seamlessly extends open-source
+GenAI models, employing Progressive Upscaling, Skip Residual, and Dilated
+Sampling mechanisms to achieve higher-resolution image generation. The
+progressive nature of DemoFusion requires more passes, but the intermediate
+results can serve as ""previews"", facilitating rapid prompt iteration.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Tumor Micro-environment Interactions Guided Graph Learning for Survival Analysis of Human Cancers from Whole-slide Pathological Images.,WEI SHAO · YangYang Shi · Daoqiang Zhang · Junjie Zhou · Peng Wan, ,,https://www.nature.com/articles/s41467-023-40890-x,,,,,nan
+Adapting Short-Term Transformers for Action Detection in Untrimmed Videos,Min Yang · gaohuan · Ping Guo · Limin Wang, ,https://arxiv.org/abs/2312.01897,,2312.01897.pdf,Adapting Short-Term Transformers for Action Detection in Untrimmed Videos,"Vision Transformer (ViT) has shown high potential in video recognition, owing
+to its flexible design, adaptable self-attention mechanisms, and the efficacy
+of masked pre-training. Yet, it remains unclear how to adapt these pre-trained
+short-term ViTs for temporal action detection (TAD) in untrimmed videos. The
+existing works treat them as off-the-shelf feature extractors for each
+short-trimmed snippet without capturing the fine-grained relation among
+different snippets in a broader temporal context. To mitigate this issue, this
+paper focuses on designing a new mechanism for adapting these pre-trained ViT
+models as a unified long-form video transformer to fully unleash its modeling
+power in capturing inter-snippet relation, while still keeping low computation
+overhead and memory consumption for efficient TAD. To this end, we design
+effective cross-snippet propagation modules to gradually exchange short-term
+video information among different snippets from two levels. For inner-backbone
+information propagation, we introduce a cross-snippet propagation strategy to
+enable multi-snippet temporal feature interaction inside the backbone.For
+post-backbone information propagation, we propose temporal transformer layers
+for further clip-level modeling. With the plain ViT-B pre-trained with
+VideoMAE, our end-to-end temporal action detector (ViT-TAD) yields a very
+competitive performance to previous temporal action detectors, riching up to
+69.5 average mAP on THUMOS14, 37.40 average mAP on ActivityNet-1.3 and 17.20
+average mAP on FineAction.",cs.CV,['cs.CV']
+LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment,yiming ren · xiao han · Chengfeng Zhao · Jingya Wang · Lan Xu · Jingyi Yu · Yuexin Ma, ,https://arxiv.org/abs/2402.17171,,2402.17171.pdf,LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment,"For human-centric large-scale scenes, fine-grained modeling for 3D human
+global pose and shape is significant for scene understanding and can benefit
+many real-world applications. In this paper, we present LiveHPS, a novel
+single-LiDAR-based approach for scene-level human pose and shape estimation
+without any limitation of light conditions and wearable devices. In particular,
+we design a distillation mechanism to mitigate the distribution-varying effect
+of LiDAR point clouds and exploit the temporal-spatial geometric and dynamic
+information existing in consecutive frames to solve the occlusion and noise
+disturbance. LiveHPS, with its efficient configuration and high-quality output,
+is well-suited for real-world applications. Moreover, we propose a huge human
+motion dataset, named FreeMotion, which is collected in various scenarios with
+diverse human poses, shapes and translations. It consists of multi-modal and
+multi-view acquisition data from calibrated and synchronized LiDARs, cameras,
+and IMUs. Extensive experiments on our new dataset and other public datasets
+demonstrate the SOTA performance and robustness of our approach. We will
+release our code and dataset soon.",cs.CV,['cs.CV']
+CoDe: An Explicit Content Decoupling Framework for Image Restoration,Enxuan Gu · Hongwei Ge · Yong Guo, ,https://arxiv.org/abs/2312.05006,,2312.05006.pdf,Decoupling Degradation and Content Processing for Adverse Weather Image Restoration,"Adverse weather image restoration strives to recover clear images from those
+affected by various weather types, such as rain, haze, and snow. Each weather
+type calls for a tailored degradation removal approach due to its unique impact
+on images. Conversely, content reconstruction can employ a uniform approach, as
+the underlying image content remains consistent. Although previous techniques
+can handle multiple weather types within a single network, they neglect the
+crucial distinction between these two processes, limiting the quality of
+restored images. This work introduces a novel adverse weather image restoration
+method, called DDCNet, which decouples the degradation removal and content
+reconstruction process at the feature level based on their channel statistics.
+Specifically, we exploit the unique advantages of the Fourier transform in both
+these two processes: (1) the degradation information is mainly located in the
+amplitude component of the Fourier domain, and (2) the Fourier domain contains
+global information. The former facilitates channel-dependent degradation
+removal operation, allowing the network to tailor responses to various adverse
+weather types; the latter, by integrating Fourier's global properties into
+channel-independent content features, enhances network capacity for consistent
+global content reconstruction. We further augment the degradation removal
+process with a degradation mapping loss function. Extensive experiments
+demonstrate our method achieves state-of-the-art performance in multiple
+adverse weather removal benchmarks.",cs.CV,['cs.CV']
+MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation,Xiaolong Deng · Huisi Wu · Runhao Zeng · Jing Qin,https://github.com/dengxl0520/MemSAM,https://arxiv.org/abs/2311.10529,,2311.10529.pdf,Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectification,"The Segment Anything Model (SAM) has recently emerged as a groundbreaking
+foundation model for prompt-driven image segmentation tasks. However, both the
+original SAM and its medical variants require slice-by-slice manual prompting
+of target structures, which directly increase the burden for applications.
+Despite attempts of auto-prompting to turn SAM into a fully automatic manner,
+it still exhibits subpar performance and lacks of reliability especially in the
+field of medical imaging. In this paper, we propose UR-SAM, an uncertainty
+rectified SAM framework to enhance the reliability for auto-prompting medical
+image segmentation. Building upon a localization framework for automatic prompt
+generation, our method incorporates a prompt augmentation module to obtain a
+series of input prompts for SAM for uncertainty estimation and an
+uncertainty-based rectification module to further utilize the distribution of
+estimated uncertainty to improve the segmentation performance. Extensive
+experiments on two public 3D medical datasets covering the segmentation of 35
+organs demonstrate that without supplementary training or fine-tuning, our
+method further improves the segmentation performance with up to 10.7 % and 13.8
+% in dice similarity coefficient, demonstrating efficiency and broad
+capabilities for medical image segmentation without manual prompting.",cs.CV,['cs.CV']
+Generative Powers of Ten,Xiaojuan Wang · Janne Kontkanen · Brian Curless · Steve Seitz · Ira Kemelmacher-Shlizerman · Ben Mildenhall · Pratul P. Srinivasan · Dor Verbin · Aleksander Holynski, ,https://arxiv.org/abs/2312.02149,,2312.02149.pdf,Generative Powers of Ten,"We present a method that uses a text-to-image model to generate consistent
+content across multiple image scales, enabling extreme semantic zooms into a
+scene, e.g., ranging from a wide-angle landscape view of a forest to a macro
+shot of an insect sitting on one of the tree branches. We achieve this through
+a joint multi-scale diffusion sampling approach that encourages consistency
+across different scales while preserving the integrity of each individual
+sampling process. Since each generated scale is guided by a different text
+prompt, our method enables deeper levels of zoom than traditional
+super-resolution methods that may struggle to create new contextual structure
+at vastly different scales. We compare our method qualitatively with
+alternative techniques in image super-resolution and outpainting, and show that
+our method is most effective at generating consistent multi-scale content.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.GR']"
+When Visual Grounding Meets Gigapixel-level Large-scale Scenes: Benchmark and Approach,TAO MA · Bing Bai · Haozhe Lin · Heyuan Wang · Yu Wang · Lin Luo · Lu Fang, ,https://arxiv.org/abs/2307.11558,,2307.11558.pdf,Advancing Visual Grounding with Scene Knowledge: Benchmark and Method,"Visual grounding (VG) aims to establish fine-grained alignment between vision
+and language. Ideally, it can be a testbed for vision-and-language models to
+evaluate their understanding of the images and texts and their reasoning
+abilities over their joint space. However, most existing VG datasets are
+constructed using simple description texts, which do not require sufficient
+reasoning over the images and texts. This has been demonstrated in a recent
+study~\cite{luo2022goes}, where a simple LSTM-based text encoder without
+pretraining can achieve state-of-the-art performance on mainstream VG datasets.
+Therefore, in this paper, we propose a novel benchmark of \underline{S}cene
+\underline{K}nowledge-guided \underline{V}isual \underline{G}rounding (SK-VG),
+where the image content and referring expressions are not sufficient to ground
+the target objects, forcing the models to have a reasoning ability on the
+long-form scene knowledge. To perform this task, we propose two approaches to
+accept the triple-type input, where the former embeds knowledge into the image
+features before the image-query interaction; the latter leverages linguistic
+structure to assist in computing the image-text matching. We conduct extensive
+experiments to analyze the above methods and show that the proposed approaches
+achieve promising results but still leave room for improvement, including
+performance and interpretability. The dataset and code are available at
+\url{https://github.com/zhjohnchan/SK-VG}.",cs.CV,"['cs.CV', 'cs.CL']"
+Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization,Jimyeong Kim · Jungwon Park · Wonjong Rhee, ,https://arxiv.org/abs/2403.15330,,2403.15330.pdf,Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization,"In text-to-image personalization, a timely and crucial challenge is the
+tendency of generated images overfitting to the biases present in the reference
+images. We initiate our study with a comprehensive categorization of the biases
+into background, nearby-object, tied-object, substance (in style
+re-contextualization), and pose biases. These biases manifest in the generated
+images due to their entanglement into the subject embedding. This undesired
+embedding entanglement not only results in the reflection of biases from the
+reference images into the generated images but also notably diminishes the
+alignment of the generated images with the given generation prompt. To address
+this challenge, we propose SID~(Selectively Informative Description), a text
+description strategy that deviates from the prevalent approach of only
+characterizing the subject's class identification. SID is generated utilizing
+multimodal GPT-4 and can be seamlessly integrated into optimization-based
+models. We present comprehensive experimental results along with analyses of
+cross-attention maps, subject-alignment, non-subject-disentanglement, and
+text-alignment.",cs.CV,['cs.CV']
+SFOD: Spiking Fusion Object Detector,Yimeng Fan · Wei Zhang · Changsong Liu · Mingyang Li · Wenrui Lu,https://github.com/yimeng-fan/SFOD,https://arxiv.org/abs/2403.15192,,2403.15192.pdf,SFOD: Spiking Fusion Object Detector,"Event cameras, characterized by high temporal resolution, high dynamic range,
+low power consumption, and high pixel bandwidth, offer unique capabilities for
+object detection in specialized contexts. Despite these advantages, the
+inherent sparsity and asynchrony of event data pose challenges to existing
+object detection algorithms. Spiking Neural Networks (SNNs), inspired by the
+way the human brain codes and processes information, offer a potential solution
+to these difficulties. However, their performance in object detection using
+event cameras is limited in current implementations. In this paper, we propose
+the Spiking Fusion Object Detector (SFOD), a simple and efficient approach to
+SNN-based object detection. Specifically, we design a Spiking Fusion Module,
+achieving the first-time fusion of feature maps from different scales in SNNs
+applied to event cameras. Additionally, through integrating our analysis and
+experiments conducted during the pretraining of the backbone network on the
+NCAR dataset, we delve deeply into the impact of spiking decoding strategies
+and loss functions on model performance. Thereby, we establish state-of-the-art
+classification results based on SNNs, achieving 93.7\% accuracy on the NCAR
+dataset. Experimental results on the GEN1 detection dataset demonstrate that
+the SFOD achieves a state-of-the-art mAP of 32.1\%, outperforming existing
+SNN-based approaches. Our research not only underscores the potential of SNNs
+in object detection with event cameras but also propels the advancement of
+SNNs. Code is available at https://github.com/yimeng-fan/SFOD.",cs.CV,"['cs.CV', 'cs.AI']"
+A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives,Simone Alberto Peirone · Francesca Pistilli · Antonio Alliegro · Giuseppe Averta,https://sapeirone.github.io/EgoPack/,https://arxiv.org/abs/2403.03037,,2403.03037.pdf,A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives,"Human comprehension of a video stream is naturally broad: in a few instants,
+we are able to understand what is happening, the relevance and relationship of
+objects, and forecast what will follow in the near future, everything all at
+once. We believe that - to effectively transfer such an holistic perception to
+intelligent machines - an important role is played by learning to correlate
+concepts and to abstract knowledge coming from different tasks, to
+synergistically exploit them when learning novel skills. To accomplish this, we
+seek for a unified approach to video understanding which combines shared
+temporal modelling of human actions with minimal overhead, to support multiple
+downstream tasks and enable cooperation when learning novel skills. We then
+propose EgoPack, a solution that creates a collection of task perspectives that
+can be carried across downstream tasks and used as a potential source of
+additional insights, as a backpack of skills that a robot can carry around and
+use when needed. We demonstrate the effectiveness and efficiency of our
+approach on four Ego4D benchmarks, outperforming current state-of-the-art
+methods.",cs.CV,"['cs.CV', 'cs.LG']"
+How Far Can We Compress Instant NGP-Based NeRF?,Yihang Chen · Qianyi Wu · Mehrtash Harandi · Jianfei Cai, ,https://arxiv.org/abs/2310.14695,,2310.14695.pdf,CAwa-NeRF: Instant Learning of Compression-Aware NeRF Features,"Modeling 3D scenes by volumetric feature grids is one of the promising
+directions of neural approximations to improve Neural Radiance Fields (NeRF).
+Instant-NGP (INGP) introduced multi-resolution hash encoding from a lookup
+table of trainable feature grids which enabled learning high-quality neural
+graphics primitives in a matter of seconds. However, this improvement came at
+the cost of higher storage size. In this paper, we address this challenge by
+introducing instant learning of compression-aware NeRF features (CAwa-NeRF),
+that allows exporting the zip compressed feature grids at the end of the model
+training with a negligible extra time overhead without changing neither the
+storage architecture nor the parameters used in the original INGP paper.
+Nonetheless, the proposed method is not limited to INGP but could also be
+adapted to any model. By means of extensive simulations, our proposed instant
+learning pipeline can achieve impressive results on different kinds of static
+scenes such as single object masked background scenes and real-life scenes
+captured in our studio. In particular, for single object masked background
+scenes CAwa-NeRF compresses the feature grids down to 6% (1.2 MB) of the
+original size without any loss in the PSNR (33 dB) or down to 2.4% (0.53 MB)
+with a slight virtual loss (32.31 dB).",cs.CV,['cs.CV']
+Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners,Yazhou Xing · Yingqing He · Zeyue Tian · Xintao Wang · Qifeng Chen, ,https://arxiv.org/abs/2402.17723,,2402.17723.pdf,Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners,"Video and audio content creation serves as the core technique for the movie
+industry and professional users. Recently, existing diffusion-based methods
+tackle video and audio generation separately, which hinders the technique
+transfer from academia to industry. In this work, we aim at filling the gap,
+with a carefully designed optimization-based framework for cross-visual-audio
+and joint-visual-audio generation. We observe the powerful generation ability
+of off-the-shelf video or audio generation models. Thus, instead of training
+the giant models from scratch, we propose to bridge the existing strong models
+with a shared latent representation space. Specifically, we propose a
+multimodality latent aligner with the pre-trained ImageBind model. Our latent
+aligner shares a similar core as the classifier guidance that guides the
+diffusion denoising process during inference time. Through carefully designed
+optimization strategy and loss functions, we show the superior performance of
+our method on joint video-audio generation, visual-steered audio generation,
+and audio-steered visual generation tasks. The project website can be found at
+https://yzxing87.github.io/Seeing-and-Hearing/",cs.CV,"['cs.CV', 'cs.MM', 'cs.SD', 'eess.AS']"
+VidToMe: Video Token Merging for Zero-Shot Video Editing,Xirui Li · Chao Ma · Xiaokang Yang · Ming-Hsuan Yang,https://vidtome-diffusion.github.io/,https://arxiv.org/abs/2312.10656,,2312.10656.pdf,VidToMe: Video Token Merging for Zero-Shot Video Editing,"Diffusion models have made significant advances in generating high-quality
+images, but their application to video generation has remained challenging due
+to the complexity of temporal motion. Zero-shot video editing offers a solution
+by utilizing pre-trained image diffusion models to translate source videos into
+new ones. Nevertheless, existing methods struggle to maintain strict temporal
+consistency and efficient memory consumption. In this work, we propose a novel
+approach to enhance temporal consistency in generated videos by merging
+self-attention tokens across frames. By aligning and compressing temporally
+redundant tokens across frames, our method improves temporal coherence and
+reduces memory consumption in self-attention computations. The merging strategy
+matches and aligns tokens according to the temporal correspondence between
+frames, facilitating natural temporal consistency in generated video frames. To
+manage the complexity of video processing, we divide videos into chunks and
+develop intra-chunk local token merging and inter-chunk global token merging,
+ensuring both short-term video continuity and long-term content consistency.
+Our video editing approach seamlessly extends the advancements in image editing
+to video editing, rendering favorable results in temporal consistency over
+state-of-the-art methods.",cs.CV,['cs.CV']
+Real-Time Exposure Correction via Collaborative Transformations and Adaptive Sampling,Ziwen Li · Feng Zhang · Meng Cao · Jinpu Zhang · Yuanjie Shao · Yuehuan Wang · Nong Sang,https://github.com/HUST-IAL/CoTF,,https://www.semanticscholar.org/paper/An-Efficient-Method-for-Real-Time-Image-Exposure-Yang-Zhang/b40baf5034dcc98f06f53abe907b9ac0395e2bb2,,,,,nan
+Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering,Zhiwen Yan · Weng Fei Low · Yu Chen · Gim Hee Lee, ,https://arxiv.org/abs/2311.17089,,2311.17089.pdf,Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering,"3D Gaussians have recently emerged as a highly efficient representation for
+3D reconstruction and rendering. Despite its high rendering quality and speed
+at high resolutions, they both deteriorate drastically when rendered at lower
+resolutions or from far away camera position. During low resolution or far away
+rendering, the pixel size of the image can fall below the Nyquist frequency
+compared to the screen size of each splatted 3D Gaussian and leads to aliasing
+effect. The rendering is also drastically slowed down by the sequential alpha
+blending of more splatted Gaussians per pixel. To address these issues, we
+propose a multi-scale 3D Gaussian splatting algorithm, which maintains
+Gaussians at different scales to represent the same scene. Higher-resolution
+images are rendered with more small Gaussians, and lower-resolution images are
+rendered with fewer larger Gaussians. With similar training time, our algorithm
+can achieve 13\%-66\% PSNR and 160\%-2400\% rendering speed improvement at
+4$\times$-128$\times$ scale rendering on Mip-NeRF360 dataset compared to the
+single scale 3D Gaussian splitting. Our code and more results are available on
+our project website https://jokeryan.github.io/projects/ms-gs/",cs.CV,['cs.CV']
+Construct to Associate: Cooperative Context Learning for Domain Adaptive Point Cloud Segmentation,Guangrui Li, ,,https://ieeexplore.ieee.org/document/10330760,,,,,nan
+Holistic Features are almost Sufficient for Text-to-Video Retrieval,Kaibin Tian · Ruixiang Zhao · Zijie Xin · Bangxiang Lan · Xirong Li,https://github.com/ruc-aimc-lab/TeachCLIP,,https://lixirong.net/research/cvpr2024-holistic-features-are-almost-sufficient-for-text-to-video-retrieval,,,,,nan
+TE-TAD: Towards Fully End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression,Ho-Joong Kim · Jung-Ho Hong · Heejo Kong · Seong-Whan Lee, ,https://arxiv.org/abs/2404.02405,,2404.02405.pdf,TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression,"In this paper, we investigate that the normalized coordinate expression is a
+key factor as reliance on hand-crafted components in query-based detectors for
+temporal action detection (TAD). Despite significant advancements towards an
+end-to-end framework in object detection, query-based detectors have been
+limited in achieving full end-to-end modeling in TAD. To address this issue, we
+propose \modelname{}, a full end-to-end temporal action detection transformer
+that integrates time-aligned coordinate expression. We reformulate coordinate
+expression utilizing actual timeline values, ensuring length-invariant
+representations from the extremely diverse video duration environment.
+Furthermore, our proposed adaptive query selection dynamically adjusts the
+number of queries based on video length, providing a suitable solution for
+varying video durations compared to a fixed query set. Our approach not only
+simplifies the TAD process by eliminating the need for hand-crafted components
+but also significantly improves the performance of query-based detectors. Our
+TE-TAD outperforms the previous query-based detectors and achieves competitive
+performance compared to state-of-the-art methods on popular benchmark datasets.
+Code is available at: https://github.com/Dotori-HJ/TE-TAD",cs.CV,['cs.CV']
+Dual Prototype Attention for Unsupervised Video Object Segmentation,Suhwan Cho · Minhyeok Lee · Seunghoon Lee · Dogyoon Lee · Heeseung Choi · Ig-Jae Kim · Sangyoun Lee, ,https://arxiv.org/abs/2309.14786,,2309.14786.pdf,Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation,"Unsupervised video object segmentation (VOS) is a task that aims to detect
+the most salient object in a video without external guidance about the object.
+To leverage the property that salient objects usually have distinctive
+movements compared to the background, recent methods collaboratively use motion
+cues extracted from optical flow maps with appearance cues extracted from RGB
+images. However, as optical flow maps are usually very relevant to segmentation
+masks, the network is easy to be learned overly dependent on the motion cues
+during network training. As a result, such two-stream approaches are vulnerable
+to confusing motion cues, making their prediction unstable. To relieve this
+issue, we design a novel motion-as-option network by treating motion cues as
+optional. During network training, RGB images are randomly provided to the
+motion encoder instead of optical flow maps, to implicitly reduce motion
+dependency of the network. As the learned motion encoder can deal with both RGB
+images and optical flow maps, two different predictions can be generated
+depending on which source information is used as motion input. In order to
+fully exploit this property, we also propose an adaptive output selection
+algorithm to adopt optimal prediction result at test time. Our proposed
+approach affords state-of-the-art performance on all public benchmark datasets,
+even maintaining real-time inference speed.",cs.CV,['cs.CV']
+Adaptive Slot Attention: Object Discovery with Dynamic Slot Number,Ke Fan · Zechen Bai · Tianjun Xiao · Tong He · Max Horn · Yanwei Fu · Francesco Locatello · Zheng Zhang, ,https://arxiv.org/abs/2307.09437,,2307.09437.pdf,Grounded Object Centric Learning,"The extraction of modular object-centric representations for downstream tasks
+is an emerging area of research. Learning grounded representations of objects
+that are guaranteed to be stable and invariant promises robust performance
+across different tasks and environments. Slot Attention (SA) learns
+object-centric representations by assigning objects to \textit{slots}, but
+presupposes a \textit{single} distribution from which all slots are randomly
+initialised. This results in an inability to learn \textit{specialized} slots
+which bind to specific object types and remain invariant to identity-preserving
+changes in object appearance. To address this, we present
+\emph{\textsc{Co}nditional \textsc{S}lot \textsc{A}ttention} (\textsc{CoSA})
+using a novel concept of \emph{Grounded Slot Dictionary} (GSD) inspired by
+vector quantization. Our proposed GSD comprises (i) canonical object-level
+property vectors and (ii) parametric Gaussian distributions, which define a
+prior over the slots. We demonstrate the benefits of our method in multiple
+downstream tasks such as scene generation, composition, and task adaptation,
+whilst remaining competitive with SA in popular object discovery benchmarks.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras,Ashwath Shetty · Marc Habermann · Guoxing Sun · Diogo Luvizon · Vladislav Golyanik · Christian Theobalt, ,https://arxiv.org/abs/2312.07423,,2312.07423.pdf,Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras,"We present the first approach to render highly realistic free-viewpoint
+videos of a human actor in general apparel, from sparse multi-view recording to
+display, in real-time at an unprecedented 4K resolution. At inference, our
+method only requires four camera views of the moving actor and the respective
+3D skeletal pose. It handles actors in wide clothing, and reproduces even
+fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand
+gestures. At training time, our learning-based approach expects dense
+multi-view video and a rigged static surface scan of the actor. Our method
+comprises three main stages. Stage 1 is a skeleton-driven neural approach for
+high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel
+solution to create a view-dependent texture using four test-time camera views
+as input. Finally, stage 3 comprises a new image-based refinement network
+rendering the final 4K image given the output from the previous stages. Our
+approach establishes a new benchmark for real-time rendering resolution and
+quality using sparse input camera views, unlocking possibilities for immersive
+telepresence.",cs.CV,['cs.CV']
+Seeing the World through Your Eyes,Hadi Alzayer · Kevin Zhang · Brandon Y. Feng · Christopher Metzler · Jia-Bin Huang, ,https://arxiv.org/abs/2306.09348,,2306.09348.pdf,Seeing the World through Your Eyes,"The reflective nature of the human eye is an underappreciated source of
+information about what the world around us looks like. By imaging the eyes of a
+moving person, we can collect multiple views of a scene outside the camera's
+direct line of sight through the reflections in the eyes. In this paper, we
+reconstruct a 3D scene beyond the camera's line of sight using portrait images
+containing eye reflections. This task is challenging due to 1) the difficulty
+of accurately estimating eye poses and 2) the entangled appearance of the eye
+iris and the scene reflections. Our method jointly refines the cornea poses,
+the radiance field depicting the scene, and the observer's eye iris texture. We
+further propose a simple regularization prior on the iris texture pattern to
+improve reconstruction quality. Through various experiments on synthetic and
+real-world captures featuring people with varied eye colors, we demonstrate the
+feasibility of our approach to recover 3D scenes using eye reflections.",cs.CV,['cs.CV']
+NeRF Analogies - Example-Based Visual Attribute Transfer for NeRFs,Michael Fischer · Zhengqin Li · Thu Nguyen-Phuoc · Aljaž Božič · Zhao Dong · Carl Marshall · Tobias Ritschel, ,https://arxiv.org/abs/2402.08622,,2402.08622.pdf,NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs,"A Neural Radiance Field (NeRF) encodes the specific relation of 3D geometry
+and appearance of a scene. We here ask the question whether we can transfer the
+appearance from a source NeRF onto a target 3D geometry in a semantically
+meaningful way, such that the resulting new NeRF retains the target geometry
+but has an appearance that is an analogy to the source NeRF. To this end, we
+generalize classic image analogies from 2D images to NeRFs. We leverage
+correspondence transfer along semantic affinity that is driven by semantic
+features from large, pre-trained 2D image models to achieve multi-view
+consistent appearance transfer. Our method allows exploring the mix-and-match
+product space of 3D geometry and appearance. We show that our method
+outperforms traditional stylization-based methods and that a large majority of
+users prefer our method over several typical baselines.",cs.CV,"['cs.CV', 'cs.GR']"
+MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers,Haoyu Ma · Shahin Mahdizadehaghdam · Bichen Wu · Zhipeng Fan · Yuchao Gu · Wenliang Zhao · Lior Shapira · Xiaohui Xie, ,https://arxiv.org/abs/2312.12468,,2312.12468.pdf,MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers,"Recent advances in generative AI have significantly enhanced image and video
+editing, particularly in the context of text prompt control. State-of-the-art
+approaches predominantly rely on diffusion models to accomplish these tasks.
+However, the computational demands of diffusion-based methods are substantial,
+often necessitating large-scale paired datasets for training, and therefore
+challenging the deployment in real applications. To address these issues, this
+paper breaks down the text-based video editing task into two stages. First, we
+leverage an pre-trained text-to-image diffusion model to simultaneously edit
+few keyframes in an zero-shot way. Second, we introduce an efficient model
+called MaskINT, which is built on non-autoregressive masked generative
+transformers and specializes in frame interpolation between the edited
+keyframes, using the structural guidance from intermediate frames. Experimental
+results suggest that our MaskINT achieves comparable performance with
+diffusion-based methodologies, while significantly improve the inference time.
+This research offers a practical solution for text-based video editing and
+showcases the potential of non-autoregressive masked generative transformers in
+this domain.",cs.CV,['cs.CV']
+Person in Place: Generating Associative Skeleton-Guidance Maps for Human-Object Interaction Image Editing,ChangHee Yang · ChanHee Kang · Kyeongbo Kong · Hanni Oh · Suk-Ju Kang,https://yangchanghee.github.io/Person-in-Place_page/,,https://vds.sogang.ac.kr/?cat=5,,,,,nan
+Correspondence-Free Non-Rigid Point Set Registration Using Unsupervised Clustering Analysis,Mingyang Zhao · Jiang Jingen · Lei Ma · Shiqing Xin · Gaofeng Meng · Dong-Ming Yan, ,,https://link.springer.com/article/10.1007/s11042-023-16854-0,,,,,nan
+Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion,Litu Rout · Yujia Chen · Abhishek Kumar · Constantine Caramanis · Sanjay Shakkottai · Wen-Sheng Chu,https://stsl-inverse-edit.github.io/,https://arxiv.org/abs/2312.00852,,2312.00852.pdf,Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion,"Sampling from the posterior distribution poses a major computational
+challenge in solving inverse problems using latent diffusion models. Common
+methods rely on Tweedie's first-order moments, which are known to induce a
+quality-limiting bias. Existing second-order approximations are impractical due
+to prohibitive computational costs, making standard reverse diffusion processes
+intractable for posterior sampling. This paper introduces Second-order Tweedie
+sampler from Surrogate Loss (STSL), a novel sampler that offers efficiency
+comparable to first-order Tweedie with a tractable reverse process using
+second-order approximation. Our theoretical results reveal that the
+second-order approximation is lower bounded by our surrogate loss that only
+requires $O(1)$ compute using the trace of the Hessian, and by the lower bound
+we derive a new drift term to make the reverse process tractable. Our method
+surpasses SoTA solvers PSLD and P2L, achieving 4X and 8X reduction in neural
+function evaluations, respectively, while notably enhancing sampling quality on
+FFHQ, ImageNet, and COCO benchmarks. In addition, we show STSL extends to
+text-guided image editing and addresses residual distortions present from
+corrupted images in leading text-guided image editing methods. To our best
+knowledge, this is the first work to offer an efficient second-order
+approximation in solving inverse problems using latent diffusion and editing
+real-world images with corruptions.",cs.LG,"['cs.LG', 'cs.CV', 'stat.ML']"
+Human Motion Prediction under Unexpected Perturbation,Jiangbei Yue · Baiyi Li · Julien Pettré · Armin Seyfried · He Wang, ,https://arxiv.org/abs/2403.15891,,2403.15891.pdf,Human Motion Prediction under Unexpected Perturbation,"We investigate a new task in human motion prediction, which is predicting
+motions under unexpected physical perturbation potentially involving multiple
+people. Compared with existing research, this task involves predicting less
+controlled, unpremeditated and pure reactive motions in response to external
+impact and how such motions can propagate through people. It brings new
+challenges such as data scarcity and predicting complex interactions. To this
+end, we propose a new method capitalizing differential physics and deep neural
+networks, leading to an explicit Latent Differential Physics (LDP) model.
+Through experiments, we demonstrate that LDP has high data efficiency,
+outstanding prediction accuracy, strong generalizability and good
+explainability. Since there is no similar research, a comprehensive comparison
+with 11 adapted baselines from several relevant domains is conducted, showing
+LDP outperforming existing research both quantitatively and qualitatively,
+improving prediction accuracy by as much as 70%, and demonstrating
+significantly stronger generalization.",cs.CV,['cs.CV']
+TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation,Xiaopei Wu · Yuenan Hou · Xiaoshui Huang · Binbin Lin · Tong He · Xinge Zhu · Yuexin Ma · Boxi Wu · Haifeng Liu · Deng Cai · Wanli Ouyang, ,https://arxiv.org/html/2309.07849v3,,2309.07849v3.pdf,TFNet: Exploiting Temporal Cues for Fast and Accurate LiDAR Semantic Segmentation,"LiDAR semantic segmentation plays a crucial role in enabling autonomous
+driving and robots to understand their surroundings accurately and robustly. A
+multitude of methods exist within this domain, including point-based,
+range-image-based, polar-coordinate-based, and hybrid strategies. Among these,
+range-image-based techniques have gained widespread adoption in practical
+applications due to their efficiency. However, they face a significant
+challenge known as the ``many-to-one'' problem caused by the range image's
+limited horizontal and vertical angular resolution. As a result, around 20% of
+the 3D points can be occluded. In this paper, we present TFNet, a
+range-image-based LiDAR semantic segmentation method that utilizes temporal
+information to address this issue. Specifically, we incorporate a temporal
+fusion layer to extract useful information from previous scans and integrate it
+with the current scan. We then design a max-voting-based post-processing
+technique to correct false predictions, particularly those caused by the
+``many-to-one'' issue. We evaluated the approach on two benchmarks and
+demonstrated that the plug-in post-processing technique is generic and can be
+applied to various networks.",cs.CV,['cs.CV']
+ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning,Beomyoung Kim · Joonsang Yu · Sung Ju Hwang,https://github.com/clovaai/ECLIPSE,https://arxiv.org/abs/2403.20126,,2403.20126.pdf,ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning,"Panoptic segmentation, combining semantic and instance segmentation, stands
+as a cutting-edge computer vision task. Despite recent progress with deep
+learning models, the dynamic nature of real-world applications necessitates
+continual learning, where models adapt to new classes (plasticity) over time
+without forgetting old ones (catastrophic forgetting). Current continual
+segmentation methods often rely on distillation strategies like knowledge
+distillation and pseudo-labeling, which are effective but result in increased
+training complexity and computational overhead. In this paper, we introduce a
+novel and efficient method for continual panoptic segmentation based on Visual
+Prompt Tuning, dubbed ECLIPSE. Our approach involves freezing the base model
+parameters and fine-tuning only a small set of prompt embeddings, addressing
+both catastrophic forgetting and plasticity and significantly reducing the
+trainable parameters. To mitigate inherent challenges such as error propagation
+and semantic drift in continual segmentation, we propose logit manipulation to
+effectively leverage common knowledge across the classes. Experiments on ADE20K
+continual panoptic segmentation benchmark demonstrate the superiority of
+ECLIPSE, notably its robustness against catastrophic forgetting and its
+reasonable plasticity, achieving a new state-of-the-art. The code is available
+at https://github.com/clovaai/ECLIPSE.",cs.CV,['cs.CV']
+LEOD: Label-Efficient Object Detection for Event Cameras,Ziyi Wu · Mathias Gehrig · Qing Lyu · Xudong Liu · Igor Gilitschenski,https://github.com/Wuziyi616/LEOD,https://arxiv.org/abs/2311.17286,,2311.17286.pdf,LEOD: Label-Efficient Object Detection for Event Cameras,"Object detection with event cameras benefits from the sensor's low latency
+and high dynamic range. However, it is costly to fully label event streams for
+supervised training due to their high temporal resolution. To reduce this cost,
+we present LEOD, the first method for label-efficient event-based detection.
+Our approach unifies weakly- and semi-supervised object detection with a
+self-training mechanism. We first utilize a detector pre-trained on limited
+labels to produce pseudo ground truth on unlabeled events. Then, the detector
+is re-trained with both real and generated labels. Leveraging the temporal
+consistency of events, we run bi-directional inference and apply tracking-based
+post-processing to enhance the quality of pseudo labels. To stabilize training
+against label noise, we further design a soft anchor assignment strategy. We
+introduce new experimental protocols to evaluate the task of label-efficient
+event-based detection on Gen1 and 1Mpx datasets. LEOD consistently outperforms
+supervised baselines across various labeling ratios. For example, on Gen1, it
+improves mAP by 8.6% and 7.8% for RVT-S trained with 1% and 2% labels. On 1Mpx,
+RVT-S with 10% labels even surpasses its fully-supervised counterpart using
+100% labels. LEOD maintains its effectiveness even when all labeled data are
+available, reaching new state-of-the-art results. Finally, we show that our
+method readily scales to improve larger detectors as well. Code is released at
+https://github.com/Wuziyi616/LEOD",cs.CV,['cs.CV']
+Adapters Strike Back,Jan-Martin Steitz · Stefan Roth, ,,https://strikefans.com/the-ink-black-heart-has-wrapped/,,,,,nan
+CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras,Sachin Shah · Matthew Chan · Haoming Cai · Jingxi Chen · Sakshum Kulshrestha · Chahat Deep Singh · Yiannis Aloimonos · Christopher Metzler, ,https://arxiv.org/abs/2404.11511,,2404.11511.pdf,"Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging","Traditional cameras face a trade-off between low-light performance and
+high-speed imaging: longer exposure times to capture sufficient light results
+in motion blur, whereas shorter exposures result in Poisson-corrupted noisy
+images. While burst photography techniques help mitigate this tradeoff,
+conventional cameras are fundamentally limited in their sensor noise
+characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors
+have emerged as promising alternatives to conventional cameras due to their
+desirable properties. SPADs are capable of single-photon sensitivity with
+microsecond temporal resolution, and event cameras can measure brightness
+changes up to 1 MHz with low bandwidth requirements. We show that these
+properties are complementary, and can help achieve low-light, high-speed image
+reconstruction with low bandwidth requirements. We introduce a sensor fusion
+framework to combine SPADs with event cameras to improves the reconstruction of
+high-speed, low-light scenes while reducing the high bandwidth cost associated
+with using every SPAD frame. Our evaluation, on both synthetic and real sensor
+data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing
+low-light scenes at high temporal resolution (100 kHz) compared to conventional
+cameras. Event-SPAD fusion shows great promise for real-world applications,
+such as robotics or medical imaging.",eess.IV,"['eess.IV', 'cs.CV']"
+Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians,Yuelang Xu · Benwang Chen · Zhe Li · Hongwen Zhang · Lizhen Wang · Zerong Zheng · Yebin Liu,https://yuelangx.github.io/gaussianheadavatar,https://arxiv.org/abs/2312.03029,,2312.03029.pdf,Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians,"Creating high-fidelity 3D head avatars has always been a research hotspot,
+but there remains a great challenge under lightweight sparse view setups. In
+this paper, we propose Gaussian Head Avatar represented by controllable 3D
+Gaussians for high-fidelity head avatar modeling. We optimize the neutral 3D
+Gaussians and a fully learned MLP-based deformation field to capture complex
+expressions. The two parts benefit each other, thereby our method can model
+fine-grained dynamic details while ensuring expression accuracy. Furthermore,
+we devise a well-designed geometry-guided initialization strategy based on
+implicit SDF and Deep Marching Tetrahedra for the stability and convergence of
+the training procedure. Experiments show our approach outperforms other
+state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering
+quality at 2K resolution even under exaggerated expressions.",cs.CV,"['cs.CV', 'cs.GR']"
+Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis,Yuchao Gu · Xintao Wang · Yixiao Ge · Ying Shan · Mike Zheng Shou, ,https://ar5iv.labs.arxiv.org/html/2310.01218,,2310.01218.pdf,Making LLaMA SEE and Draw with SEED Tokenizer,"The great success of Large Language Models (LLMs) has expanded the potential
+of multimodality, contributing to the gradual evolution of General Artificial
+Intelligence (AGI). A true AGI agent should not only possess the capability to
+perform predefined multi-tasks but also exhibit emergent abilities in an
+open-world context. However, despite the considerable advancements made by
+recent multimodal LLMs, they still fall short in effectively unifying
+comprehension and generation tasks, let alone open-world emergent abilities. We
+contend that the key to overcoming the present impasse lies in enabling text
+and images to be represented and processed interchangeably within a unified
+autoregressive Transformer. To this end, we introduce SEED, an elaborate image
+tokenizer that empowers LLMs with the ability to SEE and Draw at the same time.
+We identify two crucial design principles: (1) Image tokens should be
+independent of 2D physical patch positions and instead be produced with a 1D
+causal dependency, exhibiting intrinsic interdependence that aligns with the
+left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens
+should capture high-level semantics consistent with the degree of semantic
+abstraction in words, and be optimized for both discriminativeness and
+reconstruction during the tokenizer training phase. With SEED tokens, LLM is
+able to perform scalable multimodal autoregression under its original training
+recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by
+large-scale pretraining and instruction tuning on the interleaved textual and
+visual data, demonstrating impressive performance on a broad range of
+multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has
+exhibited compositional emergent abilities such as multi-turn in-context
+multimodal generation, acting like your AI assistant.",cs.CV,['cs.CV']
+HDRFlow: Real-Time HDR Video Reconstruction with Large Motions,Gangwei Xu · Yujin Wang · Jinwei Gu · Tianfan Xue · Xin Yang, ,https://arxiv.org/abs/2403.03447,,2403.03447.pdf,HDRFlow: Real-Time HDR Video Reconstruction with Large Motions,"Reconstructing High Dynamic Range (HDR) video from image sequences captured
+with alternating exposures is challenging, especially in the presence of large
+camera or object motion. Existing methods typically align low dynamic range
+sequences using optical flow or attention mechanism for deghosting. However,
+they often struggle to handle large complex motions and are computationally
+expensive. To address these challenges, we propose a robust and efficient flow
+estimator tailored for real-time HDR video reconstruction, named HDRFlow.
+HDRFlow has three novel designs: an HDR-domain alignment loss (HALoss), an
+efficient flow network with a multi-size large kernel (MLK), and a new HDR flow
+training scheme. The HALoss supervises our flow network to learn an
+HDR-oriented flow for accurate alignment in saturated and dark regions. The MLK
+can effectively model large motions at a negligible cost. In addition, we
+incorporate synthetic data, Sintel, into our training dataset, utilizing both
+its provided forward flow and backward flow generated by us to supervise our
+flow network, enhancing our performance in large motion regions. Extensive
+experiments demonstrate that our HDRFlow outperforms previous methods on
+standard benchmarks. To the best of our knowledge, HDRFlow is the first
+real-time HDR video reconstruction method for video sequences captured with
+alternating exposures, capable of processing 720p resolution inputs at 25ms.",cs.CV,['cs.CV']
+LiSA: LiDAR Localization with Semantic Awareness,Bochun Yang · Zijun Li · Wen Li · zhipeng cai · Chenglu Wen · Yu Zang · Matthias Mueller · Cheng Wang, ,https://arxiv.org/abs/2402.18934,,2402.18934.pdf,RELEAD: Resilient Localization with Enhanced LiDAR Odometry in Adverse Environments,"LiDAR-based localization is valuable for applications like mining surveys and
+underground facility maintenance. However, existing methods can struggle when
+dealing with uninformative geometric structures in challenging scenarios. This
+paper presents RELEAD, a LiDAR-centric solution designed to address
+scan-matching degradation. Our method enables degeneracy-free point cloud
+registration by solving constrained ESIKF updates in the front end and
+incorporates multisensor constraints, even when dealing with outlier
+measurements, through graph optimization based on Graduated Non-Convexity
+(GNC). Additionally, we propose a robust Incremental Fixed Lag Smoother (rIFL)
+for efficient GNC-based optimization. RELEAD has undergone extensive evaluation
+in degenerate scenarios and has outperformed existing state-of-the-art
+LiDAR-Inertial odometry and LiDAR-Visual-Inertial odometry methods.",cs.RO,['cs.RO']
+Language Models as Black-Box Optimizers for Vision-Language Models,Shihong Liu · Samuel Yu · Zhiqiu Lin · Deepak Pathak · Deva Ramanan,https://llm-can-optimize-vlm.github.io/,https://arxiv.org/abs/2309.05950,,2309.05950.pdf,Language Models as Black-Box Optimizers for Vision-Language Models,"Vision-language models (VLMs) pre-trained on web-scale datasets have
+demonstrated remarkable capabilities on downstream tasks when fine-tuned with
+minimal data. However, many VLMs rely on proprietary data and are not
+open-source, which restricts the use of white-box approaches for fine-tuning.
+As such, we aim to develop a black-box approach to optimize VLMs through
+natural language prompts, thereby avoiding the need to access model parameters,
+feature embeddings, or even output logits. We propose employing chat-based LLMs
+to search for the best text prompt for VLMs. Specifically, we adopt an
+automatic hill-climbing procedure that converges to an effective prompt by
+evaluating the performance of current prompts and asking LLMs to refine them
+based on textual feedback, all within a conversational process without
+human-in-the-loop. In a challenging 1-shot image classification setup, our
+simple approach surpasses the white-box continuous prompting method (CoOp) by
+an average of 1.5% across 11 datasets including ImageNet. Our approach also
+outperforms both human-engineered and LLM-generated prompts. We highlight the
+advantage of conversational feedback that incorporates both positive and
+negative prompts, suggesting that LLMs can utilize the implicit gradient
+direction in textual feedback for a more efficient search. In addition, we find
+that the text prompts generated through our strategy are not only more
+interpretable but also transfer well across different VLM architectures in a
+black-box manner. Lastly, we apply our framework to optimize the
+state-of-the-art black-box VLM (DALL-E 3) for text-to-image generation, prompt
+inversion, and personalization.",cs.CL,"['cs.CL', 'cs.CV', 'cs.LG', 'cs.MM']"
+The Neglected Tails of Vision-Language Models,Shubham Parashar · Tian Liu · Zhiqiu Lin · Xiangjue Dong · Yanan Li · James Caverlee · Deva Ramanan · Shu Kong,https://shubhamprshr27.github.io/neglected-tails-of-vlms/,https://arxiv.org/abs/2401.12425,,2401.12425.pdf,The Neglected Tails in Vision-Language Models,"Vision-language models (VLMs) excel in zero-shot recognition but their
+performance varies greatly across different visual concepts. For example,
+although CLIP achieves impressive accuracy on ImageNet (60-80%), its
+performance drops below 10% for more than ten concepts like night snake,
+presumably due to their limited presence in the pretraining data. However,
+measuring the frequency of concepts in VLMs' large-scale datasets is
+challenging. We address this by using large language models (LLMs) to count the
+number of pretraining texts that contain synonyms of these concepts. Our
+analysis confirms that popular datasets, such as LAION, exhibit a long-tailed
+concept distribution, yielding biased performance in VLMs. We also find that
+downstream applications of VLMs, including visual chatbots (e.g., GPT-4V) and
+text-to-image models (e.g., Stable Diffusion), often fail to recognize or
+generate images of rare concepts identified by our method. To mitigate the
+imbalanced performance of zero-shot VLMs, we propose REtrieval-Augmented
+Learning (REAL). First, instead of prompting VLMs using the original class
+names, REAL uses their most frequent synonyms found in pretraining texts. This
+simple change already outperforms costly human-engineered and LLM-enriched
+prompts over nine benchmark datasets. Second, REAL trains a linear classifier
+on a small yet balanced set of pretraining data retrieved using concept
+synonyms. REAL surpasses the previous zero-shot SOTA, using 400x less storage
+and 10,000x less training time!",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data,Chengxiang Fan · Muzhi Zhu · Hao Chen · Yang Liu · Weijia Wu · Huaqi Zhang · Chunhua Shen,https://github.com/aim-uofa/DiverGen,https://arxiv.org/abs/2405.10185,,2405.10185.pdf,DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data,"Instance segmentation is data-hungry, and as model capacity increases, data
+scale becomes crucial for improving the accuracy. Most instance segmentation
+datasets today require costly manual annotation, limiting their data scale.
+Models trained on such data are prone to overfitting on the training set,
+especially for those rare categories. While recent works have delved into
+exploiting generative models to create synthetic datasets for data
+augmentation, these approaches do not efficiently harness the full potential of
+generative models.
+  To address these issues, we introduce a more efficient strategy to construct
+generative datasets for data augmentation, termed DiverGen. Firstly, we provide
+an explanation of the role of generative data from the perspective of
+distribution discrepancy. We investigate the impact of different data on the
+distribution learned by the model. We argue that generative data can expand the
+data distribution that the model can learn, thus mitigating overfitting.
+Additionally, we find that the diversity of generative data is crucial for
+improving model performance and enhance it through various strategies,
+including category diversity, prompt diversity, and generative model diversity.
+With these strategies, we can scale the data to millions while maintaining the
+trend of model performance improvement. On the LVIS dataset, DiverGen
+significantly outperforms the strong model X-Paste, achieving +1.1 box AP and
++1.1 mask AP across all categories, and +1.9 box AP and +2.5 mask AP for rare
+categories.",cs.CV,['cs.CV']
+Extend Your Own Correspondences: Unsupervised Distant Point Cloud Registration by Progressive Distance Extension,Quan Liu · Hongzi Zhu · Zhenxi Wang · Yunsong Zhou · Shan Chang · Minyi Guo, ,https://arxiv.org/abs/2403.03532,,2403.03532.pdf,Extend Your Own Correspondences: Unsupervised Distant Point Cloud Registration by Progressive Distance Extension,"Registration of point clouds collected from a pair of distant vehicles
+provides a comprehensive and accurate 3D view of the driving scenario, which is
+vital for driving safety related applications, yet existing literature suffers
+from the expensive pose label acquisition and the deficiency to generalize to
+new data distributions. In this paper, we propose EYOC, an unsupervised distant
+point cloud registration method that adapts to new point cloud distributions on
+the fly, requiring no global pose labels. The core idea of EYOC is to train a
+feature extractor in a progressive fashion, where in each round, the feature
+extractor, trained with near point cloud pairs, can label slightly farther
+point cloud pairs, enabling self-supervision on such far point cloud pairs.
+This process continues until the derived extractor can be used to register
+distant point clouds. Particularly, to enable high-fidelity correspondence
+label generation, we devise an effective spatial filtering scheme to select the
+most representative correspondences to register a point cloud pair, and then
+utilize the aligned point clouds to discover more correct correspondences.
+Experiments show that EYOC can achieve comparable performance with
+state-of-the-art supervised methods at a lower training cost. Moreover, it
+outwits supervised methods regarding generalization performance on new data
+distributions.",cs.CV,['cs.CV']
+Fine-grained Bipartite Concept Factorization for Clustering,Chong Peng · Pengfei Zhang · Yongyong Chen · zhao kang · Chenglizhao Chen · Qiang Cheng, ,,https://ieeexplore.ieee.org/document/10506642,,,,,nan
+SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration,Xu Cao · Takafumi Taketomi, ,https://arxiv.org/abs/2312.04803,,2312.04803.pdf,SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration,"We present SuperNormal, a fast, high-fidelity approach to multi-view 3D
+reconstruction using surface normal maps. With a few minutes, SuperNormal
+produces detailed surfaces on par with 3D scanners. We harness volume rendering
+to optimize a neural signed distance function (SDF) powered by multi-resolution
+hash encoding. To accelerate training, we propose directional finite difference
+and patch-based ray marching to approximate the SDF gradients numerically.
+While not compromising reconstruction quality, this strategy is nearly twice as
+efficient as analytical gradients and about three times faster than
+axis-aligned finite difference. Experiments on the benchmark dataset
+demonstrate the superiority of SuperNormal in efficiency and accuracy compared
+to existing multi-view photometric stereo methods. On our captured objects,
+SuperNormal produces more fine-grained geometry than recent neural 3D
+reconstruction methods.",cs.CV,['cs.CV']
+SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers,Jonathan F. Carter · Joao Jorge · Oliver Gibson · Lionel Tarassenko, ,https://arxiv.org/abs/2404.03831,,2404.03831.pdf,SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers,"Advances in camera-based physiological monitoring have enabled the robust,
+non-contact measurement of respiration and the cardiac pulse, which are known
+to be indicative of the sleep stage. This has led to research into camera-based
+sleep monitoring as a promising alternative to ""gold-standard"" polysomnography,
+which is cumbersome, expensive to administer, and hence unsuitable for
+longer-term clinical studies. In this paper, we introduce SleepVST, a
+transformer model which enables state-of-the-art performance in camera-based
+sleep stage classification (sleep staging). After pre-training on contact
+sensor data, SleepVST outperforms existing methods for cardio-respiratory sleep
+staging on the SHHS and MESA datasets, achieving total Cohen's kappa scores of
+0.75 and 0.77 respectively. We then show that SleepVST can be successfully
+transferred to cardio-respiratory waveforms extracted from video, enabling
+fully contact-free sleep staging. Using a video dataset of 50 nights, we
+achieve a total accuracy of 78.8\% and a Cohen's $\kappa$ of 0.71 in four-class
+video-based sleep staging, setting a new state-of-the-art in the domain.",cs.CV,"['cs.CV', 'cs.HC', 'q-bio.NC']"
+Progress-Aware Online Action Segmentation for Egocentric Procedural Task Videos,Yuhan Shen · Ehsan Elhamifar, ,https://arxiv.org/abs/2404.01933,,,PREGO: online mistake detection in PRocedural EGOcentric videos,"Promptly identifying procedural errors from egocentric videos in an online
+setting is highly challenging and valuable for detecting mistakes as soon as
+they happen. This capability has a wide range of applications across various
+fields, such as manufacturing and healthcare. The nature of procedural mistakes
+is open-set since novel types of failures might occur, which calls for
+one-class classifiers trained on correctly executed procedures. However, no
+technique can currently detect open-set procedural mistakes online. We propose
+PREGO, the first online one-class classification model for mistake detection in
+PRocedural EGOcentric videos. PREGO is based on an online action recognition
+component to model the current action, and a symbolic reasoning module to
+predict the next actions. Mistake detection is performed by comparing the
+recognized current action with the expected future one. We evaluate PREGO on
+two procedural egocentric video datasets, Assembly101 and Epic-tent, which we
+adapt for online benchmarking of procedural mistake detection to establish
+suitable benchmarks, thus defining the Assembly101-O and Epic-tent-O datasets,
+respectively.",cs.CV,['cs.CV']
+Efficient Solution of Point-Line Absolute Pose,Petr Hruby · Timothy Duff · Marc Pollefeys,https://github.com/petrhruby97/efficient_absolute,https://arxiv.org/abs/2404.16552,,2404.16552.pdf,Efficient Solution of Point-Line Absolute Pose,"We revisit certain problems of pose estimation based on 3D--2D
+correspondences between features which may be points or lines. Specifically, we
+address the two previously-studied minimal problems of estimating camera
+extrinsics from $p \in \{ 1, 2 \}$ point--point correspondences and $l=3-p$
+line--line correspondences. To the best of our knowledge, all of the
+previously-known practical solutions to these problems required computing the
+roots of degree $\ge 4$ (univariate) polynomials when $p=2$, or degree $\ge 8$
+polynomials when $p=1.$ We describe and implement two elementary solutions
+which reduce the degrees of the needed polynomials from $4$ to $2$ and from $8$
+to $4$, respectively. We show experimentally that the resulting solvers are
+numerically stable and fast: when compared to the previous state-of-the art, we
+may obtain nearly an order of magnitude speedup. The code is available at
+\url{https://github.com/petrhruby97/efficient\_absolute}",cs.CV,"['cs.CV', '68T45', 'I.4.5']"
+ProTeCt: Prompt Tuning for Taxonomic Open Set Classification,Tz-Ying Wu · Chih-Hui Ho · Nuno Vasconcelos,http://www.svcl.ucsd.edu/projects/protect/,https://arxiv.org/abs/2306.02240,,2306.02240.pdf,ProTeCt: Prompt Tuning for Taxonomic Open Set Classification,"Visual-language foundation models, like CLIP, learn generalized
+representations that enable zero-shot open-set classification. Few-shot
+adaptation methods, based on prompt tuning, have been shown to further improve
+performance on downstream datasets. However, these methods do not fare well in
+the taxonomic open set (TOS) setting, where the classifier is asked to make
+predictions from label sets across different levels of semantic granularity.
+Frequently, they infer incorrect labels at coarser taxonomic class levels, even
+when the inference at the leaf level (original class labels) is correct. To
+address this problem, we propose a prompt tuning technique that calibrates the
+hierarchical consistency of model predictions. A set of metrics of hierarchical
+consistency, the Hierarchical Consistent Accuracy (HCA) and the Mean Treecut
+Accuracy (MTA), are first proposed to evaluate TOS model performance. A new
+Prompt Tuning for Hierarchical Consistency (ProTeCt) technique is then proposed
+to calibrate classification across label set granularities. Results show that
+ProTeCt can be combined with existing prompt tuning methods to significantly
+improve TOS classification without degrading the leaf level classification
+performance.",cs.CV,['cs.CV']
+In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing,Yiran Xu · Zhixin Shu · Cameron Smith · Seoung Wug Oh · Jia-Bin Huang,https://in-n-out-3d.github.io/,,https://www.youtube.com/watch?v=JGbLEEANtnI,,,,,nan
+On the Faithfulness of Vision Transformer Explanations,Junyi Wu · Weitai Kang · Hao Tang · Yuan Hong · Yan Yan, ,https://arxiv.org/abs/2404.01415,,2404.01415.pdf,On the Faithfulness of Vision Transformer Explanations,"To interpret Vision Transformers, post-hoc explanations assign salience
+scores to input pixels, providing human-understandable heatmaps. However,
+whether these interpretations reflect true rationales behind the model's output
+is still underexplored. To address this gap, we study the faithfulness
+criterion of explanations: the assigned salience scores should represent the
+influence of the corresponding input pixels on the model's predictions. To
+evaluate faithfulness, we introduce Salience-guided Faithfulness Coefficient
+(SaCo), a novel evaluation metric leveraging essential information of salience
+distribution. Specifically, we conduct pair-wise comparisons among distinct
+pixel groups and then aggregate the differences in their salience scores,
+resulting in a coefficient that indicates the explanation's degree of
+faithfulness. Our explorations reveal that current metrics struggle to
+differentiate between advanced explanation methods and Random Attribution,
+thereby failing to capture the faithfulness property. In contrast, our proposed
+SaCo offers a reliable faithfulness measurement, establishing a robust metric
+for interpretations. Furthermore, our SaCo demonstrates that the use of
+gradient and multi-layer aggregation can markedly enhance the faithfulness of
+attention-based explanation, shedding light on potential paths for advancing
+Vision Transformer explainability.",cs.CV,['cs.CV']
+Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models,Gihyun Kwon · Simon Jenni · Ding Li · Joon-Young Lee · Jong Chul Ye · Fabian Caba Heilbron, ,https://arxiv.org/abs/2404.03913,,2404.03913.pdf,Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models,"While there has been significant progress in customizing text-to-image
+generation models, generating images that combine multiple personalized
+concepts remains challenging. In this work, we introduce Concept Weaver, a
+method for composing customized text-to-image diffusion models at inference
+time. Specifically, the method breaks the process into two steps: creating a
+template image aligned with the semantics of input prompts, and then
+personalizing the template using a concept fusion strategy. The fusion strategy
+incorporates the appearance of the target concepts into the template image
+while retaining its structural details. The results indicate that our method
+can generate multiple custom concepts with higher identity fidelity compared to
+alternative approaches. Furthermore, the method is shown to seamlessly handle
+more than two concepts and closely follow the semantic meaning of the input
+prompt without blending appearances across different subjects.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+SPAD: Spatially Aware Multiview Diffusers,Yash Kant · Aliaksandr Siarohin · Ziyi Wu · Michael Vasilkovsky · Guocheng Qian · Jian Ren · Riza Alp Guler · Bernard Ghanem · Sergey Tulyakov · Igor Gilitschenski,https://yashkant.github.io/spad,https://arxiv.org/abs/2402.05235,,2402.05235.pdf,SPAD : Spatially Aware Multiview Diffusers,"We present SPAD, a novel approach for creating consistent multi-view images
+from text prompts or single images. To enable multi-view generation, we
+repurpose a pretrained 2D diffusion model by extending its self-attention
+layers with cross-view interactions, and fine-tune it on a high quality subset
+of Objaverse. We find that a naive extension of the self-attention proposed in
+prior work (e.g. MVDream) leads to content copying between views. Therefore, we
+explicitly constrain the cross-view attention based on epipolar geometry. To
+further enhance 3D consistency, we utilize Plucker coordinates derived from
+camera rays and inject them as positional encoding. This enables SPAD to reason
+over spatial proximity in 3D well. In contrast to recent works that can only
+generate views at fixed azimuth and elevation, SPAD offers full camera control
+and achieves state-of-the-art results in novel view synthesis on unseen objects
+from the Objaverse and Google Scanned Objects datasets. Finally, we demonstrate
+that text-to-3D generation using SPAD prevents the multi-face Janus issue. See
+more details at our webpage: https://yashkant.github.io/spad",cs.CV,['cs.CV']
+Content-Style Decoupling for Unsupervised Makeup Transfer without Generating Pseudo Ground Truth,Zhaoyang Sun · Shengwu Xiong · Yaxiong Chen · Yi Rong, ,https://arxiv.org/abs/2405.17240,,2405.17240.pdf,Content-Style Decoupling for Unsupervised Makeup Transfer without Generating Pseudo Ground Truth,"The absence of real targets to guide the model training is one of the main
+problems with the makeup transfer task. Most existing methods tackle this
+problem by synthesizing pseudo ground truths (PGTs). However, the generated
+PGTs are often sub-optimal and their imprecision will eventually lead to
+performance degradation. To alleviate this issue, in this paper, we propose a
+novel Content-Style Decoupled Makeup Transfer (CSD-MT) method, which works in a
+purely unsupervised manner and thus eliminates the negative effects of
+generating PGTs. Specifically, based on the frequency characteristics analysis,
+we assume that the low-frequency (LF) component of a face image is more
+associated with its makeup style information, while the high-frequency (HF)
+component is more related to its content details. This assumption allows CSD-MT
+to decouple the content and makeup style information in each face image through
+the frequency decomposition. After that, CSD-MT realizes makeup transfer by
+maximizing the consistency of these two types of information between the
+transferred result and input images, respectively. Two newly designed loss
+functions are also introduced to further improve the transfer performance.
+Extensive quantitative and qualitative analyses show the effectiveness of our
+CSD-MT method. Our code is available at
+https://github.com/Snowfallingplum/CSD-MT.",cs.CV,['cs.CV']
+SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimation,Yamei Chen · Yan Di · Guangyao Zhai · Fabian Manhardt · Chenyangguang Zhang · Ruida Zhang · Federico Tombari · Nassir Navab · Benjamin Busam, ,https://arxiv.org/abs/2311.11125,,2311.11125.pdf,SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimation,"Category-level object pose estimation, aiming to predict the 6D pose and 3D
+size of objects from known categories, typically struggles with large
+intra-class shape variation. Existing works utilizing mean shapes often fall
+short of capturing this variation. To address this issue, we present
+SecondPose, a novel approach integrating object-specific geometric features
+with semantic category priors from DINOv2. Leveraging the advantage of DINOv2
+in providing SE(3)-consistent semantic features, we hierarchically extract two
+types of SE(3)-invariant geometric features to further encapsulate
+local-to-global object-specific information. These geometric features are then
+point-aligned with DINOv2 features to establish a consistent object
+representation under SE(3) transformations, facilitating the mapping from
+camera space to the pre-defined canonical space, thus further enhancing pose
+estimation. Extensive experiments on NOCS-REAL275 demonstrate that SecondPose
+achieves a 12.4% leap forward over the state-of-the-art. Moreover, on a more
+complex dataset HouseCat6D which provides photometrically challenging objects,
+SecondPose still surpasses other competitors by a large margin.",cs.CV,['cs.CV']
+Rethinking FID: Towards a Better Evaluation Metric for Image Generation,Sadeep Jayasumana · Srikumar Ramalingam · Andreas Veit · Daniel Glasner · Ayan Chakrabarti · Sanjiv Kumar, ,https://arxiv.org/abs/2401.09603,,2401.09603.pdf,Rethinking FID: Towards a Better Evaluation Metric for Image Generation,"As with many machine learning problems, the progress of image generation
+methods hinges on good evaluation metrics. One of the most popular is the
+Frechet Inception Distance (FID). FID estimates the distance between a
+distribution of Inception-v3 features of real images, and those of images
+generated by the algorithm. We highlight important drawbacks of FID:
+Inception's poor representation of the rich and varied content generated by
+modern text-to-image models, incorrect normality assumptions, and poor sample
+complexity. We call for a reevaluation of FID's use as the primary quality
+metric for generated images. We empirically demonstrate that FID contradicts
+human raters, it does not reflect gradual improvement of iterative
+text-to-image models, it does not capture distortion levels, and that it
+produces inconsistent results when varying the sample size. We also propose an
+alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
+mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
+estimator that does not make any assumptions on the probability distribution of
+the embeddings and is sample efficient. Through extensive experiments and
+analysis, we demonstrate that FID-based evaluations of text-to-image models may
+be unreliable, and that CMMD offers a more robust and reliable assessment of
+image quality.",cs.CV,['cs.CV']
+CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation,Xi Liu · Ying Guo · Cheng Zhen · Tong Li · Yingying Ao · Pengfei Yan,https://customlistener.github.io/,https://arxiv.org/abs/2403.00274,,2403.00274.pdf,CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation,"Listening head generation aims to synthesize a non-verbal responsive listener
+head by modeling the correlation between the speaker and the listener in
+dynamic conversion.The applications of listener agent generation in virtual
+interaction have promoted many works achieving the diverse and fine-grained
+motion generation. However, they can only manipulate motions through simple
+emotional labels, but cannot freely control the listener's motions. Since
+listener agents should have human-like attributes (e.g. identity, personality)
+which can be freely customized by users, this limits their realism. In this
+paper, we propose a user-friendly framework called CustomListener to realize
+the free-form text prior guided listener generation. To achieve
+speaker-listener coordination, we design a Static to Dynamic Portrait module
+(SDP), which interacts with speaker information to transform static text into
+dynamic portrait token with completion rhythm and amplitude information. To
+achieve coherence between segments, we design a Past Guided Generation Module
+(PGG) to maintain the consistency of customized listener attributes through the
+motion prior, and utilize a diffusion-based structure conditioned on the
+portrait token and the motion prior to realize the controllable generation. To
+train and evaluate our model, we have constructed two text-annotated listening
+head datasets based on ViCo and RealTalk, which provide text-video paired
+labels. Extensive experiments have verified the effectiveness of our model.",cs.CV,"['cs.CV', 'cs.SD', 'eess.AS']"
+Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields,TIANQI LIU · Xinyi Ye · Min Shi · Zihao Huang · Zhiyu Pan · Zhan Peng · Zhiguo Cao, ,https://arxiv.org/abs/2404.17528,,2404.17528.pdf,Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields,"Generalizable NeRF aims to synthesize novel views for unseen scenes. Common
+practices involve constructing variance-based cost volumes for geometry
+reconstruction and encoding 3D descriptors for decoding novel views. However,
+existing methods show limited generalization ability in challenging conditions
+due to inaccurate geometry, sub-optimal descriptors, and decoding strategies.
+We address these issues point by point. First, we find the variance-based cost
+volume exhibits failure patterns as the features of pixels corresponding to the
+same point can be inconsistent across different views due to occlusions or
+reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to
+amplify the contribution of consistent pixel pairs and suppress inconsistent
+ones. Unlike previous methods that solely fuse 2D features into descriptors,
+our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D
+context into descriptors through spatial and inter-view interaction. When
+decoding the descriptors, we observe the two existing decoding strategies excel
+in different areas, which are complementary. A Consistency-Aware Fusion (CAF)
+strategy is proposed to leverage the advantages of both. We incorporate the
+above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware
+Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains
+state-of-the-art performance across multiple datasets. Code is available at
+https://github.com/TQTQliu/GeFu .",cs.CV,['cs.CV']
+Rethinking Few-shot 3D Point Cloud Semantic Segmentation,Zhaochong An · Guolei Sun · Yun Liu · Fayao Liu · Zongwei Wu · Dan Wang · Luc Van Gool · Serge Belongie, ,https://arxiv.org/abs/2403.00592,,2403.00592.pdf,Rethinking Few-shot 3D Point Cloud Semantic Segmentation,"This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS),
+with a focus on two significant issues in the state-of-the-art: foreground
+leakage and sparse point distribution. The former arises from non-uniform point
+sampling, allowing models to distinguish the density disparities between
+foreground and background for easier segmentation. The latter results from
+sampling only 2,048 points, limiting semantic information and deviating from
+the real-world practice. To address these issues, we introduce a standardized
+FS-PCS setting, upon which a new benchmark is built. Moreover, we propose a
+novel FS-PCS model. While previous methods are based on feature optimization by
+mainly refining support features to enhance prototypes, our method is based on
+correlation optimization, referred to as Correlation Optimization Segmentation
+(COSeg). Specifically, we compute Class-specific Multi-prototypical Correlation
+(CMC) for each query point, representing its correlations to category
+prototypes. Then, we propose the Hyper Correlation Augmentation (HCA) module to
+enhance CMC. Furthermore, tackling the inherent property of few-shot training
+to incur base susceptibility for models, we propose to learn non-parametric
+prototypes for the base classes during training. The learned base prototypes
+are used to calibrate correlations for the background class through a Base
+Prototypes Calibration (BPC) module. Experiments on popular datasets
+demonstrate the superiority of COSeg over existing methods. The code is
+available at: https://github.com/ZhaochongAn/COSeg",cs.CV,['cs.CV']
+MESA: Matching Everything by Segmenting Anything,Yesheng Zhang · Xu Zhao, ,https://arxiv.org/abs/2401.16741v1,,2401.16741v1.pdf,MESA: Matching Everything by Segmenting Anything,"Feature matching is a crucial task in the field of computer vision, which
+involves finding correspondences between images. Previous studies achieve
+remarkable performance using learning-based feature comparison. However, the
+pervasive presence of matching redundancy between images gives rise to
+unnecessary and error-prone computations in these methods, imposing limitations
+on their accuracy. To address this issue, we propose MESA, a novel approach to
+establish precise area (or region) matches for efficient matching redundancy
+reduction. MESA first leverages the advanced image understanding capability of
+SAM, a state-of-the-art foundation model for image segmentation, to obtain
+image areas with implicit semantic. Then, a multi-relational graph is proposed
+to model the spatial structure of these areas and construct their scale
+hierarchy. Based on graphical models derived from the graph, the area matching
+is reformulated as an energy minimization task and effectively resolved.
+Extensive experiments demonstrate that MESA yields substantial precision
+improvement for multiple point matchers in indoor and outdoor downstream tasks,
+e.g. +13.61% for DKM in indoor pose estimation.",cs.CV,['cs.CV']
+DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets,Harsh Rangwani · Pradipto Mondal · Mayank Mishra · Ashish Asokan · R. Venkatesh Babu,https://rangwani-harsh.github.io/DeiT-LT/,https://arxiv.org/abs/2404.02900,,2404.02900.pdf,DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets,"Vision Transformer (ViT) has emerged as a prominent architecture for various
+computer vision tasks. In ViT, we divide the input image into patch tokens and
+process them through a stack of self attention blocks. However, unlike
+Convolutional Neural Networks (CNN), ViTs simple architecture has no
+informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a
+large amount of data for pre-training. Various data efficient approaches (DeiT)
+have been proposed to train ViT on balanced datasets effectively. However,
+limited literature discusses the use of ViT for datasets with long-tailed
+imbalances. In this work, we introduce DeiT-LT to tackle the problem of
+training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an
+efficient and effective way of distillation from CNN via distillation DIST
+token by using out-of-distribution images and re-weighting the distillation
+loss to enhance focus on tail classes. This leads to the learning of local
+CNN-like features in early ViT blocks, improving generalization for tail
+classes. Further, to mitigate overfitting, we propose distilling from a flat
+CNN teacher, which leads to learning low-rank generalizable features for DIST
+tokens across all ViT blocks. With the proposed DeiT-LT scheme, the
+distillation DIST token becomes an expert on the tail classes, and the
+classifier CLS token becomes an expert on the head classes. The experts help to
+effectively learn features corresponding to both the majority and minority
+classes using a distinct set of tokens within the same ViT architecture. We
+show the effectiveness of DeiT-LT for training ViT from scratch on datasets
+ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis,Jiapeng Tang · Yinyu Nie · Lev Markhasin · Angela Dai · Justus Thies · Matthias Nießner,https://tangjiapeng.github.io/projects/DiffuScene/,,https://justusthies.github.io/posts/diffuscene/,,,,,nan
+TokenCompose: Text-to-Image Diffusion with Token-level Supervision,Zirui Wang · Zhizhou Sha · Zheng Ding · Yilin Wang · Zhuowen Tu,https://mlpc-ucsd.github.io/TokenCompose/,https://arxiv.org/abs/2312.03626,,2312.03626.pdf,TokenCompose: Grounding Diffusion with Token-level Supervision,"We present TokenCompose, a Latent Diffusion Model for text-to-image
+generation that achieves enhanced consistency between user-specified text
+prompts and model-generated images. Despite its tremendous success, the
+standard denoising process in the Latent Diffusion Model takes text prompts as
+conditions only, absent explicit constraint for the consistency between the
+text prompts and the image contents, leading to unsatisfactory results for
+composing multiple object categories. TokenCompose aims to improve
+multi-category instance composition by introducing the token-wise consistency
+terms between the image content and object segmentation maps in the finetuning
+stage. TokenCompose can be applied directly to the existing training pipeline
+of text-conditioned diffusion models without extra human labeling information.
+By finetuning Stable Diffusion, the model exhibits significant improvements in
+multi-category instance composition and enhanced photorealism for its generated
+images.",cs.CV,['cs.CV']
+Unbiased Estimator for Distorted Conic in Camera Calibration,Chaehyeon Song · Jaeho Shin · Myung-Hwan Jeon · Jongwoo Lim · Ayoung Kim,https://github.com/chaehyeonsong/discocal,https://arxiv.org/abs/2403.04583,,2403.04583.pdf,Unbiased Estimator for Distorted Conics in Camera Calibration,"In the literature, points and conics have been major features for camera
+geometric calibration. Although conics are more informative features than
+points, the loss of the conic property under distortion has critically limited
+the utility of conic features in camera calibration. Many existing approaches
+addressed conic-based calibration by ignoring distortion or introducing 3D
+spherical targets to circumvent this limitation. In this paper, we present a
+novel formulation for conic-based calibration using moments. Our derivation is
+based on the mathematical finding that the first moment can be estimated
+without bias even under distortion. This allows us to track moment changes
+during projection and distortion, ensuring the preservation of the first moment
+of the distorted conic. With an unbiased estimator, the circular patterns can
+be accurately detected at the sub-pixel level and can now be fully exploited
+for an entire calibration pipeline, resulting in significantly improved
+calibration. The entire code is readily available from
+https://github.com/ChaehyeonSong/discocal.",cs.CV,['cs.CV']
+Unleashing Channel Potential: Space-Frequency Selection Convolution for SAR Object Detection,Ke Li · Di Wang · Zhangyuan Hu · Wenxuan Zhu · Shaofeng Li · Quan Wang, ,https://arxiv.org/abs/2312.16943,,2312.16943.pdf,Multi-scale direction-aware SAR object detection network via global information fusion,"Deep learning has driven significant progress in object detection using
+Synthetic Aperture Radar (SAR) imagery. Existing methods, while achieving
+promising results, often struggle to effectively integrate local and global
+information, particularly direction-aware features. This paper proposes
+SAR-Net, a novel framework specifically designed for global fusion of
+direction-aware information in SAR object detection. SAR-Net leverages two key
+innovations: the Unity Compensation Mechanism (UCM) and the Direction-aware
+Attention Module (DAM). UCM facilitates the establishment of complementary
+relationships among features across different scales, enabling efficient global
+information fusion and transmission. Additionally, DAM, through bidirectional
+attention polymerization, captures direction-aware information, effectively
+eliminating background interference. Extensive experiments demonstrate the
+effectiveness of SAR-Net, achieving state-of-the-art results on aircraft
+(SAR-AIRcraft-1.0) and ship datasets (SSDD, HRSID), confirming its
+generalization capability and robustness.",cs.CV,['cs.CV']
+FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models,LIn Zhao · Tianchen Zhao · Zinan Lin · Xuefei Ning · Guohao Dai · Huazhong Yang · Yu Wang, ,https://arxiv.org/abs/2403.16379,,2403.16379.pdf,FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models,"In recent years, there has been significant progress in the development of
+text-to-image generative models. Evaluating the quality of the generative
+models is one essential step in the development process. Unfortunately, the
+evaluation process could consume a significant amount of computational
+resources, making the required periodic evaluation of model performance (e.g.,
+monitoring training progress) impractical. Therefore, we seek to improve the
+evaluation efficiency by selecting the representative subset of the text-image
+dataset. We systematically investigate the design choices, including the
+selection criteria (textural features or image-based metrics) and the selection
+granularity (prompt-level or set-level). We find that the insights from prior
+work on subset selection for training data do not generalize to this problem,
+and we propose FlashEval, an iterative search algorithm tailored to evaluation
+data selection. We demonstrate the effectiveness of FlashEval on ranking
+diffusion models with various configurations, including architectures,
+quantization levels, and sampler schedules on COCO and DiffusionDB datasets.
+Our searched 50-item subset could achieve comparable evaluation quality to the
+randomly sampled 500-item subset for COCO annotations on unseen models,
+achieving a 10x evaluation speedup. We release the condensed subset of these
+commonly used datasets to help facilitate diffusion algorithm design and
+evaluation, and open-source FlashEval as a tool for condensing future datasets,
+accessible at https://github.com/thu-nics/FlashEval.",cs.CV,['cs.CV']
+Fair-VPT: Fair Visual Prompt Tuning for Image Classification,Sungho Park · Hyeran Byun, ,https://arxiv.org/abs/2404.05207,,2404.05207.pdf,iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connection,"Recent progress has shown great potential of visual prompt tuning (VPT) when
+adapting pre-trained vision transformers to various downstream tasks. However,
+most existing solutions independently optimize prompts at each layer, thereby
+neglecting the usage of task-relevant information encoded in prompt tokens
+across layers. Additionally, existing prompt structures are prone to
+interference from task-irrelevant noise in input images, which can do harm to
+the sharing of task-relevant information. In this paper, we propose a novel VPT
+approach, \textbf{iVPT}. It innovatively incorporates a cross-layer dynamic
+connection (CDC) for input prompt tokens from adjacent layers, enabling
+effective sharing of task-relevant information. Furthermore, we design a
+dynamic aggregation (DA) module that facilitates selective sharing of
+information between layers. The combination of CDC and DA enhances the
+flexibility of the attention process within the VPT framework. Building upon
+these foundations, iVPT introduces an attentive reinforcement (AR) mechanism,
+by automatically identifying salient image tokens, which are further enhanced
+by prompt tokens in an additive manner. Extensive experiments on 24 image
+classification and semantic segmentation benchmarks clearly demonstrate the
+advantage of the proposed iVPT, compared to the state-of-the-art counterparts.",cs.CV,['cs.CV']
+"Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion",Junjiao Tian · Lavisha Aggarwal · Andrea Colaco · Zsolt Kira · Mar Gonzalez-Franco,https://sites.google.com/view/diffseg,https://arxiv.org/abs/2308.12469,,2308.12469.pdf,"Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion","Producing quality segmentation masks for images is a fundamental problem in
+computer vision. Recent research has explored large-scale supervised training
+to enable zero-shot segmentation on virtually any image style and unsupervised
+training to enable segmentation without dense annotations. However,
+constructing a model capable of segmenting anything in a zero-shot manner
+without any annotations is still challenging. In this paper, we propose to
+utilize the self-attention layers in stable diffusion models to achieve this
+goal because the pre-trained stable diffusion model has learned inherent
+concepts of objects within its attention layers. Specifically, we introduce a
+simple yet effective iterative merging process based on measuring KL divergence
+among attention maps to merge them into valid segmentation masks. The proposed
+method does not require any training or language dependency to extract quality
+segmentation for any images. On COCO-Stuff-27, our method surpasses the prior
+unsupervised zero-shot SOTA method by an absolute 26% in pixel accuracy and 17%
+in mean IoU. The project page is at
+\url{https://sites.google.com/view/diffseg/home}.",cs.CV,['cs.CV']
+DPHMs: Diffusion Parametric Head Models for Depth-based Tracking,Jiapeng Tang · Angela Dai · Yinyu Nie · Lev Markhasin · Justus Thies · Matthias Nießner,https://tangjiapeng.github.io/projects/DPHMs/,https://arxiv.org/abs/2312.01068,,2312.01068.pdf,DPHMs: Diffusion Parametric Head Models for Depth-based Tracking,"We introduce Diffusion Parametric Head Models (DPHMs), a generative model
+that enables robust volumetric head reconstruction and tracking from monocular
+depth sequences. While recent volumetric head models, such as NPHMs, can now
+excel in representing high-fidelity head geometries, tracking and
+reconstructing heads from real-world single-view depth sequences remains very
+challenging, as the fitting to partial and noisy observations is
+underconstrained. To tackle these challenges, we propose a latent
+diffusion-based prior to regularize volumetric head reconstruction and
+tracking. This prior-based regularizer effectively constrains the identity and
+expression codes to lie on the underlying latent manifold which represents
+plausible head shapes. To evaluate the effectiveness of the diffusion-based
+prior, we collect a dataset of monocular Kinect sequences consisting of various
+complex facial expression motions and rapid transitions. We compare our method
+to state-of-the-art tracking methods and demonstrate improved head identity
+reconstruction as well as robust expression tracking.",cs.CV,['cs.CV']
+A Unified Approach for Text- and Image-guided 4D Scene Generation,Yufeng Zheng · Xueting Li · Koki Nagano · Sifei Liu · Otmar Hilliges · Shalini De Mello, ,https://arxiv.org/abs/2311.16854,,2311.16854.pdf,A Unified Approach for Text- and Image-guided 4D Scene Generation,"Large-scale diffusion generative models are greatly simplifying image, video
+and 3D asset creation from user-provided text prompts and images. However, the
+challenging problem of text-to-4D dynamic 3D scene generation with diffusion
+guidance remains largely unexplored. We propose Dream-in-4D, which features a
+novel two-stage approach for text-to-4D synthesis, leveraging (1) 3D and 2D
+diffusion guidance to effectively learn a high-quality static 3D asset in the
+first stage; (2) a deformable neural radiance field that explicitly
+disentangles the learned static asset from its deformation, preserving quality
+during motion learning; and (3) a multi-resolution feature grid for the
+deformation field with a displacement total variation loss to effectively learn
+motion with video diffusion guidance in the second stage. Through a user
+preference study, we demonstrate that our approach significantly advances image
+and motion quality, 3D consistency and text fidelity for text-to-4D generation
+compared to baseline approaches. Thanks to its motion-disentangled
+representation, Dream-in-4D can also be easily adapted for controllable
+generation where appearance is defined by one or multiple images, without the
+need to modify the motion learning stage. Thus, our method offers, for the
+first time, a unified approach for text-to-4D, image-to-4D and personalized 4D
+generation tasks.",cs.CV,['cs.CV']
+Continuous Pose for Monocular Cameras in Neural Implicit Representation,Qi Ma · Danda Paudel · Ajad Chhatkuli · Luc Van Gool,https://github.com/qimaqi/Continuous-Pose-in-NeRF,https://arxiv.org/abs/2311.17119,,2311.17119.pdf,Continuous Pose for Monocular Cameras in Neural Implicit Representation,"In this paper, we showcase the effectiveness of optimizing monocular camera
+poses as a continuous function of time. The camera poses are represented using
+an implicit neural function which maps the given time to the corresponding
+camera pose. The mapped camera poses are then used for the downstream tasks
+where joint camera pose optimization is also required. While doing so, the
+network parameters -- that implicitly represent camera poses -- are optimized.
+We exploit the proposed method in four diverse experimental settings, namely,
+(1) NeRF from noisy poses; (2) NeRF from asynchronous Events; (3) Visual
+Simultaneous Localization and Mapping (vSLAM); and (4) vSLAM with IMUs. In all
+four settings, the proposed method performs significantly better than the
+compared baselines and the state-of-the-art methods. Additionally, using the
+assumption of continuous motion, changes in pose may actually live in a
+manifold that has lower than 6 degrees of freedom (DOF) is also realized. We
+call this low DOF motion representation as the \emph{intrinsic motion} and use
+the approach in vSLAM settings, showing impressive camera tracking performance.",cs.CV,['cs.CV']
+Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning,Christopher Liao · Theodoros Tsiligkaridis · Brian Kulis, ,https://arxiv.org/abs/2311.13612,,2311.13612.pdf,Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning,"Over the past year, a large body of multimodal research has emerged around
+zero-shot evaluation using GPT descriptors. These studies boost the zero-shot
+accuracy of pretrained VL models with an ensemble of label-specific text
+generated by GPT. A recent study, WaffleCLIP, demonstrated that similar
+zero-shot accuracy can be achieved with an ensemble of random descriptors.
+However, both zero-shot methods are un-trainable and consequently sub-optimal
+when some few-shot out-of-distribution (OOD) training data is available.
+Inspired by these prior works, we present two more flexible methods called
+descriptor and word soups, which do not require an LLM at test time and can
+leverage training data to increase OOD target accuracy. Descriptor soup
+greedily selects a small set of textual descriptors using generic few-shot
+training data, then calculates robust class embeddings using the selected
+descriptors. Word soup greedily assembles a chain of words in a similar manner.
+Compared to existing few-shot soft prompt tuning methods, word soup requires
+fewer parameters by construction and less GPU memory, since it does not require
+backpropagation. Both soups outperform current published few-shot methods, even
+when combined with SoTA zero-shot methods, on cross-dataset and domain
+generalization benchmarks. Compared with SoTA prompt and descriptor ensembling
+methods, such as ProDA and WaffleCLIP, word soup achieves higher OOD accuracy
+with fewer ensemble members. Please checkout our code:
+github.com/Chris210634/word_soups",cs.CV,['cs.CV']
+ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis,Muhammad Hamza Mughal · Rishabh Dabral · Ikhsanul Habibie · Lucia Donatelli · Marc Habermann · Christian Theobalt, ,https://arxiv.org/abs/2403.17936,,2403.17936.pdf,ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis,"Gestures play a key role in human communication. Recent methods for co-speech
+gesture generation, while managing to generate beat-aligned motions, struggle
+generating gestures that are semantically aligned with the utterance. Compared
+to beat gestures that align naturally to the audio signal, semantically
+coherent gestures require modeling the complex interactions between the
+language and human motion, and can be controlled by focusing on certain words.
+Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal
+gesture synthesis, which can not only generate gestures based on multi-modal
+speech inputs, but can also facilitate controllability in gesture synthesis.
+Our method proposes two guidance objectives that allow the users to modulate
+the impact of different conditioning modalities (e.g. audio vs text) as well as
+to choose certain words to be emphasized during gesturing. Our method is
+versatile in that it can be trained either for generating monologue gestures or
+even the conversational gestures. To further advance the research on
+multi-party interactive gestures, the DnD Group Gesture dataset is released,
+which contains 6 hours of gesture data showing 5 people interacting with one
+another. We compare our method with several recent works and demonstrate
+effectiveness of our method on a variety of tasks. We urge the reader to watch
+our supplementary video at our website.",cs.CV,['cs.CV']
+SEAS: ShapE-Aligned Supervision for Person Re-Identification,Haidong Zhu · Pranav Budhwant · Zhaoheng Zheng · Ram Nevatia, ,https://arxiv.org/abs/2312.05634,,2312.05634.pdf,PGDS: Pose-Guidance Deep Supervision for Mitigating Clothes-Changing in Person Re-Identification,"Person Re-Identification (Re-ID) task seeks to enhance the tracking of
+multiple individuals by surveillance cameras. It supports multimodal tasks,
+including text-based person retrieval and human matching. One of the most
+significant challenges faced in Re-ID is clothes-changing, where the same
+person may appear in different outfits. While previous methods have made
+notable progress in maintaining clothing data consistency and handling clothing
+change data, they still rely excessively on clothing information, which can
+limit performance due to the dynamic nature of human appearances. To mitigate
+this challenge, we propose the Pose-Guidance Deep Supervision (PGDS), an
+effective framework for learning pose guidance within the Re-ID task. It
+consists of three modules: a human encoder, a pose encoder, and a Pose-to-Human
+Projection module (PHP). Our framework guides the human encoder, i.e., the main
+re-identification model, with pose information from the pose encoder through
+multiple layers via the knowledge transfer mechanism from the PHP module,
+helping the human encoder learn body parts information without increasing
+computation resources in the inference stage. Through extensive experiments,
+our method surpasses the performance of current state-of-the-art methods,
+demonstrating its robustness and effectiveness for real-world applications. Our
+code is available at https://github.com/huyquoctrinh/PGDS.",cs.CV,['cs.CV']
+Normalizing Flows on the Product Space of SO(3) Manifolds for Probabilistic Human Pose Modeling,Olaf Dünkel · Tim Salzmann �� Florian Pfaff, ,https://arxiv.org/abs/2404.05675,,2404.05675.pdf,Normalizing Flows on the Product Space of SO(3) Manifolds for Probabilistic Human Pose Modeling,"Normalizing flows have proven their efficacy for density estimation in
+Euclidean space, but their application to rotational representations, crucial
+in various domains such as robotics or human pose modeling, remains
+underexplored. Probabilistic models of the human pose can benefit from
+approaches that rigorously consider the rotational nature of human joints. For
+this purpose, we introduce HuProSO3, a normalizing flow model that operates on
+a high-dimensional product space of SO(3) manifolds, modeling the joint
+distribution for human joints with three degrees of freedom. HuProSO3's
+advantage over state-of-the-art approaches is demonstrated through its superior
+modeling accuracy in three different applications and its capability to
+evaluate the exact likelihood. This work not only addresses the technical
+challenge of learning densities on SO(3) manifolds, but it also has broader
+implications for domains where the probabilistic regression of correlated 3D
+rotations is of importance.",cs.CV,['cs.CV']
+ES$^3$: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations,Yuanhang Zhang · Shuang Yang · Shiguang Shan · Xilin Chen, ,https://arxiv.org/abs/2312.10305,,2312.10305.pdf,Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction,"Speech signals are inherently complex as they encompass both global acoustic
+characteristics and local semantic information. However, in the task of target
+speech extraction, certain elements of global and local semantic information in
+the reference speech, which are irrelevant to speaker identity, can lead to
+speaker confusion within the speech extraction network. To overcome this
+challenge, we propose a self-supervised disentangled representation learning
+method. Our approach tackles this issue through a two-phase process, utilizing
+a reference speech encoding network and a global information disentanglement
+network to gradually disentangle the speaker identity information from other
+irrelevant factors. We exclusively employ the disentangled speaker identity
+information to guide the speech extraction network. Moreover, we introduce the
+adaptive modulation Transformer to ensure that the acoustic representation of
+the mixed signal remains undisturbed by the speaker embeddings. This component
+incorporates speaker embeddings as conditional information, facilitating
+natural and efficient guidance for the speech extraction network. Experimental
+results substantiate the effectiveness of our meticulously crafted approach,
+showcasing a substantial reduction in the likelihood of speaker confusion.",cs.SD,"['cs.SD', 'cs.AI', 'cs.LG', 'eess.AS']"
+Video Interpolation with Diffusion Models,Siddhant Jain · Daniel Watson · Aleksander Holynski · Eric Tabellion · Ben Poole · Janne Kontkanen,https://vidim-interpolation.github.io/,https://arxiv.org/abs/2404.01203,,2404.01203.pdf,Video Interpolation with Diffusion Models,"We present VIDIM, a generative model for video interpolation, which creates
+short videos given a start and end frame. In order to achieve high fidelity and
+generate motions unseen in the input data, VIDIM uses cascaded diffusion models
+to first generate the target video at low resolution, and then generate the
+high-resolution video conditioned on the low-resolution generated video. We
+compare VIDIM to previous state-of-the-art methods on video interpolation, and
+demonstrate how such works fail in most settings where the underlying motion is
+complex, nonlinear, or ambiguous while VIDIM can easily handle such cases. We
+additionally demonstrate how classifier-free guidance on the start and end
+frame and conditioning the super-resolution model on the original
+high-resolution frames without additional parameters unlocks high-fidelity
+results. VIDIM is fast to sample from as it jointly denoises all the frames to
+be generated, requires less than a billion parameters per diffusion model to
+produce compelling results, and still enjoys scalability and improved quality
+at larger parameter counts.",cs.CV,['cs.CV']
+Infer from What You Have Seen Before: Temporally-dependent Classifier for Semi-supervised Video Semantic Segmentation,Jiafan Zhuang · Zilei Wang · Yixin Zhang · Zhun Fan, ,,https://www.youtube.com/watch?v=k50sUgxC09o,,,,,nan
+IReNe: Instant Recoloring of Neural Radiance Fields,Alessio Mazzucchelli · Adrian Garcia-Garcia · Elena Garces · Fernando Rivas-Manzaneque · Francesc Moreno-Noguer · Adrian Penate-Sanchez,https://iviazz97.github.io/irene/,https://arxiv.org/abs/2405.19876,,2405.19876.pdf,IReNe: Instant Recoloring in Neural Radiance Fields,"Advances in NERFs have allowed for 3D scene reconstructions and novel view
+synthesis. Yet, efficiently editing these representations while retaining
+photorealism is an emerging challenge. Recent methods face three primary
+limitations: they're slow for interactive use, lack precision at object
+boundaries, and struggle to ensure multi-view consistency. We introduce IReNe
+to address these limitations, enabling swift, near real-time color editing in
+NeRF. Leveraging a pre-trained NeRF model and a single training image with
+user-applied color edits, IReNe swiftly adjusts network parameters in seconds.
+This adjustment allows the model to generate new scene views, accurately
+representing the color changes from the training image while also controlling
+object boundaries and view-specific effects. Object boundary control is
+achieved by integrating a trainable segmentation module into the model. The
+process gains efficiency by retraining only the weights of the last network
+layer. We observed that neurons in this layer can be classified into those
+responsible for view-dependent appearance and those contributing to diffuse
+appearance. We introduce an automated classification approach to identify these
+neuron types and exclusively fine-tune the weights of the diffuse neurons. This
+further accelerates training and ensures consistent color edits across
+different views. A thorough validation on a new dataset, with edited object
+colors, shows significant quantitative and qualitative advancements over
+competitors, accelerating speeds by 5x to 500x.",cs.CV,['cs.CV']
+FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization,Shuai Tan · Bin Ji · Ye Pan, ,https://arxiv.org/abs/2403.06375,,2403.06375.pdf,FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization,"Generating emotional talking faces is a practical yet challenging endeavor.
+To create a lifelike avatar, we draw upon two critical insights from a human
+perspective: 1) The connection between audio and the non-deterministic facial
+dynamics, encompassing expressions, blinks, poses, should exhibit synchronous
+and one-to-many mapping. 2) Vibrant expressions are often accompanied by
+emotion-aware high-definition (HD) textures and finely detailed teeth. However,
+both aspects are frequently overlooked by existing methods. To this end, this
+paper proposes using normalizing Flow and Vector-Quantization modeling to
+produce emotional talking faces that satisfy both insights concurrently
+(FlowVQTalker). Specifically, we develop a flow-based coefficient generator
+that encodes the dynamics of facial emotion into a multi-emotion-class latent
+space represented as a mixture distribution. The generation process commences
+with random sampling from the modeled distribution, guided by the accompanying
+audio, enabling both lip-synchronization and the uncertain nonverbal facial
+cues generation. Furthermore, our designed vector-quantization image generator
+treats the creation of expressive facial images as a code query task, utilizing
+a learned codebook to provide rich, high-quality textures that enhance the
+emotional perception of the results. Extensive experiments are conducted to
+showcase the effectiveness of our approach.",cs.CV,['cs.CV']
+Self-Supervised Class-Agnostic Motion Prediction with Spatial and Temporal Consistency Regularizations,Kewei Wang · Yizheng Wu · Jun Cen · Zhiyu Pan · Xingyi Li · Zhe Wang · Zhiguo Cao · Guosheng Lin, ,https://arxiv.org/abs/2403.13261,,2403.13261.pdf,Self-Supervised Class-Agnostic Motion Prediction with Spatial and Temporal Consistency Regularizations,"The perception of motion behavior in a dynamic environment holds significant
+importance for autonomous driving systems, wherein class-agnostic motion
+prediction methods directly predict the motion of the entire point cloud. While
+most existing methods rely on fully-supervised learning, the manual labeling of
+point cloud data is laborious and time-consuming. Therefore, several
+annotation-efficient methods have been proposed to address this challenge.
+Although effective, these methods rely on weak annotations or additional
+multi-modal data like images, and the potential benefits inherent in the point
+cloud sequence are still underexplored. To this end, we explore the feasibility
+of self-supervised motion prediction with only unlabeled LiDAR point clouds.
+Initially, we employ an optimal transport solver to establish coarse
+correspondences between current and future point clouds as the coarse pseudo
+motion labels. Training models directly using such coarse labels leads to
+noticeable spatial and temporal prediction inconsistencies. To mitigate these
+issues, we introduce three simple spatial and temporal regularization losses,
+which facilitate the self-supervised training process effectively. Experimental
+results demonstrate the significant superiority of our approach over the
+state-of-the-art self-supervised methods.",cs.CV,['cs.CV']
+Latency Correction for Event-guided Deblurring and Frame Interpolation,Yixin Yang · Jinxiu Liang · Bohan Yu · Yan Chen · Jimmy S. Ren · Boxin Shi, ,https://arxiv.org/abs/2306.15507,,2306.15507.pdf,Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames,"This paper makes the first attempt to tackle the challenging task of
+recovering arbitrary frame rate latent global shutter (GS) frames from two
+consecutive rolling shutter (RS) frames, guided by the novel event camera data.
+Although events possess high temporal resolution, beneficial for video frame
+interpolation (VFI), a hurdle in tackling this task is the lack of paired GS
+frames. Another challenge is that RS frames are susceptible to distortion when
+capturing moving objects. To this end, we propose a novel self-supervised
+framework that leverages events to guide RS frame correction and VFI in a
+unified framework. Our key idea is to estimate the displacement field (DF)
+non-linear dense 3D spatiotemporal information of all pixels during the
+exposure time, allowing for the reciprocal reconstruction between RS and GS
+frames as well as arbitrary frame rate VFI. Specifically, the displacement
+field estimation (DFE) module is proposed to estimate the spatiotemporal motion
+from events to correct the RS distortion and interpolate the GS frames in one
+step. We then combine the input RS frames and DF to learn a mapping for
+RS-to-GS frame interpolation. However, as the mapping is highly
+under-constrained, we couple it with an inverse mapping (i.e., GS-to-RS) and RS
+frame warping (i.e., RS-to-RS) for self-supervision. As there is a lack of
+labeled datasets for evaluation, we generate two synthetic datasets and collect
+a real-world dataset to train and test our method. Experimental results show
+that our method yields comparable or better performance with prior supervised
+methods.",cs.CV,"['cs.CV', 'cs.RO']"
+Single Domain Generalization for Crowd Counting,Zhuoxuan Peng · S.-H. Gary Chan,https://github.com/Shimmer93/MPCount,https://arxiv.org/abs/2403.09124,,2403.09124.pdf,Single Domain Generalization for Crowd Counting,"Due to its promising results, density map regression has been widely employed
+for image-based crowd counting. The approach, however, often suffers from
+severe performance degradation when tested on data from unseen scenarios, the
+so-called ""domain shift"" problem. To address the problem, we investigate in
+this work single domain generalization (SDG) for crowd counting. The existing
+SDG approaches are mainly for image classification and segmentation, and can
+hardly be extended to our case due to its regression nature and label ambiguity
+(i.e., ambiguous pixel-level ground truths). We propose MPCount, a novel
+effective SDG approach even for narrow source distribution. MPCount stores
+diverse density values for density map regression and reconstructs
+domain-invariant features by means of only one memory bank, a content error
+mask and attention consistency loss. By partitioning the image into grids, it
+employs patch-wise classification as an auxiliary task to mitigate label
+ambiguity. Through extensive experiments on different datasets, MPCount is
+shown to significantly improve counting accuracy compared to the state of the
+art under diverse scenarios unobserved in the training data characterized by
+narrow source distribution. Code is available at
+https://github.com/Shimmer93/MPCount.",cs.CV,['cs.CV']
+Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild,Fanghua Yu · Jinjin Gu · Zheyuan Li · Jinfan Hu · Xiangtao Kong · Xintao Wang · Jingwen He · Yu Qiao · Chao Dong, ,https://arxiv.org/abs/2401.13627,,2401.13627.pdf,Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild,"We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image
+restoration method that harnesses generative prior and the power of model
+scaling up. Leveraging multi-modal techniques and advanced generative prior,
+SUPIR marks a significant advance in intelligent and realistic image
+restoration. As a pivotal catalyst within SUPIR, model scaling dramatically
+enhances its capabilities and demonstrates new potential for image restoration.
+We collect a dataset comprising 20 million high-resolution, high-quality images
+for model training, each enriched with descriptive text annotations. SUPIR
+provides the capability to restore images guided by textual prompts, broadening
+its application scope and potential. Moreover, we introduce negative-quality
+prompts to further improve perceptual quality. We also develop a
+restoration-guided sampling method to suppress the fidelity issue encountered
+in generative-based restoration. Experiments demonstrate SUPIR's exceptional
+restoration effects and its novel capacity to manipulate restoration through
+textual prompts.",cs.CV,['cs.CV']
+ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles,Jiawei Zhang · Chejian Xu · Bo Li, ,https://arxiv.org/abs/2405.14062,,2405.14062.pdf,ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles,"We present ChatScene, a Large Language Model (LLM)-based agent that leverages
+the capabilities of LLMs to generate safety-critical scenarios for autonomous
+vehicles. Given unstructured language instructions, the agent first generates
+textually described traffic scenarios using LLMs. These scenario descriptions
+are subsequently broken down into several sub-descriptions for specified
+details such as behaviors and locations of vehicles. The agent then
+distinctively transforms the textually described sub-scenarios into
+domain-specific languages, which then generate actual code for prediction and
+control in simulators, facilitating the creation of diverse and complex
+scenarios within the CARLA simulation environment. A key part of our agent is a
+comprehensive knowledge retrieval component, which efficiently translates
+specific textual descriptions into corresponding domain-specific code snippets
+by training a knowledge database containing the scenario description and code
+pairs. Extensive experimental results underscore the efficacy of ChatScene in
+improving the safety of autonomous vehicles. For instance, the scenarios
+generated by ChatScene show a 15% increase in collision rates compared to
+state-of-the-art baselines when tested against different reinforcement
+learning-based ego vehicles. Furthermore, we show that by using our generated
+safety-critical scenarios to fine-tune different RL-based autonomous driving
+models, they can achieve a 9% reduction in collision rates, surpassing current
+SOTA methods. ChatScene effectively bridges the gap between textual
+descriptions of traffic scenarios and practical CARLA simulations, providing a
+unified way to conveniently generate safety-critical scenarios for safety
+testing and improvement for AVs.",cs.AI,"['cs.AI', 'cs.LG']"
+KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimation,Jihua Peng · Yanghong Zhou · Tracy P Y Mok, ,https://arxiv.org/abs/2404.00658,,2404.00658.pdf,KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimation,"This paper presents a novel Kinematics and Trajectory Prior
+Knowledge-Enhanced Transformer (KTPFormer), which overcomes the weakness in
+existing transformer-based methods for 3D human pose estimation that the
+derivation of Q, K, V vectors in their self-attention mechanisms are all based
+on simple linear mapping. We propose two prior attention modules, namely
+Kinematics Prior Attention (KPA) and Trajectory Prior Attention (TPA) to take
+advantage of the known anatomical structure of the human body and motion
+trajectory information, to facilitate effective learning of global dependencies
+and features in the multi-head self-attention. KPA models kinematic
+relationships in the human body by constructing a topology of kinematics, while
+TPA builds a trajectory topology to learn the information of joint motion
+trajectory across frames. Yielding Q, K, V vectors with prior knowledge, the
+two modules enable KTPFormer to model both spatial and temporal correlations
+simultaneously. Extensive experiments on three benchmarks (Human3.6M,
+MPI-INF-3DHP and HumanEva) show that KTPFormer achieves superior performance in
+comparison to state-of-the-art methods. More importantly, our KPA and TPA
+modules have lightweight plug-and-play designs and can be integrated into
+various transformer-based networks (i.e., diffusion-based) to improve the
+performance with only a very small increase in the computational overhead. The
+code is available at: https://github.com/JihuaPeng/KTPFormer.",cs.CV,['cs.CV']
+Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification,Pingping Zhang · Yuhao Wang · Yang Liu · Zhengzheng Tu · Huchuan Lu,https://github.com/924973292/EDITOR,https://arxiv.org/abs/2403.10254,,2403.10254.pdf,Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification,"Single-modal object re-identification (ReID) faces great challenges in
+maintaining robustness within complex visual scenarios. In contrast,
+multi-modal object ReID utilizes complementary information from diverse
+modalities, showing great potentials for practical applications. However,
+previous methods may be easily affected by irrelevant backgrounds and usually
+ignore the modality gaps. To address above issues, we propose a novel learning
+framework named \textbf{EDITOR} to select diverse tokens from vision
+Transformers for multi-modal object ReID. We begin with a shared vision
+Transformer to extract tokenized features from different input modalities.
+Then, we introduce a Spatial-Frequency Token Selection (SFTS) module to
+adaptively select object-centric tokens with both spatial and frequency
+information. Afterwards, we employ a Hierarchical Masked Aggregation (HMA)
+module to facilitate feature interactions within and across modalities.
+Finally, to further reduce the effect of backgrounds, we propose a Background
+Consistency Constraint (BCC) and an Object-Centric Feature Refinement (OCFR).
+They are formulated as two new loss functions, which improve the feature
+discrimination with background suppression. As a result, our framework can
+generate more discriminative features for multi-modal object ReID. Extensive
+experiments on three multi-modal ReID benchmarks verify the effectiveness of
+our methods. The code is available at https://github.com/924973292/EDITOR.",cs.CV,"['cs.CV', 'cs.IR', 'cs.MM']"
+ShapeWalk: Compositional Shape Editing through Language-Guided Chains,Habib Slim · Mohamed Elhoseiny,https://shapewalk.github.io/,https://arxiv.org/html/2405.20319v1,,2405.20319v1.pdf,ParSEL: Parameterized Shape Editing with Language,"The ability to edit 3D assets from natural language presents a compelling
+paradigm to aid in the democratization of 3D content creation. However, while
+natural language is often effective at communicating general intent, it is
+poorly suited for specifying precise manipulation. To address this gap, we
+introduce ParSEL, a system that enables controllable editing of high-quality 3D
+assets from natural language. Given a segmented 3D mesh and an editing request,
+ParSEL produces a parameterized editing program. Adjusting the program
+parameters allows users to explore shape variations with a precise control over
+the magnitudes of edits. To infer editing programs which align with an input
+edit request, we leverage the abilities of large-language models (LLMs).
+However, while we find that LLMs excel at identifying initial edit operations,
+they often fail to infer complete editing programs, and produce outputs that
+violate shape semantics. To overcome this issue, we introduce Analytical Edit
+Propagation (AEP), an algorithm which extends a seed edit with additional
+operations until a complete editing program has been formed. Unlike prior
+methods, AEP searches for analytical editing operations compatible with a range
+of possible user edits through the integration of computer algebra systems for
+geometric analysis. Experimentally we demonstrate ParSEL's effectiveness in
+enabling controllable editing of 3D objects through natural language requests
+over alternative system designs.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.HC', 'cs.SC']"
+Video-P2P: Video Editing with Cross-attention Control,Shaoteng Liu · Yuechen Zhang · Wenbo Li · Zhe Lin · Jiaya Jia, ,,https://www.researchgate.net/publication/380733385_Video-P2P_Video_Editing_with_Cross-attention_Control,,,,,nan
+R-Cyclic Diffuser: Reductive and Cyclic Latent Diffusion for 3D Clothed Human Digitalization,Kennard Chan · Fayao Liu · Guosheng Lin · Chuan-Sheng Foo · Weisi Lin, ,https://arxiv.org/html/2401.12175v2,,2401.12175v2.pdf,Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM,"Reconstructing 3D humans from a single image has been extensively
+investigated. However, existing approaches often fall short on capturing fine
+geometry and appearance details, hallucinating occluded parts with plausible
+details, and achieving generalization across unseen and in-the-wild datasets.
+We present Human-LRM, a diffusion-guided feed-forward model that predicts the
+implicit field of a human from a single image. Leveraging the power of the
+state-of-the-art reconstruction model (i.e., LRM) and generative model (i.e
+Stable Diffusion), our method is able to capture human without any template
+prior, e.g., SMPL, and effectively enhance occluded parts with rich and
+realistic details. Our approach first uses a single-view LRM model with an
+enhanced geometry decoder to get the triplane NeRF representation. The novel
+view renderings from the triplane NeRF provide strong geometry and color prior,
+from which we generate photo-realistic details for the occluded parts using a
+diffusion model. The generated multiple views then enable reconstruction with
+high-quality geometry and appearance, leading to superior overall performance
+comparing to all existing human reconstruction methods.",cs.CV,['cs.CV']
+Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model,Wenfeng Song · Xingliang Jin · Shuai Li · Chenglizhao Chen · Aimin Hao · Xia HOU · Ning Li · Hong Qin,https://xingliangjin.github.io/MCM-LDM-Web/,https://arxiv.org/abs/2306.09330,,2306.09330.pdf,ArtFusion: Controllable Arbitrary Style Transfer using Dual Conditional Latent Diffusion Models,"Arbitrary Style Transfer (AST) aims to transform images by adopting the style
+from any selected artwork. Nonetheless, the need to accommodate diverse and
+subjective user preferences poses a significant challenge. While some users
+wish to preserve distinct content structures, others might favor a more
+pronounced stylization. Despite advances in feed-forward AST methods, their
+limited customizability hinders their practical application. We propose a new
+approach, ArtFusion, which provides a flexible balance between content and
+style. In contrast to traditional methods reliant on biased similarity losses,
+ArtFusion utilizes our innovative Dual Conditional Latent Diffusion
+Probabilistic Models (Dual-cLDM). This approach mitigates repetitive patterns
+and enhances subtle artistic aspects like brush strokes and genre-specific
+features. Despite the promising results of conditional diffusion probabilistic
+models (cDM) in various generative tasks, their introduction to style transfer
+is challenging due to the requirement for paired training data. ArtFusion
+successfully navigates this issue, offering more practical and controllable
+stylization. A key element of our approach involves using a single image for
+both content and style during model training, all the while maintaining
+effective stylization during inference. ArtFusion outperforms existing
+approaches on outstanding controllability and faithful presentation of artistic
+details, providing evidence of its superior style transfer capabilities.
+Furthermore, the Dual-cLDM utilized in ArtFusion carries the potential for a
+variety of complex multi-condition generative tasks, thus greatly broadening
+the impact of our research.",cs.CV,['cs.CV']
+HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting,Hongyu Zhou · Jiahao Shao · Lu Xu · Dongfeng Bai · Weichao Qiu · Bingbing Liu · Yue Wang · Andreas Geiger · Yiyi Liao, ,https://arxiv.org/abs/2403.12722,,2403.12722.pdf,HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting,"Holistic understanding of urban scenes based on RGB images is a challenging
+yet important problem. It encompasses understanding both the geometry and
+appearance to enable novel view synthesis, parsing semantic labels, and
+tracking moving objects. Despite considerable progress, existing approaches
+often focus on specific aspects of this task and require additional inputs such
+as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we
+introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic
+urban scene understanding. Our main idea involves the joint optimization of
+geometry, appearance, semantics, and motion using a combination of static and
+dynamic 3D Gaussians, where moving object poses are regularized via physical
+constraints. Our approach offers the ability to render new viewpoints in
+real-time, yielding 2D and 3D semantic information with high accuracy, and
+reconstruct dynamic scenes, even in scenarios where 3D bounding box detection
+are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2
+demonstrate the effectiveness of our approach.",cs.CV,['cs.CV']
+HumMUSS: Human Motion Understanding using State Space Models,Arnab Mondal · Stefano Alletto · Denis Tome, ,https://arxiv.org/abs/2404.10880,,2404.10880.pdf,HumMUSS: Human Motion Understanding using State Space Models,"Understanding human motion from video is essential for a range of
+applications, including pose estimation, mesh recovery and action recognition.
+While state-of-the-art methods predominantly rely on transformer-based
+architectures, these approaches have limitations in practical scenarios.
+Transformers are slower when sequentially predicting on a continuous stream of
+frames in real-time, and do not generalize to new frame rates. In light of
+these constraints, we propose a novel attention-free spatiotemporal model for
+human motion understanding building upon recent advancements in state space
+models. Our model not only matches the performance of transformer-based models
+in various motion understanding tasks but also brings added benefits like
+adaptability to different video frame rates and enhanced training speed when
+working with longer sequence of keypoints. Moreover, the proposed model
+supports both offline and real-time applications. For real-time sequential
+prediction, our model is both memory efficient and several times faster than
+transformer-based approaches while maintaining their high accuracy.",cs.CV,"['cs.CV', 'cs.AI']"
+SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control,Jaskirat Singh · Jianming Zhang · Qing Liu · Cameron Smith · Zhe Lin · Liang Zheng, ,https://arxiv.org/abs/2312.05039,,,SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control,"The field of generative image inpainting and object insertion has made
+significant progress with the recent advent of latent diffusion models.
+Utilizing a precise object mask can greatly enhance these applications.
+However, due to the challenges users encounter in creating high-fidelity masks,
+there is a tendency for these methods to rely on more coarse masks (e.g.,
+bounding box) for these applications. This results in limited control and
+compromised background content preservation. To overcome these limitations, we
+introduce SmartMask, which allows any novice user to create detailed masks for
+precise object insertion. Combined with a ControlNet-Inpaint model, our
+experiments demonstrate that SmartMask achieves superior object insertion
+quality, preserving the background content more effectively than previous
+methods. Notably, unlike prior works the proposed approach can also be used
+even without user-mask guidance, which allows it to perform mask-free object
+insertion at diverse positions and scales. Furthermore, we find that when used
+iteratively with a novel instruction-tuning based planning model, SmartMask can
+be used to design detailed layouts from scratch. As compared with user-scribble
+based layout design, we observe that SmartMask allows for better quality
+outputs with layout-to-image generation methods. Project page is available at
+https://smartmask-gen.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.HC', 'cs.LG', 'cs.MM']"
+Towards Progressive Multi-Frequency Representation for Image Warping,Jun Xiao · Zihang Lyu · Cong Zhang · Yakun Ju · Changjian Shui · Kin-man Lam, ,https://arxiv.org/abs/2404.10716,,2404.10716.pdf,MOWA: Multiple-in-One Image Warping Model,"While recent image warping approaches achieved remarkable success on existing
+benchmarks, they still require training separate models for each specific task
+and cannot generalize well to different camera models or customized
+manipulations. To address diverse types of warping in practice, we propose a
+Multiple-in-One image WArping model (named MOWA) in this work. Specifically, we
+mitigate the difficulty of multi-task learning by disentangling the motion
+estimation at both the region level and pixel level. To further enable dynamic
+task-aware image warping, we introduce a lightweight point-based classifier
+that predicts the task type, serving as prompts to modulate the feature maps
+for better estimation. To our knowledge, this is the first work that solves
+multiple practical warping tasks in one single model. Extensive experiments
+demonstrate that our MOWA, which is trained on six tasks for multiple-in-one
+image warping, outperforms state-of-the-art task-specific models across most
+tasks. Moreover, MOWA also exhibits promising potential to generalize into
+unseen scenes, as evidenced by cross-domain and zero-shot evaluations. The code
+will be made publicly available.",cs.CV,['cs.CV']
+V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs,Penghao Wu · Saining Xie,https://vstar-seal.github.io/,https://arxiv.org/abs/2312.14135,,2312.14135.pdf,V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs,"When we look around and perform complex tasks, how we see and selectively
+process what we see is crucial. However, the lack of this visual search
+mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on
+important visual details, especially when handling high-resolution and visually
+crowded images. To address this, we introduce V*, an LLM-guided visual search
+mechanism that employs the world knowledge in LLMs for efficient visual
+querying. When combined with an MLLM, this mechanism enhances collaborative
+reasoning, contextual understanding, and precise targeting of specific visual
+elements. This integration results in a new MLLM meta-architecture, named Show,
+sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically
+designed to evaluate MLLMs in their ability to process high-resolution images
+and focus on visual details. Our study highlights the necessity of
+incorporating visual search capabilities into multimodal systems. The code is
+available https://github.com/penghao-wu/vstar.",cs.CV,['cs.CV']
+Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models,Xinpeng Ding · Jianhua Han · Hang Xu · Xiaodan Liang · Wei Zhang · Xiaomeng Li, ,https://arxiv.org/abs/2401.00988v1,,2401.00988v1.pdf,Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models,"The rise of multimodal large language models (MLLMs) has spurred interest in
+language-based driving tasks. However, existing research typically focuses on
+limited tasks and often omits key multi-view and temporal information which is
+crucial for robust autonomous driving. To bridge these gaps, we introduce
+NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17
+subtasks, where each task demands holistic information (e.g., temporal,
+multi-view, and spatial), significantly elevating the challenge level. To
+obtain NuInstruct, we propose a novel SQL-based method to generate
+instruction-response pairs automatically, which is inspired by the driving
+logical progression of humans. We further present BEV-InMLLM, an end-to-end
+method for efficiently deriving instruction-aware Bird's-Eye-View (BEV)
+features, language-aligned for large language models. BEV-InMLLM integrates
+multi-view, spatial awareness, and temporal semantics to enhance MLLMs'
+capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module
+is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct
+demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g.
+around 9% improvement on various tasks. We plan to release our NuInstruct for
+future research development.",cs.CV,['cs.CV']
+Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation,Xinyao Li · Yuke Li · Zhekai Du · Fengling Li · Ke Lu · Jingjing Li,https://github.com/TL-UESTC/UniMoS,https://arxiv.org/abs/2403.06946,,2403.06946.pdf,Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation,"Large vision-language models (VLMs) like CLIP have demonstrated good
+zero-shot learning performance in the unsupervised domain adaptation task. Yet,
+most transfer approaches for VLMs focus on either the language or visual
+branches, overlooking the nuanced interplay between both modalities. In this
+work, we introduce a Unified Modality Separation (UniMoS) framework for
+unsupervised domain adaptation. Leveraging insights from modality gap studies,
+we craft a nimble modality separation network that distinctly disentangles
+CLIP's features into language-associated and vision-associated components. Our
+proposed Modality-Ensemble Training (MET) method fosters the exchange of
+modality-agnostic information while maintaining modality-specific nuances. We
+align features across domains using a modality discriminator. Comprehensive
+evaluations on three benchmarks reveal our approach sets a new state-of-the-art
+with minimal computational costs. Code: https://github.com/TL-UESTC/UniMoS",cs.CV,['cs.CV']
+Relation Rectification in Diffusion Model,Yinwei Wu · Xingyi Yang · Xinchao Wang,https://wuyinwei-hah.github.io/rrnet.github.io/,https://arxiv.org/abs/2403.20249,,2403.20249.pdf,Relation Rectification in Diffusion Model,"Despite their exceptional generative abilities, large text-to-image diffusion
+models, much like skilled but careless artists, often struggle with accurately
+depicting visual relationships between objects. This issue, as we uncover
+through careful analysis, arises from a misaligned text encoder that struggles
+to interpret specific relationships and differentiate the logical order of
+associated objects. To resolve this, we introduce a novel task termed Relation
+Rectification, aiming to refine the model to accurately represent a given
+relationship it initially fails to generate. To address this, we propose an
+innovative solution utilizing a Heterogeneous Graph Convolutional Network
+(HGCN). It models the directional relationships between relation terms and
+corresponding objects within the input prompts. Specifically, we optimize the
+HGCN on a pair of prompts with identical relational words but reversed object
+orders, supplemented by a few reference images. The lightweight HGCN adjusts
+the text embeddings generated by the text encoder, ensuring the accurate
+reflection of the textual relation in the embedding space. Crucially, our
+method retains the parameters of the text encoder and diffusion model,
+preserving the model's robust performance on unrelated descriptions. We
+validated our approach on a newly curated dataset of diverse relational data,
+demonstrating both quantitative and qualitative enhancements in generating
+images with precise visual relations. Project page:
+https://wuyinwei-hah.github.io/rrnet.github.io/.",cs.CV,['cs.CV']
+CoralSCOP: Segment any COral Image on this Planet,"Zheng Ziqiang · Liang Haixin · Binh-Son Hua · Tim, Yue Him Wong · Put ANG · Apple CHUI · Sai-Kit Yeung", ,,https://ais.hkust.edu.hk/whats-happening/news/isd-research-team-produces-first-model-segment-and-generalize-coral-reef-image,,,,,nan
+Category-Level Multi-Part Multi-Joint 3D Shape Assembly,Yichen Li · Kaichun Mo · Yueqi Duan · He Wang · Jiequan Zhang · Lin Shao · Wojciech Matusik · Leonidas Guibas, ,,,,,,,nan
+AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings,Jamie Watson · Filippo Aleotti · Mohamed Sayed · Zawar Qureshi · Oisin Mac Aodha · Gabriel J. Brostow · Michael Firman · Sara Vicente,https://nianticlabs.github.io/airplanes/,,https://link.springer.com/article/10.1007/s00371-023-03110-7,,,,,nan
+Fun with Flags: Robust Principal Directions via Flag Manifolds,Tolga Birdal · Nathan Mankovich, ,https://arxiv.org/abs/2401.04071v1,,2401.04071v1.pdf,Fun with Flags: Robust Principal Directions via Flag Manifolds,"Principal component analysis (PCA), along with its extensions to manifolds
+and outlier contaminated data, have been indispensable in computer vision and
+machine learning. In this work, we present a unifying formalism for PCA and its
+variants, and introduce a framework based on the flags of linear subspaces, \ie
+a hierarchy of nested linear subspaces of increasing dimension, which not only
+allows for a common implementation but also yields novel variants, not explored
+previously. We begin by generalizing traditional PCA methods that either
+maximize variance or minimize reconstruction error. We expand these
+interpretations to develop a wide array of new dimensionality reduction
+algorithms by accounting for outliers and the data manifold. To devise a common
+computational approach, we recast robust and dual forms of PCA as optimization
+problems on flag manifolds. We then integrate tangent space approximations of
+principal geodesic analysis (tangent-PCA) into this flag-based framework,
+creating novel robust and dual geodesic PCA variations. The remarkable
+flexibility offered by the 'flagification' introduced here enables even more
+algorithmic variants identified by specific flag types. Last but not least, we
+propose an effective convergent solver for these flag-formulations employing
+the Stiefel manifold. Our empirical results on both real-world and synthetic
+scenarios, demonstrate the superiority of our novel algorithms, especially in
+terms of robustness to outliers on manifolds.",cs.CV,"['cs.CV', 'cs.LG', 'math.DG', 'math.OC', 'stat.ML']"
+PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors,Haley So · Laurie Bose · Piotr Dudek · Gordon Wetzstein, ,,,,,,,nan
+Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations,Chenyu You · Yifei Min · Weicheng Dai · Jasjeet Sekhon · Lawrence Staib · James Duncan, ,https://arxiv.org/abs/2403.07241,,2403.07241.pdf,Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations,"Fine-tuning pre-trained vision-language models, like CLIP, has yielded
+success on diverse downstream tasks. However, several pain points persist for
+this paradigm: (i) directly tuning entire pre-trained models becomes both
+time-intensive and computationally costly. Additionally, these tuned models
+tend to become highly specialized, limiting their practicality for real-world
+deployment; (ii) recent studies indicate that pre-trained vision-language
+classifiers may overly depend on spurious features -- patterns that correlate
+with the target in training data, but are not related to the true labeling
+function; and (iii) existing studies on mitigating the reliance on spurious
+features, largely based on the assumption that we can identify such features,
+does not provide definitive assurance for real-world applications. As a
+piloting study, this work focuses on exploring mitigating the reliance on
+spurious features for CLIP without using any group annotation. To this end, we
+systematically study the existence of spurious correlation on CLIP and
+CILP+ERM. We first, following recent work on Deep Feature Reweighting (DFR),
+verify that last-layer retraining can greatly improve group robustness on
+pretrained CLIP. In view of them, we advocate a lightweight representation
+calibration method for fine-tuning CLIP, by first generating a calibration set
+using the pretrained CLIP, and then calibrating representations of samples
+within this set through contrastive learning, all without the need for group
+labels. Extensive experiments and in-depth visualizations on several benchmarks
+validate the effectiveness of our proposals, largely reducing reliance and
+significantly boosting the model generalization.",cs.CV,"['cs.CV', 'cs.LG']"
+Guided Slot Attention for Unsupervised Video Object Segmentation,Minhyeok Lee · Suhwan Cho · Dogyoon Lee · Chaewon Park · Jungho Lee · Sangyoun Lee, ,https://arxiv.org/abs/2309.14786,,,Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation,"Unsupervised video object segmentation (VOS) is a task that aims to detect
+the most salient object in a video without external guidance about the object.
+To leverage the property that salient objects usually have distinctive
+movements compared to the background, recent methods collaboratively use motion
+cues extracted from optical flow maps with appearance cues extracted from RGB
+images. However, as optical flow maps are usually very relevant to segmentation
+masks, the network is easy to be learned overly dependent on the motion cues
+during network training. As a result, such two-stream approaches are vulnerable
+to confusing motion cues, making their prediction unstable. To relieve this
+issue, we design a novel motion-as-option network by treating motion cues as
+optional. During network training, RGB images are randomly provided to the
+motion encoder instead of optical flow maps, to implicitly reduce motion
+dependency of the network. As the learned motion encoder can deal with both RGB
+images and optical flow maps, two different predictions can be generated
+depending on which source information is used as motion input. In order to
+fully exploit this property, we also propose an adaptive output selection
+algorithm to adopt optimal prediction result at test time. Our proposed
+approach affords state-of-the-art performance on all public benchmark datasets,
+even maintaining real-time inference speed.",cs.CV,['cs.CV']
+A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning,Siddharth Srivastava · Gaurav Sharma, ,https://arxiv.org/abs/2310.09276,,2310.09276.pdf,Transformer-based Multimodal Change Detection with Multitask Consistency Constraints,"Change detection plays a fundamental role in Earth observation for analyzing
+temporal iterations over time. However, recent studies have largely neglected
+the utilization of multimodal data that presents significant practical and
+technical advantages compared to single-modal approaches. This research focuses
+on leveraging {pre-event} digital surface model (DSM) data and {post-event}
+digital aerial images captured at different times for detecting change beyond
+2D. We observe that the current change detection methods struggle with the
+multitask conflicts between semantic and height change detection tasks. To
+address this challenge, we propose an efficient Transformer-based network that
+learns shared representation between cross-dimensional inputs through
+cross-attention. {It adopts a consistency constraint to establish the
+multimodal relationship. Initially, pseudo-changes are derived by employing
+height change thresholding. Subsequently, the $L2$ distance between semantic
+and pseudo-changes within their overlapping regions is minimized. This
+explicitly endows the height change detection (regression task) and semantic
+change detection (classification task) with representation consistency.} A
+DSM-to-image multimodal dataset encompassing three cities in the Netherlands
+was constructed. It lays a new foundation for beyond-2D change detection from
+cross-dimensional inputs. Compared to five state-of-the-art change detection
+methods, our model demonstrates consistent multitask superiority in terms of
+semantic and height change detection. Furthermore, the consistency strategy can
+be seamlessly adapted to the other methods, yielding promising improvements.",cs.CV,['cs.CV']
+Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles,Rui Song · Chenwei Liang · Hu Cao · Zhiran Yan · Walter Zimmer · Markus Gross · Andreas Festag · Alois Knoll,https://rruisong.github.io/publications/CoHFF/,https://arxiv.org/abs/2402.07635,,2402.07635.pdf,Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles,"Collaborative perception in automated vehicles leverages the exchange of
+information between agents, aiming to elevate perception results. Previous
+camera-based collaborative 3D perception methods typically employ 3D bounding
+boxes or bird's eye views as representations of the environment. However, these
+approaches fall short in offering a comprehensive 3D environmental prediction.
+To bridge this gap, we introduce the first method for collaborative 3D semantic
+occupancy prediction. Particularly, it improves local 3D semantic occupancy
+predictions by hybrid fusion of (i) semantic and occupancy task features, and
+(ii) compressed orthogonal attention features shared between vehicles.
+Additionally, due to the lack of a collaborative perception dataset designed
+for semantic occupancy prediction, we augment a current collaborative
+perception dataset to include 3D collaborative semantic occupancy labels for a
+more robust evaluation. The experimental findings highlight that: (i) our
+collaborative semantic occupancy predictions excel above the results from
+single vehicles by over 30%, and (ii) models anchored on semantic occupancy
+outpace state-of-the-art collaborative 3D detection techniques in subsequent
+perception applications, showcasing enhanced accuracy and enriched
+semantic-awareness in road environments.",cs.CV,['cs.CV']
+"EVS-assisted joint Deblurring, Rolling-Shutter Correction and Video Frame Interpolation through Sensor Inverse Modeling",Rui Jiang · Fangwen Tu · Yixuan Long · Aabhaas Vaish · Bowen Zhou · Qinyi Wang · Wei Zhang · Yuntan Fang · Luis Eduardo García Capel · Bo Mu · Tiejun Dai · Andreas Suess, ,https://arxiv.org/abs/2404.18156,,,Event-based Video Frame Interpolation with Edge Guided Motion Refinement,"Video frame interpolation, the process of synthesizing intermediate frames
+between sequential video frames, has made remarkable progress with the use of
+event cameras. These sensors, with microsecond-level temporal resolution, fill
+information gaps between frames by providing precise motion cues. However,
+contemporary Event-Based Video Frame Interpolation (E-VFI) techniques often
+neglect the fact that event data primarily supply high-confidence features at
+scene edges during multi-modal feature fusion, thereby diminishing the role of
+event signals in optical flow (OF) estimation and warping refinement. To
+address this overlooked aspect, we introduce an end-to-end E-VFI learning
+method (referred to as EGMR) to efficiently utilize edge features from event
+signals for motion flow and warping enhancement. Our method incorporates an
+Edge Guided Attentive (EGA) module, which rectifies estimated video motion
+through attentive aggregation based on the local correlation of multi-modal
+features in a coarse-to-fine strategy. Moreover, given that event data can
+provide accurate visual references at scene edges between consecutive frames,
+we introduce a learned visibility map derived from event data to adaptively
+mitigate the occlusion problem in the warping refinement process. Extensive
+experiments on both synthetic and real datasets show the effectiveness of the
+proposed approach, demonstrating its potential for higher quality video frame
+interpolation.",cs.CV,['cs.CV']
+EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion,Zehuan Huang · Hao Wen · Junting Dong · Yaohui Wang · Yangguang Li · Xinyuan Chen · Yan-Pei Cao · Ding Liang · Yu Qiao · Bo Dai · Lu Sheng,https://huanngzh.github.io/EpiDiff/,https://arxiv.org/abs/2312.06725,,2312.06725.pdf,EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion,"Generating multiview images from a single view facilitates the rapid
+generation of a 3D mesh conditioned on a single image. Recent methods that
+introduce 3D global representation into diffusion models have shown the
+potential to generate consistent multiviews, but they have reduced generation
+speed and face challenges in maintaining generalizability and quality. To
+address this issue, we propose EpiDiff, a localized interactive multiview
+diffusion model. At the core of the proposed approach is to insert a
+lightweight epipolar attention block into the frozen diffusion model,
+leveraging epipolar constraints to enable cross-view interaction among feature
+maps of neighboring views. The newly initialized 3D modeling module preserves
+the original feature distribution of the diffusion model, exhibiting
+compatibility with a variety of base diffusion models. Experiments show that
+EpiDiff generates 16 multiview images in just 12 seconds, and it surpasses
+previous methods in quality evaluation metrics, including PSNR, SSIM and LPIPS.
+Additionally, EpiDiff can generate a more diverse distribution of views,
+improving the reconstruction quality from generated multiviews. Please see our
+project page at https://huanngzh.github.io/EpiDiff/.",cs.CV,['cs.CV']
+GaussianAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh,Jing Wen · Xiaoming Zhao · Jason Ren · Alexander G. Schwing · Shenlong Wang, ,https://arxiv.org/abs/2404.07991,,2404.07991.pdf,GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh,"We introduce GoMAvatar, a novel approach for real-time, memory-efficient,
+high-quality animatable human modeling. GoMAvatar takes as input a single
+monocular video to create a digital avatar capable of re-articulation in new
+poses and real-time rendering from novel viewpoints, while seamlessly
+integrating with rasterization-based graphics pipelines. Central to our method
+is the Gaussians-on-Mesh representation, a hybrid 3D model combining rendering
+quality and speed of Gaussian splatting with geometry modeling and
+compatibility of deformable meshes. We assess GoMAvatar on ZJU-MoCap data and
+various YouTube videos. GoMAvatar matches or surpasses current monocular human
+modeling algorithms in rendering quality and significantly outperforms them in
+computational efficiency (43 FPS) while being memory-efficient (3.63 MB per
+subject).",cs.CV,['cs.CV']
+Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes,Zhiyuan Yu · Zheng Qin · lintao zheng · Kai Xu, ,https://arxiv.org/abs/2404.04557,,2404.04557.pdf,Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes,"Multi-instance point cloud registration estimates the poses of multiple
+instances of a model point cloud in a scene point cloud. Extracting accurate
+point correspondence is to the center of the problem. Existing approaches
+usually treat the scene point cloud as a whole, overlooking the separation of
+instances. Therefore, point features could be easily polluted by other points
+from the background or different instances, leading to inaccurate
+correspondences oblivious to separate instances, especially in cluttered
+scenes. In this work, we propose MIRETR, Multi-Instance REgistration
+TRansformer, a coarse-to-fine approach to the extraction of instance-aware
+correspondences. At the coarse level, it jointly learns instance-aware
+superpoint features and predicts per-instance masks. With instance masks, the
+influence from outside of the instance being concerned is minimized, such that
+highly reliable superpoint correspondences can be extracted. The superpoint
+correspondences are then extended to instance candidates at the fine level
+according to the instance masks. At last, an efficient candidate selection and
+refinement algorithm is devised to obtain the final registrations. Extensive
+experiments on three public benchmarks demonstrate the efficacy of our
+approach. In particular, MIRETR outperforms the state of the arts by 16.6
+points on F1 score on the challenging ROBI benchmark. Code and models are
+available at https://github.com/zhiyuanYU134/MIRETR.",cs.CV,['cs.CV']
+RTracker: Recoverable Tracking via PN Tree Structured Memory,Yuqing Huang · Xin Li · Zikun Zhou · Yaowei Wang · Zhenyu He · Ming-Hsuan Yang, ,https://arxiv.org/abs/2403.19242,,2403.19242.pdf,RTracker: Recoverable Tracking via PN Tree Structured Memory,"Existing tracking methods mainly focus on learning better target
+representation or developing more robust prediction models to improve tracking
+performance. While tracking performance has significantly improved, the target
+loss issue occurs frequently due to tracking failures, complete occlusion, or
+out-of-view situations. However, considerably less attention is paid to the
+self-recovery issue of tracking methods, which is crucial for practical
+applications. To this end, we propose a recoverable tracking framework,
+RTracker, that uses a tree-structured memory to dynamically associate a tracker
+and a detector to enable self-recovery ability. Specifically, we propose a
+Positive-Negative Tree-structured memory to chronologically store and maintain
+positive and negative target samples. Upon the PN tree memory, we develop
+corresponding walking rules for determining the state of the target and define
+a set of control flows to unite the tracker and the detector in different
+tracking scenarios. Our core idea is to use the support samples of positive and
+negative target categories to establish a relative distance-based criterion for
+a reliable assessment of target loss. The favorable performance in comparison
+against the state-of-the-art methods on numerous challenging benchmarks
+demonstrates the effectiveness of the proposed algorithm.",cs.CV,['cs.CV']
+Supervised Anomaly Detection for Complex Industrial Images,Aimira Baitieva · David Hurych · Victor Besnier · Olivier BERNARD, ,https://arxiv.org/abs/2405.04953,,2405.04953.pdf,Supervised Anomaly Detection for Complex Industrial Images,"Automating visual inspection in industrial production lines is essential for
+increasing product quality across various industries. Anomaly detection (AD)
+methods serve as robust tools for this purpose. However, existing public
+datasets primarily consist of images without anomalies, limiting the practical
+application of AD methods in production settings. To address this challenge, we
+present (1) the Valeo Anomaly Dataset (VAD), a novel real-world industrial
+dataset comprising 5000 images, including 2000 instances of challenging real
+defects across more than 20 subclasses. Acknowledging that traditional AD
+methods struggle with this dataset, we introduce (2) Segmentation-based Anomaly
+Detector (SegAD). First, SegAD leverages anomaly maps as well as segmentation
+maps to compute local statistics. Next, SegAD uses these statistics and an
+optional supervised classifier score as input features for a Boosted Random
+Forest (BRF) classifier, yielding the final anomaly score. Our SegAD achieves
+state-of-the-art performance on both VAD (+2.1% AUROC) and the VisA dataset
+(+0.4% AUROC). The code and the models are publicly available.",cs.CV,"['cs.CV', 'cs.LG']"
+InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization,Xiefan Guo · Jinlin Liu · Miaomiao Cui · Jiankai Li · Hongyu Yang · Di Huang, ,https://arxiv.org/abs/2404.04650,,2404.04650.pdf,InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization,"Recent strides in the development of diffusion models, exemplified by
+advancements such as Stable Diffusion, have underscored their remarkable
+prowess in generating visually compelling images. However, the imperative of
+achieving a seamless alignment between the generated image and the provided
+prompt persists as a formidable challenge. This paper traces the root of these
+difficulties to invalid initial noise, and proposes a solution in the form of
+Initial Noise Optimization (InitNO), a paradigm that refines this noise.
+Considering text prompts, not all random noises are effective in synthesizing
+semantically-faithful images. We design the cross-attention response score and
+the self-attention conflict score to evaluate the initial noise, bifurcating
+the initial latent space into valid and invalid sectors. A strategically
+crafted noise optimization pipeline is developed to guide the initial noise
+towards valid regions. Our method, validated through rigorous experimentation,
+shows a commendable proficiency in generating images in strict accordance with
+text prompts. Our code is available at https://github.com/xiefan-guo/initno.",cs.CV,['cs.CV']
+MFP: Making Full use of Probability Maps for Interactive Image Segmentation,Chaewon Lee · Seon-Ho Lee · Chang-Su Kim, ,https://arxiv.org/abs/2404.18448,,2404.18448.pdf,MFP: Making Full Use of Probability Maps for Interactive Image Segmentation,"In recent interactive segmentation algorithms, previous probability maps are
+used as network input to help predictions in the current segmentation round.
+However, despite the utilization of previous masks, useful information
+contained in the probability maps is not well propagated to the current
+predictions. In this paper, to overcome this limitation, we propose a novel and
+effective algorithm for click-based interactive image segmentation, called MFP,
+which attempts to make full use of probability maps. We first modulate previous
+probability maps to enhance their representations of user-specified objects.
+Then, we feed the modulated probability maps as additional input to the
+segmentation network. We implement the proposed MFP algorithm based on the
+ResNet-34, HRNet-18, and ViT-B backbones and assess the performance extensively
+on various datasets. It is demonstrated that MFP meaningfully outperforms the
+existing algorithms using identical backbones. The source codes are available
+at https://github.com/cwlee00/MFP.",cs.CV,['cs.CV']
+A Conditional Denoising Diffusion Probabilistic Model for Point Cloud Upsampling,Wentao Qu · Yuantian Shao · Lingwu Meng · Xiaoshui Huang · Liang Xiao, ,https://arxiv.org/abs/2312.02719,,2312.02719.pdf,A Conditional Denoising Diffusion Probabilistic Model for Point Cloud Upsampling,"Point cloud upsampling (PCU) enriches the representation of raw point clouds,
+significantly improving the performance in downstream tasks such as
+classification and reconstruction. Most of the existing point cloud upsampling
+methods focus on sparse point cloud feature extraction and upsampling module
+design. In a different way, we dive deeper into directly modelling the gradient
+of data distribution from dense point clouds. In this paper, we proposed a
+conditional denoising diffusion probability model (DDPM) for point cloud
+upsampling, called PUDM. Specifically, PUDM treats the sparse point cloud as a
+condition, and iteratively learns the transformation relationship between the
+dense point cloud and the noise. Simultaneously, PUDM aligns with a dual
+mapping paradigm to further improve the discernment of point features. In this
+context, PUDM enables learning complex geometry details in the ground truth
+through the dominant features, while avoiding an additional upsampling module
+design. Furthermore, to generate high-quality arbitrary-scale point clouds
+during inference, PUDM exploits the prior knowledge of the scale between sparse
+point clouds and dense point clouds during training by parameterizing a rate
+factor. Moreover, PUDM exhibits strong noise robustness in experimental
+results. In the quantitative and qualitative evaluations on PU1K and PUGAN,
+PUDM significantly outperformed existing methods in terms of Chamfer Distance
+(CD) and Hausdorff Distance (HD), achieving state of the art (SOTA)
+performance.",cs.CV,['cs.CV']
+Bridging the Synthetic-to-Authentic Gap: Distortion-Guided Unsupervised Domain Adaptation for Blind Image Quality Assessment,Aobo Li · Jinjian Wu · Yongxu Liu · Leida Li, ,https://arxiv.org/abs/2405.04167,,2405.04167.pdf,Bridging the Synthetic-to-Authentic Gap: Distortion-Guided Unsupervised Domain Adaptation for Blind Image Quality Assessment,"The annotation of blind image quality assessment (BIQA) is labor-intensive
+and time-consuming, especially for authentic images. Training on synthetic data
+is expected to be beneficial, but synthetically trained models often suffer
+from poor generalization in real domains due to domain gaps. In this work, we
+make a key observation that introducing more distortion types in the synthetic
+dataset may not improve or even be harmful to generalizing authentic image
+quality assessment. To solve this challenge, we propose distortion-guided
+unsupervised domain adaptation for BIQA (DGQA), a novel framework that
+leverages adaptive multi-domain selection via prior knowledge from distortion
+to match the data distribution between the source domains and the target
+domain, thereby reducing negative transfer from the outlier source domains.
+Extensive experiments on two cross-domain settings (synthetic distortion to
+authentic distortion and synthetic distortion to algorithmic distortion) have
+demonstrated the effectiveness of our proposed DGQA. Besides, DGQA is
+orthogonal to existing model-based BIQA methods, and can be used in combination
+with such models to improve performance with less training data.",cs.CV,"['cs.CV', 'eess.IV']"
+Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters,Jiazuo Yu · Yunzhi Zhuge · Lu Zhang · Ping Hu · Dong Wang · Huchuan Lu · You He, ,https://arxiv.org/abs/2403.11549,,2403.11549.pdf,Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters,"Continual learning can empower vision-language models to continuously acquire
+new knowledge, without the need for access to the entire historical dataset.
+However, mitigating the performance degradation in large-scale models is
+non-trivial due to (i) parameter shifts throughout lifelong learning and (ii)
+significant computational burdens associated with full-model tuning. In this
+work, we present a parameter-efficient continual learning framework to
+alleviate long-term forgetting in incremental learning with vision-language
+models. Our approach involves the dynamic expansion of a pre-trained CLIP
+model, through the integration of Mixture-of-Experts (MoE) adapters in response
+to new tasks. To preserve the zero-shot recognition capability of
+vision-language models, we further introduce a Distribution Discriminative
+Auto-Selector (DDAS) that automatically routes in-distribution and
+out-of-distribution inputs to the MoE Adapter and the original CLIP,
+respectively. Through extensive experiments across various settings, our
+proposed method consistently outperforms previous state-of-the-art approaches
+while concurrently reducing parameter training burdens by 60%. Our code locates
+at https://github.com/JiazuoYu/MoE-Adapters4CL",cs.CV,['cs.CV']
+Unsupervised Blind Image Deblurring Based on Self-Enhancement,Lufei Chen · Xiangpeng Tian · Shuhua Xiong · Yinjie Lei · Chao Ren, ,,https://dl.acm.org/doi/abs/10.1145/3581783.3612535,,,,,nan
+UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity,Jialong Zuo · Hanyu Zhou · Ying Nie · Feng Zhang · Tianyu Guo · Nong Sang · Yunhe Wang · Changxin Gao, ,https://arxiv.org/abs/2312.03441v4,,2312.03441v4.pdf,UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity,"Existing text-based person retrieval datasets often have relatively
+coarse-grained text annotations. This hinders the model to comprehend the
+fine-grained semantics of query texts in real scenarios. To address this
+problem, we contribute a new benchmark named \textbf{UFineBench} for text-based
+person retrieval with ultra-fine granularity.
+  Firstly, we construct a new \textbf{dataset} named UFine6926. We collect a
+large number of person images and manually annotate each image with two
+detailed textual descriptions, averaging 80.8 words each. The average word
+count is three to four times that of the previous datasets. In addition of
+standard in-domain evaluation, we also propose a special \textbf{evaluation
+paradigm} more representative of real scenarios. It contains a new evaluation
+set with cross domains, cross textual granularity and cross textual styles,
+named UFine3C, and a new evaluation metric for accurately measuring retrieval
+ability, named mean Similarity Distribution (mSD). Moreover, we propose CFAM, a
+more efficient \textbf{algorithm} especially designed for text-based person
+retrieval with ultra fine-grained texts. It achieves fine granularity mining by
+adopting a shared cross-modal granularity decoder and hard negative match
+mechanism.
+  With standard in-domain evaluation, CFAM establishes competitive performance
+across various datasets, especially on our ultra fine-grained UFine6926.
+Furthermore, by evaluating on UFine3C, we demonstrate that training on our
+UFine6926 significantly improves generalization to real scenarios compared with
+other coarse-grained datasets. The dataset and code will be made publicly
+available at \url{https://github.com/Zplusdragon/UFineBench}.",cs.CV,['cs.CV']
+Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency,Xu Yingjie · Bangzhen Liu · Hao Tang · Bailin Deng · Shengfeng He, ,https://arxiv.org/abs/2403.17638v1,,2403.17638v1.pdf,Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency,"We propose a voxel-based optimization framework, ReVoRF, for few-shot
+radiance fields that strategically address the unreliability in pseudo novel
+view synthesis. Our method pivots on the insight that relative depth
+relationships within neighboring regions are more reliable than the absolute
+color values in disoccluded areas. Consequently, we devise a bilateral
+geometric consistency loss that carefully navigates the trade-off between color
+fidelity and geometric accuracy in the context of depth consistency for
+uncertain regions. Moreover, we present a reliability-guided learning strategy
+to discern and utilize the variable quality across synthesized views,
+complemented by a reliability-aware voxel smoothing algorithm that smoothens
+the transition between reliable and unreliable data patches. Our approach
+allows for a more nuanced use of all available data, promoting enhanced
+learning from regions previously considered unsuitable for high-quality
+reconstruction. Extensive experiments across diverse datasets reveal that our
+approach attains significant gains in efficiency and accuracy, delivering
+rendering speeds of 3 FPS, 7 mins to train a $360^\circ$ scene, and a 5\%
+improvement in PSNR over existing few-shot methods. Code is available at
+https://github.com/HKCLynn/ReVoRF.",cs.CV,['cs.CV']
+Entity-NeRF: Detecting and Removing Moving Entities in Urban Scenes,Takashi Otonari · Satoshi Ikehata · Kiyoharu Aizawa, ,https://arxiv.org/abs/2403.16141,,2403.16141.pdf,Entity-NeRF: Detecting and Removing Moving Entities in Urban Scenes,"Recent advancements in the study of Neural Radiance Fields (NeRF) for dynamic
+scenes often involve explicit modeling of scene dynamics. However, this
+approach faces challenges in modeling scene dynamics in urban environments,
+where moving objects of various categories and scales are present. In such
+settings, it becomes crucial to effectively eliminate moving objects to
+accurately reconstruct static backgrounds. Our research introduces an
+innovative method, termed here as Entity-NeRF, which combines the strengths of
+knowledge-based and statistical strategies. This approach utilizes entity-wise
+statistics, leveraging entity segmentation and stationary entity classification
+through thing/stuff segmentation. To assess our methodology, we created an
+urban scene dataset masked with moving objects. Our comprehensive experiments
+demonstrate that Entity-NeRF notably outperforms existing techniques in
+removing moving objects and reconstructing static urban backgrounds, both
+quantitatively and qualitatively.",cs.CV,['cs.CV']
+PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation,Yuqi Wang · Yuntao Chen · Xingyu Liao · Lue Fan · Zhaoxiang Zhang, ,https://arxiv.org/abs/2306.10013,,2306.10013.pdf,PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation,"Comprehensive modeling of the surrounding 3D world is key to the success of
+autonomous driving. However, existing perception tasks like object detection,
+road structure segmentation, depth & elevation estimation, and open-set object
+localization each only focus on a small facet of the holistic 3D scene
+understanding task. This divide-and-conquer strategy simplifies the algorithm
+development procedure at the cost of losing an end-to-end unified solution to
+the problem. In this work, we address this limitation by studying camera-based
+3D panoptic segmentation, aiming to achieve a unified occupancy representation
+for camera-only 3D scene understanding. To achieve this, we introduce a novel
+method called PanoOcc, which utilizes voxel queries to aggregate spatiotemporal
+information from multi-frame and multi-view images in a coarse-to-fine scheme,
+integrating feature learning and scene representation into a unified occupancy
+representation. We have conducted extensive ablation studies to verify the
+effectiveness and efficiency of the proposed method. Our approach achieves new
+state-of-the-art results for camera-based semantic segmentation and panoptic
+segmentation on the nuScenes dataset. Furthermore, our method can be easily
+extended to dense occupancy prediction and has shown promising performance on
+the Occ3D benchmark. The code will be released at
+https://github.com/Robertwyq/PanoOcc.",cs.CV,"['cs.CV', 'cs.RO']"
+HIT: Estimating Internal Human Implicit Tissues from the Body Surface,Marilyn Keller · Vaibhav ARORA · Abdelmouttaleb Dakri · Shivam Chandhok · Jürgen Machann · Andreas Fritsche · Michael J. Black · Sergi Pujades,https://hit.is.tue.mpg.de,,https://www.youtube.com/watch?v=3u4emFF3DcE,,,,,nan
+Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation,Renshuai Liu · Bowen Ma · Wei Zhang · Zhipeng Hu · Changjie Fan · Tangjie Lv · Yu Ding · Xuan Cheng, ,https://arxiv.org/abs/2401.01207,,2401.01207.pdf,Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation,"In human-centric content generation, the pre-trained text-to-image models
+struggle to produce user-wanted portrait images, which retain the identity of
+individuals while exhibiting diverse expressions. This paper introduces our
+efforts towards personalized face generation. To this end, we propose a novel
+multi-modal face generation framework, capable of simultaneous
+identity-expression control and more fine-grained expression synthesis. Our
+expression control is so sophisticated that it can be specialized by the
+fine-grained emotional vocabulary. We devise a novel diffusion model that can
+undertake the task of simultaneously face swapping and reenactment. Due to the
+entanglement of identity and expression, it's nontrivial to separately and
+precisely control them in one framework, thus has not been explored yet. To
+overcome this, we propose several innovative designs in the conditional
+diffusion model, including balancing identity and expression encoder, improved
+midpoint sampling, and explicitly background conditioning. Extensive
+experiments have demonstrated the controllability and scalability of the
+proposed framework, in comparison with state-of-the-art text-to-image, face
+swapping, and face reenactment methods.",cs.CV,['cs.CV']
+3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation,Songchun Zhang · Yibo Zhang · Quan Zheng · Rui Ma · Wei Hua · Hujun Bao · Weiwei Xu · Changqing Zou, ,https://arxiv.org/abs/2403.09439,,2403.09439.pdf,3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation,"Text-driven 3D scene generation techniques have made rapid progress in recent
+years. Their success is mainly attributed to using existing generative models
+to iteratively perform image warping and inpainting to generate 3D scenes.
+However, these methods heavily rely on the outputs of existing models, leading
+to error accumulation in geometry and appearance that prevent the models from
+being used in various scenarios (e.g., outdoor and unreal scenarios). To
+address this limitation, we generatively refine the newly generated local views
+by querying and aggregating global 3D information, and then progressively
+generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF
+as a unified representation of the 3D scene to constrain global 3D consistency,
+and propose a generative refinement network to synthesize new contents with
+higher quality by exploiting the natural image prior from 2D diffusion model as
+well as the global 3D information of the current scene. Our extensive
+experiments demonstrate that, in comparison to previous methods, our approach
+supports wide variety of scene generation and arbitrary camera trajectories
+with improved visual quality and 3D consistency.",cs.CV,"['cs.CV', 'cs.AI']"
+Accurate Spatial Gene Expression Prediction by Integrating Multi-Resolution Features,Youngmin Chung · Ji Hun Ha · Kyeong Chan Im · Joo Sang Lee, ,https://arxiv.org/abs/2403.07592v1,,2403.07592v1.pdf,Accurate Spatial Gene Expression Prediction by integrating Multi-resolution features,"Recent advancements in Spatial Transcriptomics (ST) technology have
+facilitated detailed gene expression analysis within tissue contexts. However,
+the high costs and methodological limitations of ST necessitate a more robust
+predictive model. In response, this paper introduces TRIPLEX, a novel deep
+learning framework designed to predict spatial gene expression from Whole Slide
+Images (WSIs). TRIPLEX uniquely harnesses multi-resolution features, capturing
+cellular morphology at individual spots, the local context around these spots,
+and the global tissue organization. By integrating these features through an
+effective fusion strategy, TRIPLEX achieves accurate gene expression
+prediction. Our comprehensive benchmark study, conducted on three public ST
+datasets and supplemented with Visium data from 10X Genomics, demonstrates that
+TRIPLEX outperforms current state-of-the-art models in Mean Squared Error
+(MSE), Mean Absolute Error (MAE), and Pearson Correlation Coefficient (PCC).
+The model's predictions align closely with ground truth gene expression
+profiles and tumor annotations, underscoring TRIPLEX's potential in advancing
+cancer diagnosis and treatment.",cs.CV,['cs.CV']
+Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM,Pingping Zhang · Tianyu Yan · Yang Liu · Huchuan Lu, ,https://arxiv.org/abs/2404.04996,,2404.04996.pdf,Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM,"As an important pillar of underwater intelligence, Marine Animal Segmentation
+(MAS) involves segmenting animals within marine environments. Previous methods
+don't excel in extracting long-range contextual features and overlook the
+connectivity between discrete pixels. Recently, Segment Anything Model (SAM)
+offers a universal framework for general segmentation tasks. Unfortunately,
+trained with natural images, SAM does not obtain the prior knowledge from
+marine images. In addition, the single-position prompt of SAM is very
+insufficient for prior guidance. To address these issues, we propose a novel
+feature learning framework, named Dual-SAM for high-performance MAS. To this
+end, we first introduce a dual structure with SAM's paradigm to enhance feature
+learning of marine images. Then, we propose a Multi-level Coupled Prompt (MCP)
+strategy to instruct comprehensive underwater prior information, and enhance
+the multi-level features of SAM's encoder with adapters. Subsequently, we
+design a Dilated Fusion Attention Module (DFAM) to progressively integrate
+multi-level features from SAM's encoder. Finally, instead of directly
+predicting the masks of marine animals, we propose a Criss-Cross Connectivity
+Prediction (C$^3$P) paradigm to capture the inter-connectivity between discrete
+pixels. With dual decoders, it generates pseudo-labels and achieves mutual
+supervision for complementary feature representations, resulting in
+considerable improvements over previous techniques. Extensive experiments
+verify that our proposed method achieves state-of-the-art performances on five
+widely-used MAS datasets. The code is available at
+https://github.com/Drchip61/Dual_SAM.",cs.CV,"['cs.CV', 'cs.MM']"
+Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans,Romain Loiseau · Elliot Vincent · Mathieu Aubry · Loic Landrieu,https://romainloiseau.fr/learnable-earth-parser/,,https://www.youtube.com/watch?v=0PkxeT17e8Q,,,,,nan
+Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications,Yuwen Xiong · Zhiqi Li · Yuntao Chen · Feng Wang · Xizhou Zhu · Jiapeng Luo · Wenhai Wang · Tong Lu · Hongsheng Li · Yu Qiao · Lewei Lu · Jie Zhou · Jifeng Dai, ,https://arxiv.org/abs/2401.06197,,2401.06197.pdf,Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications,"We introduce Deformable Convolution v4 (DCNv4), a highly efficient and
+effective operator designed for a broad spectrum of vision applications. DCNv4
+addresses the limitations of its predecessor, DCNv3, with two key enhancements:
+1. removing softmax normalization in spatial aggregation to enhance its dynamic
+property and expressive power and 2. optimizing memory access to minimize
+redundant operations for speedup. These improvements result in a significantly
+faster convergence compared to DCNv3 and a substantial increase in processing
+speed, with DCNv4 achieving more than three times the forward speed. DCNv4
+demonstrates exceptional performance across various tasks, including image
+classification, instance and semantic segmentation, and notably, image
+generation. When integrated into generative models like U-Net in the latent
+diffusion model, DCNv4 outperforms its baseline, underscoring its possibility
+to enhance generative models. In practical applications, replacing DCNv3 with
+DCNv4 in the InternImage model to create FlashInternImage results in up to 80%
+speed increase and further performance improvement without further
+modifications. The advancements in speed and efficiency of DCNv4, combined with
+its robust performance across diverse vision tasks, show its potential as a
+foundational building block for future vision models.",cs.CV,['cs.CV']
+A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization,Hongwei Ren · Jiadong Zhu · Yue Zhou · Haotian FU · Yulong Huang · Bojun Cheng, ,https://arxiv.org/abs/2403.19412,,2403.19412.pdf,A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization,"Event cameras exhibit remarkable attributes such as high dynamic range,
+asynchronicity, and low latency, making them highly suitable for vision tasks
+that involve high-speed motion in challenging lighting conditions. These
+cameras implicitly capture movement and depth information in events, making
+them appealing sensors for Camera Pose Relocalization (CPR) tasks.
+Nevertheless, existing CPR networks based on events neglect the pivotal
+fine-grained temporal information in events, resulting in unsatisfactory
+performance. Moreover, the energy-efficient features are further compromised by
+the use of excessively complex models, hindering efficient deployment on edge
+devices. In this paper, we introduce PEPNet, a simple and effective point-based
+network designed to regress six degrees of freedom (6-DOFs) event camera poses.
+We rethink the relationship between the event camera and CPR tasks, leveraging
+the raw Point Cloud directly as network input to harness the high-temporal
+resolution and inherent sparsity of events. PEPNet is adept at abstracting the
+spatial and implicit temporal features through hierarchical structure and
+explicit temporal features by Attentive Bi-directional Long Short-Term Memory
+(A-Bi-LSTM). By employing a carefully crafted lightweight design, PEPNet
+delivers state-of-the-art (SOTA) performance on both indoor and outdoor
+datasets with meager computational resources. Specifically, PEPNet attains a
+significant 38% and 33% performance improvement on the random split IJRR and
+M3ED datasets, respectively. Moreover, the lightweight design version
+PEPNet$_{tiny}$ accomplishes results comparable to the SOTA while employing a
+mere 0.5% of the parameters.",cs.CV,['cs.CV']
+Attention Calibration for Disentangled Text-to-Image Personalization,Yanbing Zhang · Mengping Yang · Qin Zhou · Zhe Wang, ,https://arxiv.org/abs/2403.18551,,2403.18551.pdf,Attention Calibration for Disentangled Text-to-Image Personalization,"Recent thrilling progress in large-scale text-to-image (T2I) models has
+unlocked unprecedented synthesis quality of AI-generated content (AIGC)
+including image generation, 3D and video composition. Further, personalized
+techniques enable appealing customized production of a novel concept given only
+several images as reference. However, an intriguing problem persists: Is it
+possible to capture multiple, novel concepts from one single reference image?
+In this paper, we identify that existing approaches fail to preserve visual
+consistency with the reference image and eliminate cross-influence from
+concepts. To alleviate this, we propose an attention calibration mechanism to
+improve the concept-level understanding of the T2I model. Specifically, we
+first introduce new learnable modifiers bound with classes to capture
+attributes of multiple concepts. Then, the classes are separated and
+strengthened following the activation of the cross-attention operation,
+ensuring comprehensive and self-contained concepts. Additionally, we suppress
+the attention activation of different classes to mitigate mutual influence
+among concepts. Together, our proposed method, dubbed DisenDiff, can learn
+disentangled multiple concepts from one single image and produce novel
+customized images with learned concepts. We demonstrate that our method
+outperforms the current state of the art in both qualitative and quantitative
+evaluations. More importantly, our proposed techniques are compatible with LoRA
+and inpainting pipelines, enabling more interactive experiences.",cs.CV,['cs.CV']
+LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching,Yixun Liang · Xin Yang · Jiantao Lin · Haodong LI · Xiaogang Xu · Ying-Cong Chen, ,https://arxiv.org/abs/2311.11284,,2311.11284.pdf,LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching,"The recent advancements in text-to-3D generation mark a significant milestone
+in generative models, unlocking new possibilities for creating imaginative 3D
+assets across various real-world scenarios. While recent advancements in
+text-to-3D generation have shown promise, they often fall short in rendering
+detailed and high-quality 3D models. This problem is especially prevalent as
+many methods base themselves on Score Distillation Sampling (SDS). This paper
+identifies a notable deficiency in SDS, that it brings inconsistent and
+low-quality updating direction for the 3D model, causing the over-smoothing
+effect. To address this, we propose a novel approach called Interval Score
+Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes
+interval-based score matching to counteract over-smoothing. Furthermore, we
+incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline.
+Extensive experiments show that our model largely outperforms the
+state-of-the-art in quality and training efficiency.",cs.CV,"['cs.CV', 'cs.GR', 'cs.MM']"
+Object Dynamics Modeling with Hierarchical Point Cloud-based Representations,Chanho Kim · Li Fuxin, ,https://arxiv.org/abs/2404.06044,,2404.06044.pdf,Object Dynamics Modeling with Hierarchical Point Cloud-based Representations,"Modeling object dynamics with a neural network is an important problem with
+numerous applications. Most recent work has been based on graph neural
+networks. However, physics happens in 3D space, where geometric information
+potentially plays an important role in modeling physical phenomena. In this
+work, we propose a novel U-net architecture based on continuous point
+convolution which naturally embeds information from 3D coordinates and allows
+for multi-scale feature representations with established downsampling and
+upsampling procedures. Bottleneck layers in the downsampled point clouds lead
+to better long-range interaction modeling. Besides, the flexibility of point
+convolutions allows our approach to generalize to sparsely sampled points from
+mesh vertices and dynamically generate features on important interaction points
+on mesh faces. Experimental results demonstrate that our approach significantly
+improves the state-of-the-art, especially in scenarios that require accurate
+gravity or collision reasoning.",cs.CV,['cs.CV']
+CAD: Photorealistic 3D Generation via Adversarial Distillation,Ziyu Wan · Despoina Paschalidou · Ian Huang · Hongyu Liu · Bokui Shen · Xiaoyu Xiang · Jing Liao · Leonidas Guibas,http://raywzy.com/CAD/,https://arxiv.org/abs/2312.06663,,2312.06663.pdf,CAD: Photorealistic 3D Generation via Adversarial Distillation,"The increased demand for 3D data in AR/VR, robotics and gaming applications,
+gave rise to powerful generative pipelines capable of synthesizing high-quality
+3D objects. Most of these models rely on the Score Distillation Sampling (SDS)
+algorithm to optimize a 3D representation such that the rendered image
+maintains a high likelihood as evaluated by a pre-trained diffusion model.
+However, finding a correct mode in the high-dimensional distribution produced
+by the diffusion model is challenging and often leads to issues such as
+over-saturation, over-smoothing, and Janus-like artifacts. In this paper, we
+propose a novel learning paradigm for 3D synthesis that utilizes pre-trained
+diffusion models. Instead of focusing on mode-seeking, our method directly
+models the distribution discrepancy between multi-view renderings and diffusion
+priors in an adversarial manner, which unlocks the generation of high-fidelity
+and photorealistic 3D content, conditioned on a single image and prompt.
+Moreover, by harnessing the latent space of GANs and expressive diffusion model
+priors, our method facilitates a wide variety of 3D applications including
+single-view reconstruction, high diversity generation and continuous 3D
+interpolation in the open domain. The experiments demonstrate the superiority
+of our pipeline compared to previous works in terms of generation quality and
+diversity.",cs.CV,"['cs.CV', 'cs.GR']"
+Gaussian Shell Maps for Efficient 3D Human Generation,Rameen Abdal · Wang Yifan · Zifan Shi · Yinghao Xu · Ryan Po · Zhengfei Kuang · Qifeng Chen · Dit-Yan Yeung · Gordon Wetzstein,https://rameenabdal.github.io/GaussianShellMaps/,https://arxiv.org/abs/2311.17857v1,,2311.17857v1.pdf,Gaussian Shell Maps for Efficient 3D Human Generation,"Efficient generation of 3D digital humans is important in several industries,
+including virtual reality, social media, and cinematic production. 3D
+generative adversarial networks (GANs) have demonstrated state-of-the-art
+(SOTA) quality and diversity for generated assets. Current 3D GAN
+architectures, however, typically rely on volume representations, which are
+slow to render, thereby hampering the GAN training and requiring
+multi-view-inconsistent 2D upsamplers. Here, we introduce Gaussian Shell Maps
+(GSMs) as a framework that connects SOTA generator network architectures with
+emerging 3D Gaussian rendering primitives using an articulable multi
+shell--based scaffold. In this setting, a CNN generates a 3D texture stack with
+features that are mapped to the shells. The latter represent inflated and
+deflated versions of a template surface of a digital human in a canonical body
+pose. Instead of rasterizing the shells directly, we sample 3D Gaussians on the
+shells whose attributes are encoded in the texture features. These Gaussians
+are efficiently and differentiably rendered. The ability to articulate the
+shells is important during GAN training and, at inference time, to deform a
+body into arbitrary user-defined poses. Our efficient rendering scheme bypasses
+the need for view-inconsistent upsamplers and achieves high-quality multi-view
+consistent renderings at a native resolution of $512 \times 512$ pixels. We
+demonstrate that GSMs successfully generate 3D humans when trained on
+single-view datasets, including SHHQ and DeepFashion.",cs.CV,"['cs.CV', 'cs.GR']"
+3D-Aware Face Editing via Warping-Guided Latent Direction Learning,Yuhao Cheng · Zhuo Chen · Xingyu Ren · Wenhan Zhu · Zhengqin Xu · Di Xu · Yang Changpeng · Yichao Yan, ,https://arxiv.org/abs/2402.14000,,2402.14000.pdf,Real-time 3D-aware Portrait Editing from a Single Image,"This work presents 3DPE, a practical method that can efficiently edit a face
+image following given prompts, like reference images or text descriptions, in a
+3D-aware manner. To this end, a lightweight module is distilled from a 3D
+portrait generator and a text-to-image model, which provide prior knowledge of
+face geometry and superior editing capability, respectively. Such a design
+brings two compelling advantages over existing approaches. First, our system
+achieves real-time editing with a feedforward network (i.e., ~0.04s per image),
+over 100x faster than the second competitor. Second, thanks to the powerful
+priors, our module could focus on the learning of editing-related variations,
+such that it manages to handle various types of editing simultaneously in the
+training phase and further supports fast adaptation to user-specified
+customized types of editing during inference (e.g., with ~5min fine-tuning per
+style). The code, the model, and the interface will be made publicly available
+to facilitate future research.",cs.CV,['cs.CV']
+NeRFiller: Completing Scenes via Generative 3D Inpainting,Ethan Weber · Aleksander Holynski · Varun Jampani · Saurabh Saxena · Noah Snavely · Abhishek Kar · Angjoo Kanazawa,https://ethanweber.me/nerfiller/,https://arxiv.org/abs/2312.04560,,2312.04560.pdf,NeRFiller: Completing Scenes via Generative 3D Inpainting,"We propose NeRFiller, an approach that completes missing portions of a 3D
+capture via generative 3D inpainting using off-the-shelf 2D visual generative
+models. Often parts of a captured 3D scene or object are missing due to mesh
+reconstruction failures or a lack of observations (e.g., contact regions, such
+as the bottom of objects, or hard-to-reach areas). We approach this challenging
+3D inpainting problem by leveraging a 2D inpainting diffusion model. We
+identify a surprising behavior of these models, where they generate more 3D
+consistent inpaints when images form a 2$\times$2 grid, and show how to
+generalize this behavior to more than four images. We then present an iterative
+framework to distill these inpainted regions into a single consistent 3D scene.
+In contrast to related works, we focus on completing scenes rather than
+deleting foreground objects, and our approach does not require tight 2D object
+masks or text. We compare our approach to relevant baselines adapted to our
+setting on a variety of scenes, where NeRFiller creates the most 3D consistent
+and plausible scene completions. Our project page is at
+https://ethanweber.me/nerfiller.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+Exploring the Transferability of Visual Prompting for Multimodal Large Language Models,Yichi Zhang · Yinpeng Dong · Siyuan Zhang · Tianzan Min · Hang Su · Jun Zhu, ,https://arxiv.org/abs/2404.11207v1,,2404.11207v1.pdf,Exploring the Transferability of Visual Prompting for Multimodal Large Language Models,"Although Multimodal Large Language Models (MLLMs) have demonstrated promising
+versatile capabilities, their performance is still inferior to specialized
+models on downstream tasks, which makes adaptation necessary to enhance their
+utility. However, fine-tuning methods require independent training for every
+model, leading to huge computation and memory overheads. In this paper, we
+propose a novel setting where we aim to improve the performance of diverse
+MLLMs with a group of shared parameters optimized for a downstream task. To
+achieve this, we propose Transferable Visual Prompting (TVP), a simple and
+effective approach to generate visual prompts that can transfer to different
+models and improve their performance on downstream tasks after trained on only
+one model. We introduce two strategies to address the issue of cross-model
+feature corruption of existing visual prompting methods and enhance the
+transferability of the learned prompts, including 1) Feature Consistency
+Alignment: which imposes constraints to the prompted feature changes to
+maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which
+encourages the prompted images to contain richer task-specific semantics with
+language guidance. We validate the effectiveness of TVP through extensive
+experiments with 6 modern MLLMs on a wide variety of tasks ranging from object
+recognition and counting to multimodal reasoning and hallucination correction.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Utility-Fairness Trade-Offs and How to Find Them,Sepehr Dehdashtian · Bashir Sadeghi · Vishnu Naresh Boddeti,https://sepehrdehdashtian.github.io/Papers/U-FaTE/index.html,https://arxiv.org/abs/2404.09454v1,,2404.09454v1.pdf,Utility-Fairness Trade-Offs and How to Find Them,"When building classification systems with demographic fairness
+considerations, there are two objectives to satisfy: 1) maximizing utility for
+the specific task and 2) ensuring fairness w.r.t. a known demographic
+attribute. These objectives often compete, so optimizing both can lead to a
+trade-off between utility and fairness. While existing works acknowledge the
+trade-offs and study their limits, two questions remain unanswered: 1) What are
+the optimal trade-offs between utility and fairness? and 2) How can we
+numerically quantify these trade-offs from data for a desired prediction task
+and demographic attribute of interest? This paper addresses these questions. We
+introduce two utility-fairness trade-offs: the Data-Space and Label-Space
+Trade-off. The trade-offs reveal three regions within the utility-fairness
+plane, delineating what is fully and partially possible and impossible. We
+propose U-FaTE, a method to numerically quantify the trade-offs for a given
+prediction task and group fairness definition from data samples. Based on the
+trade-offs, we introduce a new scheme for evaluating representations. An
+extensive evaluation of fair representation learning methods and
+representations from over 1000 pre-trained models revealed that most current
+approaches are far from the estimated and achievable fairness-utility
+trade-offs across multiple datasets and prediction tasks.",cs.CV,"['cs.CV', 'cs.CY', 'cs.LG']"
+Observation-Guided Diffusion Probabilistic Models,Junoh Kang · Jinyoung Choi · Sungik Choi · Bohyung Han, ,https://arxiv.org/abs/2310.04041,,2310.04041.pdf,Observation-Guided Diffusion Probabilistic Models,"We propose a novel diffusion-based image generation method called the
+observation-guided diffusion probabilistic model (OGDM), which effectively
+addresses the tradeoff between quality control and fast sampling. Our approach
+reestablishes the training objective by integrating the guidance of the
+observation process with the Markov chain in a principled way. This is achieved
+by introducing an additional loss term derived from the observation based on a
+conditional discriminator on noise level, which employs a Bernoulli
+distribution indicating whether its input lies on the (noisy) real manifold or
+not. This strategy allows us to optimize the more accurate negative
+log-likelihood induced in the inference stage especially when the number of
+function evaluations is limited. The proposed training scheme is also
+advantageous even when incorporated only into the fine-tuning process, and it
+is compatible with various fast inference strategies since our method yields
+better denoising networks using the exactly the same inference procedure
+without incurring extra computational cost. We demonstrate the effectiveness of
+our training algorithm using diverse inference techniques on strong diffusion
+model baselines. Our implementation is available at
+https://github.com/Junoh-Kang/OGDM_edm.",cs.LG,"['cs.LG', 'cs.AI']"
+FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition,Ganggui Ding · Canyu Zhao · Wen Wang · Zhen Yang · Zide Liu · Hao Chen · Chunhua Shen,https://aim-uofa.github.io/FreeCustom/,https://arxiv.org/abs/2405.13870,,2405.13870.pdf,FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition,"Benefiting from large-scale pre-trained text-to-image (T2I) generative
+models, impressive progress has been achieved in customized image generation,
+which aims to generate user-specified concepts. Existing approaches have
+extensively focused on single-concept customization and still encounter
+challenges when it comes to complex scenarios that involve combining multiple
+concepts. These approaches often require retraining/fine-tuning using a few
+images, leading to time-consuming training processes and impeding their swift
+implementation. Furthermore, the reliance on multiple images to represent a
+singular concept increases the difficulty of customization. To this end, we
+propose FreeCustom, a novel tuning-free method to generate customized images of
+multi-concept composition based on reference concepts, using only one image per
+concept as input. Specifically, we introduce a new multi-reference
+self-attention (MRSA) mechanism and a weighted mask strategy that enables the
+generated image to access and focus more on the reference concepts. In
+addition, MRSA leverages our key finding that input concepts are better
+preserved when providing images with context interactions. Experiments show
+that our method's produced images are consistent with the given concepts and
+better aligned with the input text. Our method outperforms or performs on par
+with other training-based methods in terms of multi-concept composition and
+single-concept customization, but is simpler. Codes can be found at
+https://github.com/aim-uofa/FreeCustom.",cs.CV,['cs.CV']
+ModaVerse: Efficiently Transforming Modalities with LLMs,Xinyu Wang · Bohan Zhuang · Qi Wu, ,https://arxiv.org/abs/2401.06395,,2401.06395.pdf,ModaVerse: Efficiently Transforming Modalities with LLMs,"Humans possess the capability to comprehend diverse modalities and seamlessly
+transfer information between them. In this work, we introduce ModaVerse, a
+Multi-modal Large Language Model (MLLM) capable of comprehending and
+transforming content across various modalities including images, videos, and
+audio. Predominant MLLM frameworks have largely relied on the alignment of
+latent spaces of textual and non-textual features. This alignment process,
+which synchronizes a language model trained on textual data with encoders and
+decoders trained on multi-modal data, often necessitates extensive training of
+several projection layers in multiple stages. Inspired by LLM-as-agent
+methodologies, we propose a novel Input/Output (I/O) alignment mechanism that
+operates directly at the level of natural language. It aligns the LLM's output
+with the input of generative models, avoiding the complexities associated with
+latent feature alignments, and simplifying the multiple training stages of
+existing MLLMs into a single, efficient process. This conceptual advancement
+leads to significant reductions in both data and computational costs. By
+conducting experiments on several benchmarks, we demonstrate that our approach
+attains comparable performance with the state of the art while achieving
+considerable efficiencies in data usage and training duration.",cs.CV,['cs.CV']
+Targeted Representation Alignment for Open-World Semi-Supervised Learning,Ruixuan Xiao · Lei Feng · Kai Tang · Junbo Zhao · Yixuan Li · Gang Chen · Haobo Wang, ,https://arxiv.org/abs/2311.03524,,2311.03524.pdf,A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning,"Open-world semi-supervised learning aims at inferring both known and novel
+classes in unlabeled data, by harnessing prior knowledge from a labeled set
+with known classes. Despite its importance, there is a lack of theoretical
+foundations for this problem. This paper bridges the gap by formalizing a
+graph-theoretic framework tailored for the open-world setting, where the
+clustering can be theoretically characterized by graph factorization. Our
+graph-theoretic framework illuminates practical algorithms and provides
+guarantees. In particular, based on our graph formulation, we apply the
+algorithm called Spectral Open-world Representation Learning (SORL), and show
+that minimizing our loss is equivalent to performing spectral decomposition on
+the graph. Such equivalence allows us to derive a provable error bound on the
+clustering performance for both known and novel classes, and analyze rigorously
+when labeled data helps. Empirically, SORL can match or outperform several
+strong baselines on common benchmark datasets, which is appealing for practical
+usage while enjoying theoretical guarantees.",cs.LG,['cs.LG']
+PELA: Learning Parameter-Efficient Models with Low-Rank Approximation,Yangyang Guo · Guangzhi Wang · Mohan Kankanhalli, ,https://arxiv.org/abs/2310.10700,,2310.10700.pdf,PELA: Learning Parameter-Efficient Models with Low-Rank Approximation,"Applying a pre-trained large model to downstream tasks is prohibitive under
+resource-constrained conditions. Recent dominant approaches for addressing
+efficiency issues involve adding a few learnable parameters to the fixed
+backbone model. This strategy, however, leads to more challenges in loading
+large models for downstream fine-tuning with limited resources. In this paper,
+we propose a novel method for increasing the parameter efficiency of
+pre-trained models by introducing an intermediate pre-training stage. To this
+end, we first employ low-rank approximation to compress the original large
+model and then devise a feature distillation module and a weight perturbation
+regularization module. These modules are specifically designed to enhance the
+low-rank model. In particular, we update only the low-rank model while freezing
+the backbone parameters during pre-training. This allows for direct and
+efficient utilization of the low-rank model for downstream fine-tuning tasks.
+The proposed method achieves both efficiencies in terms of required parameters
+and computation time while maintaining comparable results with minimal
+modifications to the backbone architecture. Specifically, when applied to three
+vision-only and one vision-language Transformer models, our approach often
+demonstrates a merely $\sim$0.6 point decrease in performance while reducing
+the original parameter size by 1/3 to 2/3.",cs.CV,['cs.CV']
+Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning,Shiming Chen · Wenjin Hou · Salman Khan · Fahad Shahbaz Khan, ,https://arxiv.org/abs/2404.07713,,2404.07713.pdf,Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning,"Zero-shot learning (ZSL) recognizes the unseen classes by conducting
+visual-semantic interactions to transfer semantic knowledge from seen classes
+to unseen ones, supported by semantic information (e.g., attributes). However,
+existing ZSL methods simply extract visual features using a pre-trained network
+backbone (i.e., CNN or ViT), which fail to learn matched visual-semantic
+correspondences for representing semantic-related visual features as lacking of
+the guidance of semantic information, resulting in undesirable visual-semantic
+interactions. To tackle this issue, we propose a progressive semantic-guided
+vision transformer for zero-shot learning (dubbed ZSLViT). ZSLViT mainly
+considers two properties in the whole network: i) discover the semantic-related
+visual representations explicitly, and ii) discard the semantic-unrelated
+visual information. Specifically, we first introduce semantic-embedded token
+learning to improve the visual-semantic correspondences via semantic
+enhancement and discover the semantic-related visual tokens explicitly with
+semantic-guided token attention. Then, we fuse low semantic-visual
+correspondence visual tokens to discard the semantic-unrelated visual
+information for visual enhancement. These two operations are integrated into
+various encoders to progressively learn semantic-related visual representations
+for accurate visual-semantic interactions in ZSL. The extensive experiments
+show that our ZSLViT achieves significant performance gains on three popular
+benchmark datasets, i.e., CUB, SUN, and AWA2.",cs.CV,"['cs.CV', 'cs.LG']"
+Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection,Wenjun Hui · Zhenfeng Zhu · Shuai Zheng · Yao Zhao, ,https://arxiv.org/html/2403.01968v1,,2403.01968v1.pdf,Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection,"Camouflage poses challenges in distinguishing a static target, whereas any
+movement of the target can break this disguise. Existing video camouflaged
+object detection (VCOD) approaches take noisy motion estimation as input or
+model motion implicitly, restricting detection performance in complex dynamic
+scenes. In this paper, we propose a novel Explicit Motion handling and
+Interactive Prompting framework for VCOD, dubbed EMIP, which handles motion
+cues explicitly using a frozen pre-trained optical flow fundamental model. EMIP
+is characterized by a two-stream architecture for simultaneously conducting
+camouflaged segmentation and optical flow estimation. Interactions across the
+dual streams are realized in an interactive prompting way that is inspired by
+emerging visual prompt learning. Two learnable modules, i.e. the camouflaged
+feeder and motion collector, are designed to incorporate segmentation-to-motion
+and motion-to-segmentation prompts, respectively, and enhance outputs of the
+both streams. The prompt fed to the motion stream is learned by supervising
+optical flow in a self-supervised manner. Furthermore, we show that long-term
+historical information can also be incorporated as a prompt into EMIP and
+achieve more robust results with temporal consistency. Experimental results
+demonstrate that our EMIP achieves new state-of-the-art records on popular VCOD
+benchmarks. The code will be publicly available.",cs.CV,['cs.CV']
+Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment,Zheren Fu · Lei Zhang · Hou Xia · Zhendong Mao,https://github.com/CrossmodalGroup/LAPS,https://arxiv.org/html/2312.05278v2,,2312.05278v2.pdf,Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects,"Large Vision Language Models (LVLMs) have demonstrated impressive zero-shot
+capabilities in various vision-language dialogue scenarios. However, the
+absence of fine-grained visual object detection hinders the model from
+understanding the details of images, leading to irreparable visual
+hallucinations and factual errors. In this paper, we propose Lyrics, a novel
+multi-modal pre-training and instruction fine-tuning paradigm that bootstraps
+vision-language alignment from fine-grained cross-modal collaboration. Building
+on the foundation of BLIP-2, Lyrics infuses local visual features extracted
+from a visual refiner that includes image tagging, object detection and
+semantic segmentation modules into the Querying Transformer, while on the text
+side, the language inputs equip the boundary boxes and tags derived from the
+visual refiner. We further introduce a two-stage training scheme, in which the
+pre-training stage bridges the modality gap through explicit and comprehensive
+vision-language alignment targets. During the instruction fine-tuning stage, we
+introduce semantic-aware visual feature extraction, a crucial method that
+enables the model to extract informative features from concrete visual objects.
+Our approach achieves robust performance on 13 datasets across various
+vision-language tasks, and demonstrates promising multi-modal understanding,
+perception and conversation capabilities in 11 scenario-based benchmark
+toolkits.",cs.CL,['cs.CL']
+"GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitation",Zifan Wang · Junyu Chen · Ziqing Chen · Pengwei Xie · Rui Chen · Li Yi, ,https://arxiv.org/abs/2401.00929,,2401.00929.pdf,"GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitation","This paper presents GenH2R, a framework for learning generalizable
+vision-based human-to-robot (H2R) handover skills. The goal is to equip robots
+with the ability to reliably receive objects with unseen geometry handed over
+by humans in various complex trajectories. We acquire such generalizability by
+learning H2R handover at scale with a comprehensive solution including
+procedural simulation assets creation, automated demonstration generation, and
+effective imitation learning. We leverage large-scale 3D model repositories,
+dexterous grasp generation methods, and curve-based 3D animation to create an
+H2R handover simulation environment named \simabbns, surpassing the number of
+scenes in existing simulators by three orders of magnitude. We further
+introduce a distillation-friendly demonstration generation method that
+automatically generates a million high-quality demonstrations suitable for
+learning. Finally, we present a 4D imitation learning method augmented by a
+future forecasting objective to distill demonstrations into a visuo-motor
+handover policy. Experimental evaluations in both simulators and the real world
+demonstrate significant improvements (at least +10\% success rate) over
+baselines in all cases. The project page is https://GenH2R.github.io/.",cs.RO,"['cs.RO', 'cs.CV']"
+HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data,Mengqi Zhang · Yang Fu · Zheng Ding · Sifei Liu · Zhuowen Tu · Xiaolong Wang, ,https://arxiv.org/abs/2403.12011,,2403.12011.pdf,HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data,"3D hand-object interaction data is scarce due to the hardware constraints in
+scaling up the data collection process. In this paper, we propose HOIDiffusion
+for generating realistic and diverse 3D hand-object interaction data. Our model
+is a conditional diffusion model that takes both the 3D hand-object geometric
+structure and text description as inputs for image synthesis. This offers a
+more controllable and realistic synthesis as we can specify the structure and
+style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a
+diffusion model pre-trained on large-scale natural images and a few 3D human
+demonstrations. Beyond controllable image synthesis, we adopt the generated 3D
+data for learning 6D object pose estimation and show its effectiveness in
+improving perception systems. Project page:
+https://mq-zhang1.github.io/HOIDiffusion",cs.CV,['cs.CV']
+Learning to navigate efficiently and precisely in real environments,Guillaume Bono · Hervé Poirier · Leonid Antsfeld · Gianluca Monaci · Boris Chidlovskii · Christian Wolf, ,https://arxiv.org/abs/2401.14349,,2401.14349.pdf,Learning to navigate efficiently and precisely in real environments,"In the context of autonomous navigation of terrestrial robots, the creation
+of realistic models for agent dynamics and sensing is a widespread habit in the
+robotics literature and in commercial applications, where they are used for
+model based control and/or for localization and mapping. The more recent
+Embodied AI literature, on the other hand, focuses on modular or end-to-end
+agents trained in simulators like Habitat or AI-Thor, where the emphasis is put
+on photo-realistic rendering and scene diversity, but high-fidelity robot
+motion is assigned a less privileged role. The resulting sim2real gap
+significantly impacts transfer of the trained models to real robotic platforms.
+In this work we explore end-to-end training of agents in simulation in settings
+which minimize the sim2real gap both, in sensing and in actuation. Our agent
+directly predicts (discretized) velocity commands, which are maintained through
+closed-loop control in the real robot. The behavior of the real robot
+(including the underlying low-level controller) is identified and simulated in
+a modified Habitat simulator. Noise models for odometry and localization
+further contribute in lowering the sim2real gap. We evaluate on real navigation
+scenarios, explore different localization and point goal calculation methods
+and report significant gains in performance and robustness compared to prior
+work.",cs.RO,"['cs.RO', 'cs.CV']"
+TexOct: Generating Textures of 3D Models with Octree-based Diffusion,Jialun Liu · Chenming Wu · Xinqi Liu · Xing Liu · Jinbo Wu · Haotian Peng · Chen Zhao · Haocheng Feng · Jingtuo Liu · Errui Ding, ,https://arxiv.org/html/2403.15009v1,,2403.15009v1.pdf,TexRO: Generating Delicate Textures of 3D Models by Recursive Optimization,"This paper presents TexRO, a novel method for generating delicate textures of
+a known 3D mesh by optimizing its UV texture. The key contributions are
+two-fold. We propose an optimal viewpoint selection strategy, that finds the
+most miniature set of viewpoints covering all the faces of a mesh. Our
+viewpoint selection strategy guarantees the completeness of a generated result.
+We propose a recursive optimization pipeline that optimizes a UV texture at
+increasing resolutions, with an adaptive denoising method that re-uses existing
+textures for new texture generation. Through extensive experimentation, we
+demonstrate the superior performance of TexRO in terms of texture quality,
+detail preservation, visual consistency, and, notably runtime speed,
+outperforming other current methods. The broad applicability of TexRO is
+further confirmed through its successful use on diverse 3D models.",cs.CV,['cs.CV']
+Explaining CLIP's performance disparities on data from blind/low vision users,Daniela Massiceti · Camilla Longden · Agnieszka Słowik · Samuel Wills · Martin Grayson · Cecily Morrison, ,https://arxiv.org/abs/2311.17315,,2311.17315.pdf,Explaining CLIP's performance disparities on data from blind/low vision users,"Large multi-modal models (LMMs) hold the potential to usher in a new era of
+automated visual assistance for people who are blind or low vision (BLV). Yet,
+these models have not been systematically evaluated on data captured by BLV
+users. We address this by empirically assessing CLIP, a widely-used LMM likely
+to underpin many assistive technologies. Testing 25 CLIP variants in a
+zero-shot classification task, we find that their accuracy is 15 percentage
+points lower on average for images captured by BLV users than web-crawled
+images. This disparity stems from CLIP's sensitivities to 1) image content
+(e.g. not recognizing disability objects as well as other objects); 2) image
+quality (e.g. not being robust to lighting variation); and 3) text content
+(e.g. not recognizing objects described by tactile adjectives as well as visual
+ones). We delve deeper with a textual analysis of three common pre-training
+datasets: LAION-400M, LAION-2B and DataComp-1B, showing that disability content
+is rarely mentioned. We then provide three examples that illustrate how the
+performance disparities extend to three downstream models underpinned by CLIP:
+OWL-ViT, CLIPSeg and DALL-E2. We find that few-shot learning with as few as 5
+images can mitigate CLIP's quality-of-service disparities for BLV users in some
+scenarios, which we discuss alongside a set of other possible mitigations.",cs.CV,['cs.CV']
+Virtual Immunohistochemistry Staining for Histological Images Assisted by Weakly-supervised Learning,Jiahan Li · Jiuyang Dong · Shenjin Huang · Xi Li · Junjun Jiang · Xiaopeng Fan · Yongbing Zhang, ,,https://www.sciencedirect.com/science/article/pii/S0167779924000386,,,,,nan
+Bridging Remote Sensors with Multisensor Geospatial Foundation Models,Boran Han · Shuai Zhang · Xingjian Shi · Markus Reichstein, ,https://arxiv.org/abs/2404.01260,,2404.01260.pdf,Bridging Remote Sensors with Multisensor Geospatial Foundation Models,"In the realm of geospatial analysis, the diversity of remote sensors,
+encompassing both optical and microwave technologies, offers a wealth of
+distinct observational capabilities. Recognizing this, we present msGFM, a
+multisensor geospatial foundation model that effectively unifies data from four
+key sensor modalities. This integration spans an expansive dataset of two
+million multisensor images. msGFM is uniquely adept at handling both paired and
+unpaired sensor data. For data originating from identical geolocations, our
+model employs an innovative cross-sensor pretraining approach in masked image
+modeling, enabling the synthesis of joint representations from diverse sensors.
+msGFM, incorporating four remote sensors, upholds strong performance, forming a
+comprehensive model adaptable to various sensor types. msGFM has demonstrated
+enhanced proficiency in a range of both single-sensor and multisensor
+downstream tasks. These include scene classification, segmentation, cloud
+removal, and pan-sharpening. A key discovery of our research is that
+representations derived from natural images are not always compatible with the
+distinct characteristics of geospatial remote sensors, underscoring the
+limitations of existing representations in this field. Our work can serve as a
+guide for developing multisensor geospatial pretraining models, paving the way
+for more advanced geospatial capabilities.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Active Generalized Category Discovery,Shijie Ma · Fei Zhu · Zhun Zhong · Xu-Yao Zhang · Cheng-Lin Liu,https://github.com/mashijie1028/ActiveGCD,https://arxiv.org/abs/2403.04272v1,,2403.04272v1.pdf,Active Generalized Category Discovery,"Generalized Category Discovery (GCD) is a pragmatic and challenging
+open-world task, which endeavors to cluster unlabeled samples from both novel
+and old classes, leveraging some labeled data of old classes. Given that
+knowledge learned from old classes is not fully transferable to new classes,
+and that novel categories are fully unlabeled, GCD inherently faces intractable
+problems, including imbalanced classification performance and inconsistent
+confidence between old and new classes, especially in the low-labeling regime.
+Hence, some annotations of new classes are deemed necessary. However, labeling
+new classes is extremely costly. To address this issue, we take the spirit of
+active learning and propose a new setting called Active Generalized Category
+Discovery (AGCD). The goal is to improve the performance of GCD by actively
+selecting a limited amount of valuable samples for labeling from the oracle. To
+solve this problem, we devise an adaptive sampling strategy, which jointly
+considers novelty, informativeness and diversity to adaptively select novel
+samples with proper uncertainty. However, owing to the varied orderings of
+label indices caused by the clustering of novel classes, the queried labels are
+not directly applicable to subsequent training. To overcome this issue, we
+further propose a stable label mapping algorithm that transforms ground truth
+labels to the label space of the classifier, thereby ensuring consistent
+training across different active selection stages. Our method achieves
+state-of-the-art performance on both generic and fine-grained datasets. Our
+code is available at https://github.com/mashijie1028/ActiveGCD",cs.CV,['cs.CV']
+PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation,Zhenyu Li · Shariq Bhat · Peter Wonka, ,https://arxiv.org/abs/2312.02284,,2312.02284.pdf,PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation,"Single image depth estimation is a foundational task in computer vision and
+generative modeling. However, prevailing depth estimation models grapple with
+accommodating the increasing resolutions commonplace in today's consumer
+cameras and devices. Existing high-resolution strategies show promise, but they
+often face limitations, ranging from error propagation to the loss of
+high-frequency details. We present PatchFusion, a novel tile-based framework
+with three key components to improve the current state of the art: (1) A
+patch-wise fusion network that fuses a globally-consistent coarse prediction
+with finer, inconsistent tiled predictions via high-level feature guidance, (2)
+A Global-to-Local (G2L) module that adds vital context to the fusion network,
+discarding the need for patch selection heuristics, and (3) A Consistency-Aware
+Training (CAT) and Inference (CAI) approach, emphasizing patch overlap
+consistency and thereby eradicating the necessity for post-processing.
+Experiments on UnrealStereo4K, MVS-Synth, and Middleburry 2014 demonstrate that
+our framework can generate high-resolution depth maps with intricate details.
+PatchFusion is independent of the base model for depth estimation. Notably, our
+framework built on top of SOTA ZoeDepth brings improvements for a total of
+17.3% and 29.4% in terms of the root mean squared error (RMSE) on
+UnrealStereo4K and MVS-Synth, respectively.",cs.CV,['cs.CV']
+Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation,Xianghui Xie · Bharat Lal Bhatnagar · Jan Lenssen · Gerard Pons-Moll, ,https://arxiv.org/abs/2312.07063,,2312.07063.pdf,Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation,"Reconstructing human-object interaction in 3D from a single RGB image is a
+challenging task and existing data driven methods do not generalize beyond the
+objects present in the carefully curated 3D interaction datasets. Capturing
+large-scale real data to learn strong interaction and 3D shape priors is very
+expensive due to the combinatorial nature of human-object interactions. In this
+paper, we propose ProciGen (Procedural interaction Generation), a method to
+procedurally generate datasets with both, plausible interaction and diverse
+object variation. We generate 1M+ human-object interaction pairs in 3D and
+leverage this large-scale data to train our HDM (Hierarchical Diffusion Model),
+a novel method to reconstruct interacting human and unseen objects, without any
+templates. Our HDM is an image-conditioned diffusion model that learns both
+realistic interaction and highly accurate human and object shapes. Experiments
+show that our HDM trained with ProciGen significantly outperforms prior methods
+that requires template meshes and that our dataset allows training methods with
+strong generalization ability to unseen object instances. Our code and data are
+released.",cs.CV,['cs.CV']
+Transferable and Principled Efficiency for Open-Vocabulary Segmentation,Jingxuan Xu · Wuyang Chen · Yao Zhao · Yunchao Wei, ,https://arxiv.org/abs/2404.07448,,2404.07448.pdf,Transferable and Principled Efficiency for Open-Vocabulary Segmentation,"Recent success of pre-trained foundation vision-language models makes
+Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance,
+this approach introduces heavy computational overheads for two challenges: 1)
+large model sizes of the backbone; 2) expensive costs during the fine-tuning.
+These challenges hinder this OVS strategy from being widely applicable and
+affordable in real-world scenarios. Although traditional methods such as model
+compression and efficient fine-tuning can address these challenges, they often
+rely on heuristics. This means that their solutions cannot be easily
+transferred and necessitate re-training on different models, which comes at a
+cost. In the context of efficient OVS, we target achieving performance that is
+comparable to or even better than prior OVS works based on large
+vision-language foundation models, by utilizing smaller models that incur lower
+training costs. The core strategy is to make our efficiency principled and thus
+seamlessly transferable from one OVS framework to others without further
+customization. Comprehensive experiments on diverse OVS benchmarks demonstrate
+our superior trade-off between segmentation accuracy and computation costs over
+previous works. Our code is available on https://github.com/Xujxyang/OpenTrans",cs.CV,"['cs.CV', 'cs.CL', 'eess.IV']"
+Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning,Haoyu Chen · Wenbo Li · Jinjin Gu · Jingjing Ren · Haoze Sun · Xueyi Zou · Youliang Yan · Zhensong Zhang · Lei Zhu, ,https://arxiv.org/abs/2403.02601,,2403.02601.pdf,Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning,"For image super-resolution (SR), bridging the gap between the performance on
+synthetic datasets and real-world degradation scenarios remains a challenge.
+This work introduces a novel ""Low-Res Leads the Way"" (LWay) training framework,
+merging Supervised Pre-training with Self-supervised Learning to enhance the
+adaptability of SR models to real-world images. Our approach utilizes a
+low-resolution (LR) reconstruction network to extract degradation embeddings
+from LR images, merging them with super-resolved outputs for LR reconstruction.
+Leveraging unseen LR images for self-supervised learning guides the model to
+adapt its modeling space to the target domain, facilitating fine-tuning of SR
+models without requiring paired high-resolution (HR) images. The integration of
+Discrete Wavelet Transform (DWT) further refines the focus on high-frequency
+details. Extensive evaluations show that our method significantly improves the
+generalization and detail restoration capabilities of SR models on unseen
+real-world datasets, outperforming existing methods. Our training regime is
+universally compatible, requiring no network architecture modifications, making
+it a practical solution for real-world SR applications.",eess.IV,"['eess.IV', 'cs.CV']"
+BodyMAP - Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed,Abhishek Tandon · Anujraaj Goyal · Henry M. Clever · Zackory Erickson, ,https://arxiv.org/abs/2404.03183,,2404.03183.pdf,BodyMAP -- Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed,"Accurately predicting the 3D human posture and the pressure exerted on the
+body for people resting in bed, visualized as a body mesh (3D pose & shape)
+with a 3D pressure map, holds significant promise for healthcare applications,
+particularly, in the prevention of pressure ulcers. Current methods focus on
+singular facets of the problem -- predicting only 2D/3D poses, generating 2D
+pressure images, predicting pressure only for certain body regions instead of
+the full body, or forming indirect approximations to the 3D pressure map. In
+contrast, we introduce BodyMAP, which jointly predicts the human body mesh and
+3D applied pressure map across the entire human body. Our network leverages
+multiple visual modalities, incorporating both a depth image of a person in bed
+and its corresponding 2D pressure image acquired from a pressure-sensing
+mattress. The 3D pressure map is represented as a pressure value at each mesh
+vertex and thus allows for precise localization of high-pressure regions on the
+body. Additionally, we present BodyMAP-WS, a new formulation of pressure
+prediction in which we implicitly learn pressure in 3D by aligning sensed 2D
+pressure images with a differentiable 2D projection of the predicted 3D
+pressure maps. In evaluations with real-world human data, our method
+outperforms the current state-of-the-art technique by 25% on both body mesh and
+3D applied pressure map prediction tasks for people in bed.",cs.CV,['cs.CV']
+OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising,Haichao Zhang · Yi Xu · Hongsheng Lu · Takayuki Shimizu · Yun Fu, ,https://arxiv.org/abs/2404.02227,,2404.02227.pdf,OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising,"Trajectory prediction is fundamental in computer vision and autonomous
+driving, particularly for understanding pedestrian behavior and enabling
+proactive decision-making. Existing approaches in this field often assume
+precise and complete observational data, neglecting the challenges associated
+with out-of-view objects and the noise inherent in sensor data due to limited
+camera range, physical obstructions, and the absence of ground truth for
+denoised sensor data. Such oversights are critical safety concerns, as they can
+result in missing essential, non-visible objects. To bridge this gap, we
+present a novel method for out-of-sight trajectory prediction that leverages a
+vision-positioning technique. Our approach denoises noisy sensor observations
+in an unsupervised manner and precisely maps sensor-based trajectories of
+out-of-sight objects into visual trajectories. This method has demonstrated
+state-of-the-art performance in out-of-sight noisy sensor trajectory denoising
+and prediction on the Vi-Fi and JRDB datasets. By enhancing trajectory
+prediction accuracy and addressing the challenges of out-of-sight objects, our
+work significantly contributes to improving the safety and reliability of
+autonomous driving in complex environments. Our work represents the first
+initiative towards Out-Of-Sight Trajectory prediction (OOSTraj), setting a new
+benchmark for future research. The code is available at
+\url{https://github.com/Hai-chao-Zhang/OOSTraj}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+PAD: Patch-Agnostic Defense against Adversarial Patch Attacks,Lihua Jing · Rui Wang · Wenqi Ren · Xin Dong · Cong Zou, ,https://arxiv.org/abs/2404.16452,,2404.16452.pdf,PAD: Patch-Agnostic Defense against Adversarial Patch Attacks,"Adversarial patch attacks present a significant threat to real-world object
+detectors due to their practical feasibility. Existing defense methods, which
+rely on attack data or prior knowledge, struggle to effectively address a wide
+range of adversarial patches. In this paper, we show two inherent
+characteristics of adversarial patches, semantic independence and spatial
+heterogeneity, independent of their appearance, shape, size, quantity, and
+location. Semantic independence indicates that adversarial patches operate
+autonomously within their semantic context, while spatial heterogeneity
+manifests as distinct image quality of the patch area that differs from
+original clean image due to the independent generation process. Based on these
+observations, we propose PAD, a novel adversarial patch localization and
+removal method that does not require prior knowledge or additional training.
+PAD offers patch-agnostic defense against various adversarial patches,
+compatible with any pre-trained object detectors. Our comprehensive digital and
+physical experiments involving diverse patch types, such as localized noise,
+printable, and naturalistic patches, exhibit notable improvements over
+state-of-the-art works. Our code is available at
+https://github.com/Lihua-Jing/PAD.",cs.CV,['cs.CV']
+ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images,Jinseo Jeong · Junseo Koo · Qimeng Zhang · Gunhee Kim, ,https://arxiv.org/abs/2404.15707,,2404.15707.pdf,ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images,"Existing NeRF-based inverse rendering methods suppose that scenes are
+exclusively illuminated by distant light sources, neglecting the potential
+influence of emissive sources within a scene. In this work, we confront this
+limitation using LDR multi-view images captured with emissive sources turned on
+and off. Two key issues must be addressed: 1) ambiguity arising from the
+limited dynamic range along with unknown lighting details, and 2) the expensive
+computational cost in volume rendering to backtrace the paths leading to final
+object colors. We present a novel approach, ESR-NeRF, leveraging neural
+networks as learnable functions to represent ray-traced fields. By training
+networks to satisfy light transport segments, we regulate outgoing radiances,
+progressively identifying emissive sources while being aware of reflection
+areas. The results on scenes encompassing emissive sources with various
+properties demonstrate the superiority of ESR-NeRF in qualitative and
+quantitative ways. Our approach also extends its applicability to the scenes
+devoid of emissive sources, achieving lower CD metrics on the DTU dataset.",cs.CV,['cs.CV']
+Enhancing Video Super-Resolution via Implicit Resampling-based Alignment,Kai Xu · Ziwei Yu · Xin Wang · Michael Bi Mi · Angela Yao,https://github.com/kai422/IART,https://arxiv.org/html/2305.00163v2,,2305.00163v2.pdf,Enhancing Video Super-Resolution via Implicit Resampling-based Alignment,"In video super-resolution, it is common to use a frame-wise alignment to
+support the propagation of information over time. The role of alignment is
+well-studied for low-level enhancement in video, but existing works overlook a
+critical step -- resampling. We show through extensive experiments that for
+alignment to be effective, the resampling should preserve the reference
+frequency spectrum while minimizing spatial distortions. However, most existing
+works simply use a default choice of bilinear interpolation for resampling even
+though bilinear interpolation has a smoothing effect and hinders
+super-resolution. From these observations, we propose an implicit
+resampling-based alignment. The sampling positions are encoded by a sinusoidal
+positional encoding, while the value is estimated with a coordinate network and
+a window-based cross-attention. We show that bilinear interpolation inherently
+attenuates high-frequency information while an MLP-based coordinate network can
+approximate more frequencies. Experiments on synthetic and real-world datasets
+show that alignment with our proposed implicit resampling enhances the
+performance of state-of-the-art frameworks with minimal impact on both compute
+and parameters.",cs.CV,['cs.CV']
+UniPTS: A Unified Framework for Proficient Post-Training Sparsity,JingJing Xie · Yuxin Zhang · Mingbao Lin · ZhiHang Lin · Liujuan Cao · Rongrong Ji, ,https://arxiv.org/abs/2405.18810,,2405.18810.pdf,UniPTS: A Unified Framework for Proficient Post-Training Sparsity,"Post-training Sparsity (PTS) is a recently emerged avenue that chases
+efficient network sparsity with limited data in need. Existing PTS methods,
+however, undergo significant performance degradation compared with traditional
+methods that retrain the sparse networks via the whole dataset, especially at
+high sparsity ratios. In this paper, we attempt to reconcile this disparity by
+transposing three cardinal factors that profoundly alter the performance of
+conventional sparsity into the context of PTS. Our endeavors particularly
+comprise (1) A base-decayed sparsity objective that promotes efficient
+knowledge transferring from dense network to the sparse counterpart. (2) A
+reducing-regrowing search algorithm designed to ascertain the optimal sparsity
+distribution while circumventing overfitting to the small calibration set in
+PTS. (3) The employment of dynamic sparse training predicated on the preceding
+aspects, aimed at comprehensively optimizing the sparsity structure while
+ensuring training stability. Our proposed framework, termed UniPTS, is
+validated to be much superior to existing PTS methods across extensive
+benchmarks. As an illustration, it amplifies the performance of POT, a recently
+proposed recipe, from 3.9% to 68.6% when pruning ResNet-50 at 90% sparsity
+ratio on ImageNet. We release the code of our paper at
+https://github.com/xjjxmu/UniPTS.",cs.CV,"['cs.CV', 'cs.AI']"
+MMA-Diffusion: MultiModal Attack on Diffusion Models,Yijun Yang · Ruiyuan Gao · Xiaosen Wang · Tsung-Yi Ho · Xu Nan · Qiang Xu,https://github.com/cure-lab/MMA-Diffusion,https://arxiv.org/abs/2311.17516,,2311.17516.pdf,MMA-Diffusion: MultiModal Attack on Diffusion Models,"In recent years, Text-to-Image (T2I) models have seen remarkable
+advancements, gaining widespread adoption. However, this progress has
+inadvertently opened avenues for potential misuse, particularly in generating
+inappropriate or Not-Safe-For-Work (NSFW) content. Our work introduces
+MMA-Diffusion, a framework that presents a significant and realistic threat to
+the security of T2I models by effectively circumventing current defensive
+measures in both open-source models and commercial online services. Unlike
+previous approaches, MMA-Diffusion leverages both textual and visual modalities
+to bypass safeguards like prompt filters and post-hoc safety checkers, thus
+exposing and highlighting the vulnerabilities in existing defense mechanisms.",cs.CR,"['cs.CR', 'cs.CV']"
+Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures,Huijie Zhang · Yifu Lu · Ismail Alkhouri · Saiprasad Ravishankar · Dogyoon Song · Qing Qu, ,https://arxiv.org/abs/2312.09181,,2312.09181.pdf,Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures,"Diffusion models, emerging as powerful deep generative tools, excel in
+various applications. They operate through a two-steps process: introducing
+noise into training samples and then employing a model to convert random noise
+into new samples (e.g., images). However, their remarkable generative
+performance is hindered by slow training and sampling. This is due to the
+necessity of tracking extensive forward and reverse diffusion trajectories, and
+employing a large model with numerous parameters across multiple timesteps
+(i.e., noise levels). To tackle these challenges, we present a multi-stage
+framework inspired by our empirical findings. These observations indicate the
+advantages of employing distinct parameters tailored to each timestep while
+retaining universal parameters shared across all time steps. Our approach
+involves segmenting the time interval into multiple stages where we employ
+custom multi-decoder U-net architecture that blends time-dependent models with
+a universally shared encoder. Our framework enables the efficient distribution
+of computational resources and mitigates inter-stage interference, which
+substantially improves training efficiency. Extensive numerical experiments
+affirm the effectiveness of our framework, showcasing significant training and
+sampling efficiency enhancements on three state-of-the-art diffusion models,
+including large-scale latent diffusion models. Furthermore, our ablation
+studies illustrate the impact of two important components in our framework: (i)
+a novel timestep clustering algorithm for stage division, and (ii) an
+innovative multi-decoder U-net architecture, seamlessly integrating universal
+and customized hyperparameters.",cs.CV,['cs.CV']
+DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing,Chong Mou · Xintao Wang · Jiechong Song · Ying Shan · Jian Zhang, ,https://arxiv.org/abs/2402.02583,,2402.02583.pdf,DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing,"Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
+generation over the last few years. Although owning diverse and high-quality
+generation capabilities, translating these abilities to fine-grained image
+editing remains challenging. In this paper, we propose DiffEditor to rectify
+two weaknesses in existing diffusion-based image editing: (1) in complex
+scenarios, editing results often lack editing accuracy and exhibit unexpected
+artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
+imagine new content. In our solution, we introduce image prompts in
+fine-grained image editing, cooperating with the text prompt to better describe
+the editing content. To increase the flexibility while maintaining content
+consistency, we locally combine stochastic differential equation (SDE) into the
+ordinary differential equation (ODE) sampling. In addition, we incorporate
+regional score-based gradient guidance and a time travel strategy into the
+diffusion sampling, further improving the editing quality. Extensive
+experiments demonstrate that our method can efficiently achieve
+state-of-the-art performance on various fine-grained image editing tasks,
+including editing within a single image (e.g., object moving, resizing, and
+content dragging) and across images (e.g., appearance replacing and object
+pasting). Our source code is released at
+https://github.com/MC-E/DragonDiffusion.",cs.CV,"['cs.CV', 'cs.LG']"
+DiVAS: Video and Audio Synchronization with Dynamic Frame Rates,Clara Maria Fernandez Labrador · Mertcan Akcay · Eitan Abecassis · Joan Massich · Christopher Schroers, ,,https://link.springer.com/article/10.1007/s11042-023-17728-1,,,,,nan
+EvDiG: Event-guided Direct and Global Components Separation,xinyu zhou · Peiqi Duan · Boyu Li · Chu Zhou · Chao Xu · Boxin Shi, ,http://export.arxiv.org/abs/2312.16933v1,,2312.16933v1.pdf,EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion,"Event cameras and RGB cameras exhibit complementary characteristics in
+imaging: the former possesses high dynamic range (HDR) and high temporal
+resolution, while the latter provides rich texture and color information. This
+makes the integration of event cameras into middle- and high-level RGB-based
+vision tasks highly promising. However, challenges arise in multi-modal fusion,
+data annotation, and model architecture design. In this paper, we propose
+EvPlug, which learns a plug-and-play event and image fusion module from the
+supervision of the existing RGB-based model. The learned fusion module
+integrates event streams with image features in the form of a plug-in, endowing
+the RGB-based model to be robust to HDR and fast motion scenes while enabling
+high temporal resolution inference. Our method only requires unlabeled
+event-image pairs (no pixel-wise alignment required) and does not alter the
+structure or weights of the RGB-based model. We demonstrate the superiority of
+EvPlug in several vision tasks such as object detection, semantic segmentation,
+and 3D hand pose estimation",cs.CV,"['cs.CV', 'cs.AI']"
+Face2Diffusion for Fast and Editable Face Personalization,Kaede Shiohara · Toshihiko Yamasaki,https://mapooon.github.io/Face2DiffusionPage/,https://arxiv.org/abs/2403.05094,,2403.05094.pdf,Face2Diffusion for Fast and Editable Face Personalization,"Face personalization aims to insert specific faces, taken from images, into
+pretrained text-to-image diffusion models. However, it is still challenging for
+previous methods to preserve both the identity similarity and editability due
+to overfitting to training samples. In this paper, we propose Face2Diffusion
+(F2D) for high-editability face personalization. The core idea behind F2D is
+that removing identity-irrelevant information from the training pipeline
+prevents the overfitting problem and improves editability of encoded faces. F2D
+consists of the following three novel components: 1) Multi-scale identity
+encoder provides well-disentangled identity features while keeping the benefits
+of multi-scale information, which improves the diversity of camera poses. 2)
+Expression guidance disentangles face expressions from identities and improves
+the controllability of face expressions. 3) Class-guided denoising
+regularization encourages models to learn how faces should be denoised, which
+boosts the text-alignment of backgrounds. Extensive experiments on the
+FaceForensics++ dataset and diverse prompts demonstrate our method greatly
+improves the trade-off between the identity- and text-fidelity compared to
+previous state-of-the-art methods.",cs.CV,['cs.CV']
+PeVL: Pose-Enhanced Vision-Language Model for Fine-Grained Human Action Recognition,Haosong Zhang · Mei Leong · Liyuan Li · Weisi Lin, ,https://ar5iv.labs.arxiv.org/html/2205.11169,,2205.11169.pdf,PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models,"Vision-language pre-training (VLP) has shown impressive performance on a wide
+range of cross-modal tasks, where VLP models without reliance on object
+detectors are becoming the mainstream due to their superior computation
+efficiency and competitive performance. However, the removal of object
+detectors also deprives the capability of VLP models in explicit object
+modeling, which is essential to various position-sensitive vision-language (VL)
+tasks, such as referring expression comprehension and visual commonsense
+reasoning. To address the challenge, we introduce PEVL that enhances the
+pre-training and prompt tuning of VLP models with explicit object position
+modeling. Specifically, PEVL reformulates discretized object positions and
+language in a unified language modeling framework, which facilitates explicit
+VL alignment during pre-training, and also enables flexible prompt tuning for
+various downstream tasks. We show that PEVL enables state-of-the-art
+performance of detector-free VLP models on position-sensitive tasks such as
+referring expression comprehension and phrase grounding, and also improves the
+performance on position-insensitive tasks with grounded inputs. We make the
+data and code for this paper publicly available at
+https://github.com/thunlp/PEVL.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+HOI-M$^3$: Capture Multiple Humans and Objects Interaction within Contextual Environment,Juze Zhang · Jingyan Zhang · Zining Song · Zhanhe Shi · Chengfeng Zhao · Ye Shi · Jingyi Yu · Lan Xu · Jingya Wang, ,https://arxiv.org/abs/2404.00299,,2404.00299.pdf,HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment,"Humans naturally interact with both others and the surrounding multiple
+objects, engaging in various social activities. However, recent advances in
+modeling human-object interactions mostly focus on perceiving isolated
+individuals and objects, due to fundamental data scarcity. In this paper, we
+introduce HOI-M3, a novel large-scale dataset for modeling the interactions of
+Multiple huMans and Multiple objects. Notably, it provides accurate 3D tracking
+for both humans and objects from dense RGB and object-mounted IMU inputs,
+covering 199 sequences and 181M frames of diverse humans and objects under rich
+activities. With the unique HOI-M3 dataset, we introduce two novel data-driven
+tasks with companion strong baselines: monocular capture and unstructured
+generation of multiple human-object interactions. Extensive experiments
+demonstrate that our dataset is challenging and worthy of further research
+about multiple human-object interactions and behavior analysis. Our HOI-M3
+dataset, corresponding codes, and pre-trained models will be disseminated to
+the community for future research.",cs.CV,['cs.CV']
+GraCo: Granularity-Controllable Interactive Segmentation,Yian Zhao · Kehan Li · Zesen Cheng · Pengchong Qiao · Xiawu Zheng · Rongrong Ji · Chang Liu · Li Yuan · Jie Chen, ,https://arxiv.org/abs/2405.00587,,2405.00587.pdf,GraCo: Granularity-Controllable Interactive Segmentation,"Interactive Segmentation (IS) segments specific objects or parts in the image
+according to user input. Current IS pipelines fall into two categories:
+single-granularity output and multi-granularity output. The latter aims to
+alleviate the spatial ambiguity present in the former. However, the
+multi-granularity output pipeline suffers from limited interaction flexibility
+and produces redundant results. In this work, we introduce
+Granularity-Controllable Interactive Segmentation (GraCo), a novel approach
+that allows precise control of prediction granularity by introducing additional
+parameters to input. This enhances the customization of the interactive system
+and eliminates redundancy while resolving ambiguity. Nevertheless, the
+exorbitant cost of annotating multi-granularity masks and the lack of available
+datasets with granularity annotations make it difficult for models to acquire
+the necessary guidance to control output granularity. To address this problem,
+we design an any-granularity mask generator that exploits the semantic property
+of the pre-trained IS model to automatically generate abundant mask-granularity
+pairs without requiring additional manual annotation. Based on these pairs, we
+propose a granularity-controllable learning strategy that efficiently imparts
+the granularity controllability to the IS model. Extensive experiments on
+intricate scenarios at object and part levels demonstrate that our GraCo has
+significant advantages over previous methods. This highlights the potential of
+GraCo to be a flexible annotation tool, capable of adapting to diverse
+segmentation scenarios. The project page: https://zhao-yian.github.io/GraCo.",cs.CV,['cs.CV']
+Deep Equilibrium Diffusion Restoration with Parallel Sampling,Jiezhang Cao · Yue Shi · Kai Zhang · Yulun Zhang · Radu Timofte · Luc Van Gool, ,https://arxiv.org/abs/2311.11600,,2311.11600.pdf,Deep Equilibrium Diffusion Restoration with Parallel Sampling,"Diffusion model-based image restoration (IR) aims to use diffusion models to
+recover high-quality (HQ) images from degraded images, achieving promising
+performance. Due to the inherent property of diffusion models, most existing
+methods need long serial sampling chains to restore HQ images step-by-step,
+resulting in expensive sampling time and high computation costs. Moreover, such
+long sampling chains hinder understanding the relationship between inputs and
+restoration results since it is hard to compute the gradients in the whole
+chains. In this work, we aim to rethink the diffusion model-based IR models
+through a different perspective, i.e., a deep equilibrium (DEQ) fixed point
+system, called DeqIR. Specifically, we derive an analytical solution by
+modeling the entire sampling chain in these IR models as a joint multivariate
+fixed point system. Based on the analytical solution, we can conduct parallel
+sampling and restore HQ images without training. Furthermore, we compute fast
+gradients via DEQ inversion and found that initialization optimization can
+boost image quality and control the generation direction. Extensive experiments
+on benchmarks demonstrate the effectiveness of our method on typical IR tasks
+and real-world settings.",cs.CV,['cs.CV']
+Puff-Net: Efficient Style Transfer with Pure Content and Style Feature Fusion Network,Sizhe Zheng · Pan Gao · Peng Zhou · Jie Qin, ,https://arxiv.org/abs/2405.19775,,2405.19775.pdf,Puff-Net: Efficient Style Transfer with Pure Content and Style Feature Fusion Network,"Style transfer aims to render an image with the artistic features of a style
+image, while maintaining the original structure. Various methods have been put
+forward for this task, but some challenges still exist. For instance, it is
+difficult for CNN-based methods to handle global information and long-range
+dependencies between input images, for which transformer-based methods have
+been proposed. Although transformers can better model the relationship between
+content and style images, they require high-cost hardware and time-consuming
+inference. To address these issues, we design a novel transformer model that
+includes only the encoder, thus significantly reducing the computational cost.
+In addition, we also find that existing style transfer methods may lead to
+images under-stylied or missing content. In order to achieve better
+stylization, we design a content feature extractor and a style feature
+extractor, based on which pure content and style images can be fed to the
+transformer. Finally, we propose a novel network termed Puff-Net, i.e., pure
+content and style feature fusion network. Through qualitative and quantitative
+experiments, we demonstrate the advantages of our model compared to
+state-of-the-art ones in the literature.",cs.CV,['cs.CV']
+Tactile-Augmented Radiance Fields,Yiming Dou · Fengyu Yang · Yi Liu · Antonio Loquercio · Andrew Owens, ,https://arxiv.org/abs/2405.04534,,2405.04534.pdf,Tactile-Augmented Radiance Fields,"We present a scene representation, which we call a tactile-augmented radiance
+field (TaRF), that brings vision and touch into a shared 3D space. This
+representation can be used to estimate the visual and tactile signals for a
+given 3D position within a scene. We capture a scene's TaRF from a collection
+of photos and sparsely sampled touch probes. Our approach makes use of two
+insights: (i) common vision-based touch sensors are built on ordinary cameras
+and thus can be registered to images using methods from multi-view geometry,
+and (ii) visually and structurally similar regions of a scene share the same
+tactile features. We use these insights to register touch signals to a captured
+visual scene, and to train a conditional diffusion model that, provided with an
+RGB-D image rendered from a neural radiance field, generates its corresponding
+tactile signal. To evaluate our approach, we collect a dataset of TaRFs. This
+dataset contains more touch samples than previous real-world datasets, and it
+provides spatially aligned visual signals for each captured touch signal. We
+demonstrate the accuracy of our cross-modal generative model and the utility of
+the captured visual-tactile data on several downstream tasks. Project page:
+https://dou-yiming.github.io/TaRF",cs.CV,['cs.CV']
+The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes,Myeongseob Ko · Feiyang Kang · Weiyan Shi · Ming Jin · Zhou Yu · Ruoxi Jia, ,https://arxiv.org/abs/2402.08922,,2402.08922.pdf,The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes,"Large-scale black-box models have become ubiquitous across numerous
+applications. Understanding the influence of individual training data sources
+on predictions made by these models is crucial for improving their
+trustworthiness. Current influence estimation techniques involve computing
+gradients for every training point or repeated training on different subsets.
+These approaches face obvious computational challenges when scaled up to large
+datasets and models.
+  In this paper, we introduce and explore the Mirrored Influence Hypothesis,
+highlighting a reciprocal nature of influence between training and test data.
+Specifically, it suggests that evaluating the influence of training data on
+test predictions can be reformulated as an equivalent, yet inverse problem:
+assessing how the predictions for training samples would be altered if the
+model were trained on specific test samples. Through both empirical and
+theoretical validations, we demonstrate the wide applicability of our
+hypothesis. Inspired by this, we introduce a new method for estimating the
+influence of training data, which requires calculating gradients for specific
+test samples, paired with a forward pass for each training point. This approach
+can capitalize on the common asymmetry in scenarios where the number of test
+samples under concurrent examination is much smaller than the scale of the
+training dataset, thus gaining a significant improvement in efficiency compared
+to existing approaches.
+  We demonstrate the applicability of our method across a range of scenarios,
+including data attribution in diffusion models, data leakage detection,
+analysis of memorization, mislabeled data detection, and tracing behavior in
+language models. Our code will be made available at
+https://github.com/ruoxi-jia-group/Forward-INF.",cs.LG,"['cs.LG', 'stat.ML']"
+Logit Standardization in Knowledge Distillation,Shangquan Sun · Wenqi Ren · Jingzhi Li · Rui Wang · Xiaochun Cao,https://sunsean21.github.io/logit-stand-KD.html,https://arxiv.org/abs/2403.01427,,2403.01427.pdf,Logit Standardization in Knowledge Distillation,"Knowledge distillation involves transferring soft labels from a teacher to a
+student using a shared temperature-based softmax function. However, the
+assumption of a shared temperature between teacher and student implies a
+mandatory exact match between their logits in terms of logit range and
+variance. This side-effect limits the performance of student, considering the
+capacity discrepancy between them and the finding that the innate logit
+relations of teacher are sufficient for student to learn. To address this
+issue, we propose setting the temperature as the weighted standard deviation of
+logit and performing a plug-and-play Z-score pre-process of logit
+standardization before applying softmax and Kullback-Leibler divergence. Our
+pre-process enables student to focus on essential logit relations from teacher
+rather than requiring a magnitude match, and can improve the performance of
+existing logit-based distillation methods. We also show a typical case where
+the conventional setting of sharing temperature between teacher and student
+cannot reliably yield the authentic distillation evaluation; nonetheless, this
+challenge is successfully alleviated by our Z-score. We extensively evaluate
+our method for various student and teacher models on CIFAR-100 and ImageNet,
+showing its significant superiority. The vanilla knowledge distillation powered
+by our pre-process can achieve favorable performance against state-of-the-art
+methods, and other distillation variants can obtain considerable gain with the
+assistance of our pre-process.",cs.CV,['cs.CV']
+Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring,Xiaoqian Lv · Shengping Zhang · Chenyang Wang · Yichen Zheng · Bineng Zhong · Chongyi Li · Liqiang Nie, ,,https://www.sciencedirect.com/science/article/abs/pii/S0957417424005888,,,,,nan
+Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance,Dazhong Shen · Guanglu Song · Zeyue Xue · Fu-Yun Wang · Yu Liu, ,https://arxiv.org/abs/2404.05384,,2404.05384.pdf,Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance,"Classifier-Free Guidance (CFG) has been widely used in text-to-image
+diffusion models, where the CFG scale is introduced to control the strength of
+text guidance on the whole image space. However, we argue that a global CFG
+scale results in spatial inconsistency on varying semantic strengths and
+suboptimal image quality. To address this problem, we present a novel approach,
+Semantic-aware Classifier-Free Guidance (S-CFG), to customize the guidance
+degrees for different semantic units in text-to-image diffusion models.
+Specifically, we first design a training-free semantic segmentation method to
+partition the latent image into relatively independent semantic regions at each
+denoising step. In particular, the cross-attention map in the denoising U-net
+backbone is renormalized for assigning each patch to the corresponding token,
+while the self-attention map is used to complete the semantic regions. Then, to
+balance the amplification of diverse semantic units, we adaptively adjust the
+CFG scales across different semantic regions to rescale the text guidance
+degrees into a uniform level. Finally, extensive experiments demonstrate the
+superiority of S-CFG over the original CFG strategy on various text-to-image
+diffusion models, without requiring any extra training cost. our codes are
+available at https://github.com/SmilesDZgk/S-CFG.",cs.CV,"['cs.CV', 'cs.AI']"
+Scalable 3D Registration via Truncated Entry-wise Absolute Residuals,Tianyu Huang · Liangzu Peng · Rene Vidal · Yun-Hui Liu, ,https://arxiv.org/abs/2404.00915,,2404.00915.pdf,Scalable 3D Registration via Truncated Entry-wise Absolute Residuals,"Given an input set of $3$D point pairs, the goal of outlier-robust $3$D
+registration is to compute some rotation and translation that align as many
+point pairs as possible. This is an important problem in computer vision, for
+which many highly accurate approaches have been recently proposed. Despite
+their impressive performance, these approaches lack scalability, often
+overflowing the $16$GB of memory of a standard laptop to handle roughly
+$30,000$ point pairs. In this paper, we propose a $3$D registration approach
+that can process more than ten million ($10^7$) point pairs with over $99\%$
+random outliers. Moreover, our method is efficient, entails low memory costs,
+and maintains high accuracy at the same time. We call our method TEAR, as it
+involves minimizing an outlier-robust loss that computes Truncated Entry-wise
+Absolute Residuals. To minimize this loss, we decompose the original
+$6$-dimensional problem into two subproblems of dimensions $3$ and $2$,
+respectively, solved in succession to global optimality via a customized
+branch-and-bound method. While branch-and-bound is often slow and unscalable,
+this does not apply to TEAR as we propose novel bounding functions that are
+tight and computationally efficient. Experiments on various datasets are
+conducted to validate the scalability and efficiency of our method.",cs.CV,"['cs.CV', 'cs.RO']"
+Coupled Laplacian Eigenmaps for Locally-Aware 3D Rigid Point Cloud Matching,Matteo Bastico · Etienne Decencière · Laurent Corté · Yannick TILLIER · David Ryckelynck,https://github.com/matteo-bastico/CoupLap,,https://paperswithcode.com/paper/coupled-laplacian-eigenmaps-for-locally-aware,,,,,nan
+MatFuse: Controllable Material Generation with Diffusion Models,Giuseppe Vecchio · Renato Sortino · Simone Palazzo · Concetto Spampinato,https://gvecchio.com/matfuse/,https://arxiv.org/abs/2308.11408,,2308.11408.pdf,MatFuse: Controllable Material Generation with Diffusion Models,"Creating high-quality materials in computer graphics is a challenging and
+time-consuming task, which requires great expertise. To simplify this process,
+we introduce MatFuse, a unified approach that harnesses the generative power of
+diffusion models for creation and editing of 3D materials. Our method
+integrates multiple sources of conditioning, including color palettes,
+sketches, text, and pictures, enhancing creative possibilities and granting
+fine-grained control over material synthesis. Additionally, MatFuse enables
+map-level material editing capabilities through latent manipulation by means of
+a multi-encoder compression model which learns a disentangled latent
+representation for each map. We demonstrate the effectiveness of MatFuse under
+multiple conditioning settings and explore the potential of material editing.
+Finally, we assess the quality of the generated materials both quantitatively
+in terms of CLIP-IQA and FID scores and qualitatively by conducting a user
+study. Source code for training MatFuse and supplemental materials are publicly
+available at https://gvecchio.com/matfuse.",cs.CV,"['cs.CV', 'cs.GR']"
+Continuous Optical Zooming: A Benchmark for Arbitrary-Scale Image Super-Resolution in Real World,Huiyuan Fu · Fei Peng · Xianwei Li · Yejun Li · Xin Wang · Huadong Ma, ,,https://github.com/Weepingchestnut/Arbitrary-Scale-SR,,,,,nan
+DETRs Beat YOLOs on Real-time Object Detection,Yian Zhao · Wenyu Lv · Shangliang Xu · Jinman Wei · Guanzhong Wang · Qingqing Dang · Yi Liu · Jie Chen, ,https://arxiv.org/html/2304.08069v3,,2304.08069v3.pdf,DETRs Beat YOLOs on Real-time Object Detection,"The YOLO series has become the most popular framework for real-time object
+detection due to its reasonable trade-off between speed and accuracy. However,
+we observe that the speed and accuracy of YOLOs are negatively affected by the
+NMS. Recently, end-to-end Transformer-based detectors (DETRs) have provided an
+alternative to eliminating NMS. Nevertheless, the high computational cost
+limits their practicality and hinders them from fully exploiting the advantage
+of excluding NMS. In this paper, we propose the Real-Time DEtection TRansformer
+(RT-DETR), the first real-time end-to-end object detector to our best knowledge
+that addresses the above dilemma. We build RT-DETR in two steps, drawing on the
+advanced DETR: first we focus on maintaining accuracy while improving speed,
+followed by maintaining speed while improving accuracy. Specifically, we design
+an efficient hybrid encoder to expeditiously process multi-scale features by
+decoupling intra-scale interaction and cross-scale fusion to improve speed.
+Then, we propose the uncertainty-minimal query selection to provide
+high-quality initial queries to the decoder, thereby improving accuracy. In
+addition, RT-DETR supports flexible speed tuning by adjusting the number of
+decoder layers to adapt to various scenarios without retraining. Our
+RT-DETR-R50 / R101 achieves 53.1% / 54.3% AP on COCO and 108 / 74 FPS on T4
+GPU, outperforming previously advanced YOLOs in both speed and accuracy. We
+also develop scaled RT-DETRs that outperform the lighter YOLO detectors (S and
+M models). Furthermore, RT-DETR-R50 outperforms DINO-R50 by 2.2% AP in accuracy
+and about 21 times in FPS. After pre-training with Objects365, RT-DETR-R50 /
+R101 achieves 55.3% / 56.2% AP. The project page:
+https://zhao-yian.github.io/RTDETR.",cs.CV,['cs.CV']
+Weakly-Supervised Audio-Visual Video Parsing with Prototype-based Pseudo-Labeling,Kranthi Kumar Rachavarapu · Kalyan Ramakrishnan · A. N. Rajagopalan, ,https://arxiv.org/abs/2405.10690,,2405.10690.pdf,CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing,"Weakly supervised audio-visual video parsing (AVVP) methods aim to detect
+audible-only, visible-only, and audible-visible events using only video-level
+labels. Existing approaches tackle this by leveraging unimodal and cross-modal
+contexts. However, we argue that while cross-modal learning is beneficial for
+detecting audible-visible events, in the weakly supervised scenario, it
+negatively impacts unaligned audible or visible events by introducing
+irrelevant modality information. In this paper, we propose CoLeaF, a novel
+learning framework that optimizes the integration of cross-modal context in the
+embedding space such that the network explicitly learns to combine cross-modal
+information for audible-visible events while filtering them out for unaligned
+events. Additionally, as videos often involve complex class relationships,
+modelling them improves performance. However, this introduces extra
+computational costs into the network. Our framework is designed to leverage
+cross-class relationships during training without incurring additional
+computations at inference. Furthermore, we propose new metrics to better
+evaluate a method's capabilities in performing AVVP. Our extensive experiments
+demonstrate that CoLeaF significantly improves the state-of-the-art results by
+an average of 1.9% and 2.4% F-score on the LLP and UnAV-100 datasets,
+respectively.",cs.CV,['cs.CV']
+Estimating Noisy Class Posterior with Part-level Labels for Noisy Label Learning,Rui Zhao · Bin Shi · Jianfei Ruan · Tianze Pan · Bo Dong,https://github.com/RyanZhaoIc/PLM.git,https://arxiv.org/abs/2405.05714,,2405.05714.pdf,Estimating Noisy Class Posterior with Part-level Labels for Noisy Label Learning,"In noisy label learning, estimating noisy class posteriors plays a
+fundamental role for developing consistent classifiers, as it forms the basis
+for estimating clean class posteriors and the transition matrix. Existing
+methods typically learn noisy class posteriors by training a classification
+model with noisy labels. However, when labels are incorrect, these models may
+be misled to overemphasize the feature parts that do not reflect the instance
+characteristics, resulting in significant errors in estimating noisy class
+posteriors. To address this issue, this paper proposes to augment the
+supervised information with part-level labels, encouraging the model to focus
+on and integrate richer information from various parts. Specifically, our
+method first partitions features into distinct parts by cropping instances,
+yielding part-level labels associated with these various parts. Subsequently,
+we introduce a novel single-to-multiple transition matrix to model the
+relationship between the noisy and part-level labels, which incorporates
+part-level labels into a classifier-consistent framework. Utilizing this
+framework with part-level labels, we can learn the noisy class posteriors more
+precisely by guiding the model to integrate information from various parts,
+ultimately improving the classification performance. Our method is
+theoretically sound, while experiments show that it is empirically effective in
+synthetic and real-world noisy benchmarks.",cs.CV,"['cs.CV', 'cs.LG']"
+CA-Jaccard: Camera-aware Jaccard Distance for Person Re-identification,Yiyu Chen · Zheyi Fan · Zhaoru Chen · Yixuan Zhu, ,https://arxiv.org/abs/2311.10605,,2311.10605.pdf,CA-Jaccard: Camera-aware Jaccard Distance for Person Re-identification,"Person re-identification (re-ID) is a challenging task that aims to learn
+discriminative features for person retrieval. In person re-ID, Jaccard distance
+is a widely used distance metric, especially in re-ranking and clustering
+scenarios. However, we discover that camera variation has a significant
+negative impact on the reliability of Jaccard distance. In particular, Jaccard
+distance calculates the distance based on the overlap of relevant neighbors.
+Due to camera variation, intra-camera samples dominate the relevant neighbors,
+which reduces the reliability of the neighbors by introducing intra-camera
+negative samples and excluding inter-camera positive samples. To overcome this
+problem, we propose a novel camera-aware Jaccard (CA-Jaccard) distance that
+leverages camera information to enhance the reliability of Jaccard distance.
+Specifically, we design camera-aware k-reciprocal nearest neighbors (CKRNNs) to
+find k-reciprocal nearest neighbors on the intra-camera and inter-camera
+ranking lists, which improves the reliability of relevant neighbors and
+guarantees the contribution of inter-camera samples in the overlap. Moreover,
+we propose a camera-aware local query expansion (CLQE) to mine reliable samples
+in relevant neighbors by exploiting camera variation as a strong constraint and
+assign these samples higher weights in overlap, further improving the
+reliability. Our CA-Jaccard distance is simple yet effective and can serve as a
+general distance metric for person re-ID methods with high reliability and low
+computational cost. Extensive experiments demonstrate the effectiveness of our
+method.",cs.CV,['cs.CV']
+SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation,Aysim Toker · Marvin Eisenberger · Daniel Cremers · Laura Leal-Taixe, ,https://arxiv.org/abs/2403.16605,,2403.16605.pdf,SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation,"In recent years, semantic segmentation has become a pivotal tool in
+processing and interpreting satellite imagery. Yet, a prevalent limitation of
+supervised learning techniques remains the need for extensive manual
+annotations by experts. In this work, we explore the potential of generative
+image diffusion to address the scarcity of annotated data in earth observation
+tasks. The main idea is to learn the joint data manifold of images and labels,
+leveraging recent advancements in denoising diffusion probabilistic models. To
+the best of our knowledge, we are the first to generate both images and
+corresponding masks for satellite segmentation. We find that the obtained pairs
+not only display high quality in fine-scale features but also ensure a wide
+sampling diversity. Both aspects are crucial for earth observation data, where
+semantic classes can vary severely in scale and occurrence frequency. We employ
+the novel data instances for downstream segmentation, as a form of data
+augmentation. In our experiments, we provide comparisons to prior works based
+on discriminative diffusion models or GANs. We demonstrate that integrating
+generated samples yields significant quantitative improvements for satellite
+semantic segmentation -- both compared to baselines and when training only on
+the original data.",cs.CV,['cs.CV']
+EASE-DETR: Easing the Competition among Object Queries,Yulu Gao · Yifan Sun · Xudong Ding · Chuyang Zhao · Si Liu, ,https://arxiv.org/abs/2310.08854,,2310.08854.pdf,Rank-DETR for High Quality Object Detection,"Modern detection transformers (DETRs) use a set of object queries to predict
+a list of bounding boxes, sort them by their classification confidence scores,
+and select the top-ranked predictions as the final detection results for the
+given input image. A highly performant object detector requires accurate
+ranking for the bounding box predictions. For DETR-based detectors, the
+top-ranked bounding boxes suffer from less accurate localization quality due to
+the misalignment between classification scores and localization accuracy, thus
+impeding the construction of high-quality detectors. In this work, we introduce
+a simple and highly performant DETR-based object detector by proposing a series
+of rank-oriented designs, combinedly called Rank-DETR. Our key contributions
+include: (i) a rank-oriented architecture design that can prompt positive
+predictions and suppress the negative ones to ensure lower false positive
+rates, as well as (ii) a rank-oriented loss function and matching cost design
+that prioritizes predictions of more accurate localization accuracy during
+ranking to boost the AP under high IoU thresholds. We apply our method to
+improve the recent SOTA methods (e.g., H-DETR and DINO-DETR) and report strong
+COCO object detection results when using different backbones such as
+ResNet-$50$, Swin-T, and Swin-L, demonstrating the effectiveness of our
+approach. Code is available at \url{https://github.com/LeapLabTHU/Rank-DETR}.",cs.CV,"['cs.CV', 'cs.LG']"
+Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation,Razvan Pasca · Alexey Gavryushin · Muhammad Hamza · Yen-Ling Kuo · Kaichun Mo · Luc Van Gool · Otmar Hilliges · Xi Wang, ,,https://dblp.org/rec/journals/corr/abs-2301-09209,,,,,nan
+LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs,Yunsheng Ma · Can Cui · Xu Cao · Wenqian Ye · Peiran Liu · Juanwu Lu · Amr Abdelraouf · Rohit Gupta · Kyungtae Han · Aniket Bera · James Rehg · Ziran Wang, ,https://arxiv.org/abs/2312.04372v2,,2312.04372v2.pdf,LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs,"Autonomous driving (AD) has made significant strides in recent years.
+However, existing frameworks struggle to interpret and execute spontaneous user
+instructions, such as ""overtake the car ahead."" Large Language Models (LLMs)
+have demonstrated impressive reasoning capabilities showing potential to bridge
+this gap. In this paper, we present LaMPilot, a novel framework that integrates
+LLMs into AD systems, enabling them to follow user instructions by generating
+code that leverages established functional primitives. We also introduce
+LaMPilot-Bench, the first benchmark dataset specifically designed to
+quantitatively evaluate the efficacy of language model programs in AD. Adopting
+the LaMPilot framework, we conduct extensive experiments to assess the
+performance of off-the-shelf LLMs on LaMPilot-Bench. Our results demonstrate
+the potential of LLMs in handling diverse driving scenarios and following user
+instructions in driving. To facilitate further research in this area, we
+release our code and data at https://github.com/PurdueDigitalTwin/LaMPilot.",cs.CL,"['cs.CL', 'cs.AI']"
+C3: High-performance and low-complexity neural compression from a single image or video,Hyunjik Kim · Matthias Bauer · Lucas Theis · Jonathan Richard Schwarz · Emilien Dupont, ,https://arxiv.org/abs/2312.02753,,2312.02753.pdf,C3: High-performance and low-complexity neural compression from a single image or video,"Most neural compression models are trained on large datasets of images or
+videos in order to generalize to unseen data. Such generalization typically
+requires large and expressive architectures with a high decoding complexity.
+Here we introduce C3, a neural compression method with strong rate-distortion
+(RD) performance that instead overfits a small model to each image or video
+separately. The resulting decoding complexity of C3 can be an order of
+magnitude lower than neural baselines with similar RD performance. C3 builds on
+COOL-CHIC (Ladune et al.) and makes several simple and effective improvements
+for images. We further develop new methodology to apply C3 to videos. On the
+CLIC2020 image benchmark, we match the RD performance of VTM, the reference
+implementation of the H.266 codec, with less than 3k MACs/pixel for decoding.
+On the UVG video benchmark, we match the RD performance of the Video
+Compression Transformer (Mentzer et al.), a well-established neural video
+codec, with less than 5k MACs/pixel for decoding.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG', 'stat.ML']"
+Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture,Juanwu Lu · Can Cui · Yunsheng Ma · Aniket Bera · Ziran Wang, ,https://arxiv.org/abs/2404.03789,,2404.03789.pdf,Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture,"Safety and robustness are crucial factors in developing trustworthy
+autonomous vehicles. One essential aspect of addressing these factors is to
+equip vehicles with the capability to predict future trajectories for all
+moving objects in the surroundings and quantify prediction uncertainties. In
+this paper, we propose the Sequential Neural Variational Agent (SeNeVA), a
+generative model that describes the distribution of future trajectories for a
+single moving object. Our approach can distinguish Out-of-Distribution data
+while quantifying uncertainty and achieving competitive performance compared to
+state-of-the-art methods on the Argoverse 2 and INTERACTION datasets.
+Specifically, a 0.446 meters minimum Final Displacement Error, a 0.203 meters
+minimum Average Displacement Error, and a 5.35% Miss Rate are achieved on the
+INTERACTION test set. Extensive qualitative and quantitative analysis is also
+provided to evaluate the proposed model. Our open-source code is available at
+https://github.com/PurdueDigitalTwin/seneva.",cs.CV,"['cs.CV', 'cs.AI']"
+Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers,Sanghyeok Lee · Joonmyung Choi · Hyunwoo J. Kim,https://github.com/mlvlab/MCTF,https://arxiv.org/abs/2403.10030,,2403.10030.pdf,Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers,"Vision Transformer (ViT) has emerged as a prominent backbone for computer
+vision. For more efficient ViTs, recent works lessen the quadratic cost of the
+self-attention layer by pruning or fusing the redundant tokens. However, these
+works faced the speed-accuracy trade-off caused by the loss of information.
+Here, we argue that token fusion needs to consider diverse relations between
+tokens to minimize information loss. In this paper, we propose a Multi-criteria
+Token Fusion (MCTF), that gradually fuses the tokens based on multi-criteria
+(e.g., similarity, informativeness, and size of fused tokens). Further, we
+utilize the one-step-ahead attention, which is the improved approach to capture
+the informativeness of the tokens. By training the model equipped with MCTF
+using a token reduction consistency, we achieve the best speed-accuracy
+trade-off in the image classification (ImageNet1K). Experimental results prove
+that MCTF consistently surpasses the previous reduction methods with and
+without training. Specifically, DeiT-T and DeiT-S with MCTF reduce FLOPs by
+about 44% while improving the performance (+0.5%, and +0.3%) over the base
+model, respectively. We also demonstrate the applicability of MCTF in various
+Vision Transformers (e.g., T2T-ViT, LV-ViT), achieving at least 31% speedup
+without performance degradation. Code is available at
+https://github.com/mlvlab/MCTF.",cs.CV,['cs.CV']
+Fooling Polarization-based Vision using Locally Controllable Polarizing Projection,Zhuoxiao Li · Zhihang Zhong · Shohei Nobuhara · Ko Nishino · Yinqiang Zheng, ,,https://paperswithcode.com/search?q=author:Ko+Nishino,,,,,nan
+MuGE: Multiple Granularity Edge Detection,Caixia Zhou · Yaping Huang · Mengyang Pu · Qingji Guan · Ruoxi Deng · Haibin Ling, ,,https://www.semanticscholar.org/paper/Practical-Edge-Detection-via-Robust-Collaborative-Fu-Guo/1b7f58d62ac5bcb292da96863482ade8348c9534,,,,,nan
+Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos,Leonhard Sommer · Artur Jesslen · Eddy Ilg · Adam Kortylewski, ,https://arxiv.org/abs/2404.05626,,2404.05626.pdf,Learning a Category-level Object Pose Estimator without Pose Annotations,"3D object pose estimation is a challenging task. Previous works always
+require thousands of object images with annotated poses for learning the 3D
+pose correspondence, which is laborious and time-consuming for labeling. In
+this paper, we propose to learn a category-level 3D object pose estimator
+without pose annotations. Instead of using manually annotated images, we
+leverage diffusion models (e.g., Zero-1-to-3) to generate a set of images under
+controlled pose differences and propose to learn our object pose estimator with
+those images. Directly using the original diffusion model leads to images with
+noisy poses and artifacts. To tackle this issue, firstly, we exploit an image
+encoder, which is learned from a specially designed contrastive pose learning,
+to filter the unreasonable details and extract image feature maps.
+Additionally, we propose a novel learning strategy that allows the model to
+learn object poses from those generated image sets without knowing the
+alignment of their canonical poses. Experimental results show that our method
+has the capability of category-level object pose estimation from a single shot
+setting (as pose definition), while significantly outperforming other
+state-of-the-art methods on the few-shot category-level object pose estimation
+benchmarks.",cs.CV,['cs.CV']
+Long-Tailed Anomaly Detection with Learnable Class Names,Chih-Hui Ho · Kuan-Chuan Peng · Nuno Vasconcelos,http://www.svcl.ucsd.edu/projects/ltad/,https://arxiv.org/abs/2403.20236,,,Long-Tailed Anomaly Detection with Learnable Class Names,"Anomaly detection (AD) aims to identify defective images and localize their
+defects (if any). Ideally, AD models should be able to detect defects over many
+image classes; without relying on hard-coded class names that can be
+uninformative or inconsistent across datasets; learn without anomaly
+supervision; and be robust to the long-tailed distributions of real-world
+applications. To address these challenges, we formulate the problem of
+long-tailed AD by introducing several datasets with different levels of class
+imbalance and metrics for performance evaluation. We then propose a novel
+method, LTAD, to detect defects from multiple and long-tailed classes, without
+relying on dataset class names. LTAD combines AD by reconstruction and semantic
+AD modules. AD by reconstruction is implemented with a transformer-based
+reconstruction module. Semantic AD is implemented with a binary classifier,
+which relies on learned pseudo class names and a pretrained foundation model.
+These modules are learned over two phases. Phase 1 learns the pseudo-class
+names and a variational autoencoder (VAE) for feature synthesis that augments
+the training data to combat long-tails. Phase 2 then learns the parameters of
+the reconstruction and classification modules of LTAD. Extensive experiments
+using the proposed long-tailed datasets show that LTAD substantially
+outperforms the state-of-the-art methods for most forms of dataset imbalance.
+The long-tailed dataset split is available at
+https://zenodo.org/records/10854201 .",cs.CV,['cs.CV']
+DiffusionRegPose: Enhancing Multi-Person Pose Estimation using a Diffusion-Based End-to-End Regression Approach,Dayi Tan · Hansheng Chen · Wei Tian · Lu Xiong, ,https://arxiv.org/abs/2401.04921,,2401.04921.pdf,Diffusion-based Pose Refinement and Muti-hypothesis Generation for 3D Human Pose Estimaiton,"Previous probabilistic models for 3D Human Pose Estimation (3DHPE) aimed to
+enhance pose accuracy by generating multiple hypotheses. However, most of the
+hypotheses generated deviate substantially from the true pose. Compared to
+deterministic models, the excessive uncertainty in probabilistic models leads
+to weaker performance in single-hypothesis prediction. To address these two
+challenges, we propose a diffusion-based refinement framework called DRPose,
+which refines the output of deterministic models by reverse diffusion and
+achieves more suitable multi-hypothesis prediction for the current pose
+benchmark by multi-step refinement with multiple noises. To this end, we
+propose a Scalable Graph Convolution Transformer (SGCT) and a Pose Refinement
+Module (PRM) for denoising and refining. Extensive experiments on Human3.6M and
+MPI-INF-3DHP datasets demonstrate that our method achieves state-of-the-art
+performance on both single and multi-hypothesis 3DHPE. Code is available at
+https://github.com/KHB1698/DRPose.",cs.CV,['cs.CV']
+Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning,Woo-Jin Ahn · Geun-Yeong Yang · Hyunduck Choi · Myo-Taeg Lim,https://github.com/root0yang/BlindNet,https://arxiv.org/abs/2403.06122,,2403.06122.pdf,Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning,"Deep learning models for semantic segmentation often experience performance
+degradation when deployed to unseen target domains unidentified during the
+training phase. This is mainly due to variations in image texture (\ie style)
+from different data sources. To tackle this challenge, existing domain
+generalized semantic segmentation (DGSS) methods attempt to remove style
+variations from the feature. However, these approaches struggle with the
+entanglement of style and content, which may lead to the unintentional removal
+of crucial content information, causing performance degradation. This study
+addresses this limitation by proposing BlindNet, a novel DGSS approach that
+blinds the style without external modules or datasets. The main idea behind our
+proposed approach is to alleviate the effect of style in the encoder whilst
+facilitating robust segmentation in the decoder. To achieve this, BlindNet
+comprises two key components: covariance alignment and semantic consistency
+contrastive learning. Specifically, the covariance alignment trains the encoder
+to uniformly recognize various styles and preserve the content information of
+the feature, rather than removing the style-sensitive factor. Meanwhile,
+semantic consistency contrastive learning enables the decoder to construct
+discriminative class embedding space and disentangles features that are
+vulnerable to misclassification. Through extensive experiments, our approach
+outperforms existing DGSS methods, exhibiting robustness and superior
+performance for semantic segmentation on unseen target domains.",cs.CV,['cs.CV']
+Deep-TROJ: An Inference Stage Trojan Insertion Algorithm through Efficient Weight Replacement Attack,Sabbir Ahmed · RANYANG ZHOU · Shaahin Angizi · Adnan Rakin Rakin, ,,,,,,,nan
+Robust Image Denoising through Adversarial Frequency Mixup,Donghun Ryou · Inju Ha · Hyewon Yoo · Dongwan Kim · Bohyung Han, ,https://arxiv.org/abs/2306.16050,,2306.16050.pdf,Evaluating Similitude and Robustness of Deep Image Denoising Models via Adversarial Attack,"Deep neural networks (DNNs) have shown superior performance comparing to
+traditional image denoising algorithms. However, DNNs are inevitably vulnerable
+while facing adversarial attacks. In this paper, we propose an adversarial
+attack method named denoising-PGD which can successfully attack all the current
+deep denoising models while keep the noise distribution almost unchanged. We
+surprisingly find that the current mainstream non-blind denoising models
+(DnCNN, FFDNet, ECNDNet, BRDNet), blind denoising models (DnCNN-B, Noise2Noise,
+RDDCNN-B, FAN), plug-and-play (DPIR, CurvPnP) and unfolding denoising models
+(DeamNet) almost share the same adversarial sample set on both grayscale and
+color images, respectively. Shared adversarial sample set indicates that all
+these models are similar in term of local behaviors at the neighborhood of all
+the test samples. Thus, we further propose an indicator to measure the local
+similarity of models, called robustness similitude. Non-blind denoising models
+are found to have high robustness similitude across each other, while
+hybrid-driven models are also found to have high robustness similitude with
+pure data-driven non-blind denoising models. According to our robustness
+assessment, data-driven non-blind denoising models are the most robust. We use
+adversarial training to complement the vulnerability to adversarial attacks.
+Moreover, the model-driven image denoising BM3D shows resistance on adversarial
+attacks.",cs.CV,"['cs.CV', 'cs.LG', 'eess.IV']"
+Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction,Jianping Jiang · xinyu zhou · Bingxuan Wang · Xiaoming Deng · Chao Xu · Boxin Shi, ,https://arxiv.org/abs/2403.07346v1,,2403.07346v1.pdf,Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction,"Reliable hand mesh reconstruction (HMR) from commonly-used color and depth
+sensors is challenging especially under scenarios with varied illuminations and
+fast motions. Event camera is a highly promising alternative for its high
+dynamic range and dense temporal resolution properties, but it lacks key
+texture appearance for hand mesh reconstruction. In this paper, we propose
+EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event
+camera and an RGB camera compensating for each other. By fusing two modalities
+of data across time, space, and information dimensions,EvRGBHand can tackle
+overexposure and motion blur issues in RGB-based HMR and foreground scarcity
+and background overflow issues in event-based HMR. We further propose
+EvRGBDegrader, which allows our model to generalize effectively in challenging
+scenes, even when trained solely on standard scenes, thus reducing data
+acquisition costs. Experiments on real-world data demonstrate that EvRGBHand
+can effectively solve the challenging issues when using either type of camera
+alone via retaining the merits of both, and shows the potential of
+generalization to outdoor scenes and another type of event camera.",cs.CV,['cs.CV']
+Friendly Sharpness-Aware Minimization,Tao Li · Pan Zhou · Zhengbao He · Xinwen Cheng · Xiaolin Huang, ,https://arxiv.org/abs/2403.12350,,2403.12350.pdf,Friendly Sharpness-Aware Minimization,"Sharpness-Aware Minimization (SAM) has been instrumental in improving deep
+neural network training by minimizing both training loss and loss sharpness.
+Despite the practical success, the mechanisms behind SAM's generalization
+enhancements remain elusive, limiting its progress in deep learning
+optimization. In this work, we investigate SAM's core components for
+generalization improvement and introduce ""Friendly-SAM"" (F-SAM) to further
+enhance SAM's generalization. Our investigation reveals the key role of
+batch-specific stochastic gradient noise within the adversarial perturbation,
+i.e., the current minibatch gradient, which significantly influences SAM's
+generalization performance. By decomposing the adversarial perturbation in SAM
+into full gradient and stochastic gradient noise components, we discover that
+relying solely on the full gradient component degrades generalization while
+excluding it leads to improved performance. The possible reason lies in the
+full gradient component's increase in sharpness loss for the entire dataset,
+creating inconsistencies with the subsequent sharpness minimization step solely
+on the current minibatch data. Inspired by these insights, F-SAM aims to
+mitigate the negative effects of the full gradient component. It removes the
+full gradient estimated by an exponentially moving average (EMA) of historical
+stochastic gradients, and then leverages stochastic gradient noise for improved
+generalization. Moreover, we provide theoretical validation for the EMA
+approximation and prove the convergence of F-SAM on non-convex problems.
+Extensive experiments demonstrate the superior generalization performance and
+robustness of F-SAM over vanilla SAM. Code is available at
+https://github.com/nblt/F-SAM.",cs.LG,['cs.LG']
+Efficient Hyperparameter Optimization with Adaptive Fidelity Identification,Jiantong Jiang · Zeyi Wen · Atif Mansoor · Ajmal Mian, ,https://arxiv.org/html/2405.15605v2,,2405.15605v2.pdf,Fast-PGM: Fast Probabilistic Graphical Model Learning and Inference,"Probabilistic graphical models (PGMs) serve as a powerful framework for
+modeling complex systems with uncertainty and extracting valuable insights from
+data. However, users face challenges when applying PGMs to their problems in
+terms of efficiency and usability. This paper presents Fast-PGM, an efficient
+and open-source library for PGM learning and inference. Fast-PGM supports
+comprehensive tasks on PGMs, including structure and parameter learning, as
+well as exact and approximate inference, and enhances efficiency of the tasks
+through computational and memory optimizations and parallelization techniques.
+Concurrently, Fast-PGM furnishes developers with flexible building blocks,
+furnishes learners with detailed documentation, and affords non-experts
+user-friendly interfaces, thereby ameliorating the usability of PGMs to users
+across a spectrum of expertise levels. The source code of Fast-PGM is available
+at https://github.com/jjiantong/FastPGM.",cs.LG,['cs.LG']
+Exploring Pose-Aware Human-Object Interaction via Hybrid Learning,EASTMAN Z Y WU · Yali Li · Yuan Wang · Shengjin Wang, ,https://arxiv.org/abs/2403.07246,,2403.07246.pdf,Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration,"Human-object interaction (HOI) detection aims to locate human-object pairs
+and identify their interaction categories in images. Most existing methods
+primarily focus on supervised learning, which relies on extensive manual HOI
+annotations. In this paper, we propose a novel framework, termed Knowledge
+Integration to HOI (KI2HOI), that effectively integrates the knowledge of
+visual-language model to improve zero-shot HOI detection. Specifically, the
+verb feature learning module is designed based on visual semantics, by
+employing the verb extraction decoder to convert corresponding verb queries
+into interaction-specific category representations. We develop an effective
+additive self-attention mechanism to generate more comprehensive visual
+representations. Moreover, the innovative interaction representation decoder
+effectively extracts informative regions by integrating spatial and visual
+feature information through a cross-attention mechanism. To deal with zero-shot
+learning in low-data, we leverage a priori knowledge from the CLIP text encoder
+to initialize the linear classifier for enhanced interaction understanding.
+Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets
+demonstrate that our model outperforms the previous methods in various
+zero-shot and full-supervised settings.",cs.CV,['cs.CV']
+HPL-ESS: Hybrid Pseudo-Labeling for Unsupervised Event-based Semantic Segmentation,Linglin Jing · Yiming Ding · Yunpeng Gao · Zhigang Wang · Xu Yan · Dong Wang · Gerald Schaefer · Hui Fang · Bin Zhao · Xuelong Li, ,https://arxiv.org/abs/2403.16788,,2403.16788.pdf,HPL-ESS: Hybrid Pseudo-Labeling for Unsupervised Event-based Semantic Segmentation,"Event-based semantic segmentation has gained popularity due to its capability
+to deal with scenarios under high-speed motion and extreme lighting conditions,
+which cannot be addressed by conventional RGB cameras. Since it is hard to
+annotate event data, previous approaches rely on event-to-image reconstruction
+to obtain pseudo labels for training. However, this will inevitably introduce
+noise, and learning from noisy pseudo labels, especially when generated from a
+single source, may reinforce the errors. This drawback is also called
+confirmation bias in pseudo-labeling. In this paper, we propose a novel hybrid
+pseudo-labeling framework for unsupervised event-based semantic segmentation,
+HPL-ESS, to alleviate the influence of noisy pseudo labels. In particular, we
+first employ a plain unsupervised domain adaptation framework as our baseline,
+which can generate a set of pseudo labels through self-training. Then, we
+incorporate offline event-to-image reconstruction into the framework, and
+obtain another set of pseudo labels by predicting segmentation maps on the
+reconstructed images. A noisy label learning strategy is designed to mix the
+two sets of pseudo labels and enhance the quality. Moreover, we propose a soft
+prototypical alignment module to further improve the consistency of target
+domain features. Extensive experiments show that our proposed method
+outperforms existing state-of-the-art methods by a large margin on the
+DSEC-Semantic dataset (+5.88% accuracy, +10.32% mIoU), which even surpasses
+several supervised methods.",cs.CV,['cs.CV']
+Text-to-3D Generation with Bidirectional Diffusion using both 3D and 2D priors,Lihe Ding · Shaocong Dong · Zhanpeng Huang · Zibin Wang · Yiyuan Zhang · Kaixiong Gong · Dan Xu · Tianfan Xue, ,https://arxiv.org/abs/2312.04963,,2312.04963.pdf,Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors,"Most 3D generation research focuses on up-projecting 2D foundation models
+into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS)
+loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these
+methods often lead to geometric anomalies and multi-view inconsistency.
+Recently, researchers have attempted to improve the genuineness of 3D objects
+by directly training on 3D datasets, albeit at the cost of low-quality texture
+generation due to the limited texture diversity in 3D datasets. To harness the
+advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a
+unified framework that incorporates both a 3D and a 2D diffusion process, to
+preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a
+simple combination may yield inconsistent generation results, we further bridge
+them with novel bidirectional guidance. In addition, our method can be used as
+an initialization of optimization-based models to further improve the quality
+of 3D model and efficiency of optimization, reducing the generation process
+from 3.4 hours to 20 minutes. Experimental results have shown that our model
+achieves high-quality, diverse, and scalable 3D generation. Project website:
+https://bidiff.github.io/.",cs.CV,"['cs.CV', 'cs.AI']"
+Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives,Kristen Grauman · Andrew Westbury · Lorenzo Torresani · Kris Kitani · Jitendra Malik · Triantafyllos Afouras · Kumar Ashutosh · Vijay Baiyya · Siddhant Bansal · Bikram Boote · Eugene Byrne · Zachary Chavis · Joya Chen · Feng Cheng · Fu-Jen Chu · Sean Crane · Avijit Dasgupta · Jing Dong · Maria Escobar · Cristhian David Forigua Diaz · Abrham Gebreselasie · Sanjay Haresh · Jing Huang · Md Mohaiminul Islam · Suyog Jain · Rawal Khirodkar · Devansh Kukreja · Kevin Liang · Jia-Wei Liu · Sagnik Majumder · Yongsen Mao · Miguel Martin · Effrosyni Mavroudi · Tushar Nagarajan · Francesco Ragusa · Santhosh Kumar Ramakrishnan · Luigi Seminara · Arjun Somayazulu · Yale Song · Shan Su · Zihui Xue · Edward Zhang · Jinxu Zhang · Angela Castillo · Changan Chen · Fu Xinzhu · Ryosuke Furuta · Cristina González · Gupta · Jiabo Hu · Yifei Huang · Yiming Huang · Weslie Khoo · Anush Kumar · Robert Kuo · Sach Lakhavani · Miao Liu · Mi Luo · Zhengyi Luo · Brighid Meredith · Austin Miller · Oluwatumininu Oguntola · Xiaqing Pan · Penny Peng · Shraman Pramanick · Merey Ramazanova · Fiona Ryan · Wei Shan · Kiran Somasundaram · Chenan Song · Audrey Southerland · Masatoshi Tateno · Huiyu Wang · Yuchen Wang · Takuma Yagi · Mingfei Yan · Xitong Yang · Zecheng Yu · Shengxin Zha · Chen Zhao · Ziwei Zhao · Zhifan Zhu · Jeff Zhuo · Pablo ARBELAEZ · Gedas Bertasius · Dima Damen · Jakob Engel · Giovanni Maria Farinella · Antonino Furnari · Bernard Ghanem · Judy Hoffman · C.V. Jawahar · Richard Newcombe · Hyun Soo Park · James Rehg · Yoichi Sato · Manolis Savva · Jianbo Shi · Mike Zheng Shou · Michael Wray,https://ego-exo4d-data.org,https://arxiv.org/abs/2311.18259,,2311.18259.pdf,Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives,"We present Ego-Exo4D, a diverse, large-scale multimodal multiview video
+dataset and benchmark challenge. Ego-Exo4D centers around
+simultaneously-captured egocentric and exocentric video of skilled human
+activities (e.g., sports, music, dance, bike repair). 740 participants from 13
+cities worldwide performed these activities in 123 different natural scene
+contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours
+of video combined. The multimodal nature of the dataset is unprecedented: the
+video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera
+poses, IMU, and multiple paired language descriptions -- including a novel
+""expert commentary"" done by coaches and teachers and tailored to the
+skilled-activity domain. To push the frontier of first-person video
+understanding of skilled human activity, we also present a suite of benchmark
+tasks and their annotations, including fine-grained activity understanding,
+proficiency estimation, cross-view translation, and 3D hand/body pose. All
+resources are open sourced to fuel new research in the community. Project page:
+http://ego-exo4d-data.org/",cs.CV,"['cs.CV', 'cs.AI']"
+Control4D: Efficient 4D Portrait  Editing with Text,Ruizhi Shao · Jingxiang Sun · Cheng Peng · Zerong Zheng · Boyao ZHOU · Hongwen Zhang · Yebin Liu,https://control4darxiv.github.io,https://arxiv.org/abs/2405.17405,,2405.17405.pdf,Human4DiT: Free-view Human Video Generation with 4D Diffusion Transformer,"We present a novel approach for generating high-quality, spatio-temporally
+coherent human videos from a single image under arbitrary viewpoints. Our
+framework combines the strengths of U-Nets for accurate condition injection and
+diffusion transformers for capturing global correlations across viewpoints and
+time. The core is a cascaded 4D transformer architecture that factorizes
+attention across views, time, and spatial dimensions, enabling efficient
+modeling of the 4D space. Precise conditioning is achieved by injecting human
+identity, camera parameters, and temporal signals into the respective
+transformers. To train this model, we curate a multi-dimensional dataset
+spanning images, videos, multi-view data and 3D/4D scans, along with a
+multi-dimensional training strategy. Our approach overcomes the limitations of
+previous methods based on GAN or UNet-based diffusion models, which struggle
+with complex motions and viewpoint changes. Through extensive experiments, we
+demonstrate our method's ability to synthesize realistic, coherent and
+free-view human videos, paving the way for advanced multimedia applications in
+areas such as virtual reality and animation. Our project website is
+https://human4dit.github.io.",cs.CV,['cs.CV']
+Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model,Runmin Dong · Shuai Yuan · Bin Luo · Mengxuan Chen · Jinxiao Zhang · Lixian Zhang · Weijia Li · Juepeng Zheng · Haohuan Fu, ,https://arxiv.org/abs/2403.17460,,2403.17460.pdf,Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model,"Reference-based super-resolution (RefSR) has the potential to build bridges
+across spatial and temporal resolutions of remote sensing images. However,
+existing RefSR methods are limited by the faithfulness of content
+reconstruction and the effectiveness of texture transfer in large scaling
+factors. Conditional diffusion models have opened up new opportunities for
+generating realistic high-resolution images, but effectively utilizing
+reference images within these models remains an area for further exploration.
+Furthermore, content fidelity is difficult to guarantee in areas without
+relevant reference information. To solve these issues, we propose a
+change-aware diffusion model named Ref-Diff for RefSR, using the land cover
+change priors to guide the denoising process explicitly. Specifically, we
+inject the priors into the denoising model to improve the utilization of
+reference information in unchanged areas and regulate the reconstruction of
+semantically relevant content in changed areas. With this powerful guidance, we
+decouple the semantics-guided denoising and reference texture-guided denoising
+processes to improve the model performance. Extensive experiments demonstrate
+the superior effectiveness and robustness of the proposed method compared with
+state-of-the-art RefSR methods in both quantitative and qualitative
+evaluations. The code and data are available at
+https://github.com/dongrunmin/RefDiff.",eess.IV,"['eess.IV', 'cs.CV']"
+MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning,Mohamed Abdelfattah · Mariam Hassan · Alex Alahi, ,https://arxiv.org/abs/2312.04819,,2312.04819.pdf,Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning,"Real-world multi-agent tasks usually involve dynamic team composition with
+the emergence of roles, which should also be a key to efficient cooperation in
+multi-agent reinforcement learning (MARL). Drawing inspiration from the
+correlation between roles and agent's behavior patterns, we propose a novel
+framework of **A**ttention-guided **CO**ntrastive **R**ole representation
+learning for **M**ARL (**ACORM**) to promote behavior heterogeneity, knowledge
+transfer, and skillful coordination across agents. First, we introduce mutual
+information maximization to formalize role representation learning, derive a
+contrastive learning objective, and concisely approximate the distribution of
+negative pairs. Second, we leverage an attention mechanism to prompt the global
+state to attend to learned role representations in value decomposition,
+implicitly guiding agent coordination in a skillful role space to yield more
+expressive credit assignment. Experiments on challenging StarCraft II
+micromanagement and Google research football tasks demonstrate the
+state-of-the-art performance of our method and its advantages over existing
+approaches. Our code is available at
+[https://github.com/NJU-RL/ACORM](https://github.com/NJU-RL/ACORM).",cs.MA,['cs.MA']
+"Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs",Sunghwan Hong · Jaewoo Jung · Heeseong Shin · Jiaolong Yang · Chong Luo · Seungryong Kim,https://ku-cvlab.github.io/CoPoNeRF/,https://arxiv.org/abs/2312.07246,,2312.07246.pdf,"Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs","This work delves into the task of pose-free novel view synthesis from stereo
+pairs, a challenging and pioneering task in 3D vision. Our innovative
+framework, unlike any before, seamlessly integrates 2D correspondence matching,
+camera pose estimation, and NeRF rendering, fostering a synergistic enhancement
+of these tasks. We achieve this through designing an architecture that utilizes
+a shared representation, which serves as a foundation for enhanced 3D geometry
+understanding. Capitalizing on the inherent interplay between the tasks, our
+unified framework is trained end-to-end with the proposed training strategy to
+improve overall model accuracy. Through extensive evaluations across diverse
+indoor and outdoor scenes from two real-world datasets, we demonstrate that our
+approach achieves substantial improvement over previous methodologies,
+especially in scenarios characterized by extreme viewpoint changes and the
+absence of accurate camera poses.",cs.CV,['cs.CV']
+Neural Markov Random Field for Stereo Matching,Tongfan Guan · Chen Wang · Yun-Hui Liu,https://github.com/aeolusguan/NMRF,https://arxiv.org/abs/2403.11193,,2403.11193.pdf,Neural Markov Random Field for Stereo Matching,"Stereo matching is a core task for many computer vision and robotics
+applications. Despite their dominance in traditional stereo methods, the
+hand-crafted Markov Random Field (MRF) models lack sufficient modeling accuracy
+compared to end-to-end deep models. While deep learning representations have
+greatly improved the unary terms of the MRF models, the overall accuracy is
+still severely limited by the hand-crafted pairwise terms and message passing.
+To address these issues, we propose a neural MRF model, where both potential
+functions and message passing are designed using data-driven neural networks.
+Our fully data-driven model is built on the foundation of variational inference
+theory, to prevent convergence issues and retain stereo MRF's graph inductive
+bias. To make the inference tractable and scale well to high-resolution images,
+we also propose a Disparity Proposal Network (DPN) to adaptively prune the
+search space of disparity. The proposed approach ranks $1^{st}$ on both KITTI
+2012 and 2015 leaderboards among all published methods while running faster
+than 100 ms. This approach significantly outperforms prior global methods,
+e.g., lowering D1 metric by more than 50% on KITTI 2015. In addition, our
+method exhibits strong cross-domain generalization and can recover sharp edges.
+The codes at https://github.com/aeolusguan/NMRF",cs.CV,['cs.CV']
+Self-supervised debiasing using low rank regularization,Geon Yeong Park · Chanyong Jung · Sangmin Lee · Jong Chul Ye · Sang Wan Lee, ,,https://bispl.weebly.com/bispl-news/four-papers-got-accepted-for-cvpr-2024,,,,,nan
+Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates,Ka Chun SHUM · Jaeyeon Kim · Binh-Son Hua · Thanh Nguyen · Sai-Kit Yeung,https://github.com/kcshum/pose-conditioned-NeRF-object-fusion,https://arxiv.org/abs/2309.11281,,2309.11281.pdf,Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates,"Neural radiance field is an emerging rendering method that generates
+high-quality multi-view consistent images from a neural scene representation
+and volume rendering. Although neural radiance field-based techniques are
+robust for scene reconstruction, their ability to add or remove objects remains
+limited. This paper proposes a new language-driven approach for object
+manipulation with neural radiance fields through dataset updates. Specifically,
+to insert a new foreground object represented by a set of multi-view images
+into a background radiance field, we use a text-to-image diffusion model to
+learn and generate combined images that fuse the object of interest into the
+given background across views. These combined images are then used for refining
+the background radiance field so that we can render view-consistent images
+containing both the object and the background. To ensure view consistency, we
+propose a dataset updates strategy that prioritizes radiance field training
+with camera views close to the already-trained views prior to propagating the
+training to remaining views. We show that under the same dataset updates
+strategy, we can easily adapt our method for object insertion using data from
+text-to-3D models as well as object removal. Experimental results show that our
+method generates photorealistic images of the edited scenes, and outperforms
+state-of-the-art methods in 3D reconstruction and neural radiance field
+blending.",cs.CV,['cs.CV']
+Frozen Feature Augmentation for Few-Shot Image Classification,Andreas Bär · Neil Houlsby · Mostafa Dehghani · Manoj Kumar,https://frozen-feature-augmentation.github.io/,https://arxiv.org/abs/2403.10519,,2403.10519.pdf,Frozen Feature Augmentation for Few-Shot Image Classification,"Training a linear classifier or lightweight model on top of pretrained vision
+model outputs, so-called 'frozen features', leads to impressive performance on
+a number of downstream few-shot tasks. Currently, frozen features are not
+modified during training. On the other hand, when networks are trained directly
+on images, data augmentation is a standard recipe that improves performance
+with no substantial overhead. In this paper, we conduct an extensive pilot
+study on few-shot image classification that explores applying data
+augmentations in the frozen feature space, dubbed 'frozen feature augmentation
+(FroFA)', covering twenty augmentations in total. Our study demonstrates that
+adopting a deceptively simple pointwise FroFA, such as brightness, can improve
+few-shot performance consistently across three network architectures, three
+large pretraining datasets, and eight transfer datasets.",cs.CV,['cs.CV']
+VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning,Ziyang Luo · Nian Liu · Wangbo Zhao · Xuguang Yang · Dingwen Zhang · Deng-Ping Fan · Fahad Shahbaz Khan · Junwei Han, ,https://arxiv.org/abs/2311.15011,,2311.15011.pdf,VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning,"Salient object detection (SOD) and camouflaged object detection (COD) are
+related yet distinct binary mapping tasks. These tasks involve multiple
+modalities, sharing commonalities and unique cues. Existing research often
+employs intricate task-specific specialist models, potentially leading to
+redundancy and suboptimal results. We introduce VSCode, a generalist model with
+novel 2D prompt learning, to jointly address four SOD tasks and three COD
+tasks. We utilize VST as the foundation model and introduce 2D prompts within
+the encoder-decoder architecture to learn domain and task-specific knowledge on
+two separate dimensions. A prompt discrimination loss helps disentangle
+peculiarities to benefit model optimization. VSCode outperforms
+state-of-the-art methods across six tasks on 26 datasets and exhibits zero-shot
+generalization to unseen tasks by combining 2D prompts, such as RGB-D COD.
+Source code has been available at https://github.com/Sssssuperior/VSCode.",cs.CV,['cs.CV']
+GLiDR: Topologically Regularized Graph Generative Network for Sparse LiDAR Point Clouds,Prashant Kumar · Kshitij Madhav Bhat · Vedang Bhupesh Shenvi Nadkarni · Prem Kalra, ,https://arxiv.org/abs/2312.00068,,2312.00068.pdf,GLiDR: Topologically Regularized Graph Generative Network for Sparse LiDAR Point Clouds,"Sparse LiDAR point clouds cause severe loss of detail of static structures
+and reduce the density of static points available for navigation. Reduced
+density can be detrimental to navigation under several scenarios. We observe
+that despite high sparsity, in most cases, the global topology of LiDAR
+outlining the static structures can be inferred. We utilize this property to
+obtain a backbone skeleton of a LiDAR scan in the form of a single connected
+component that is a proxy to its global topology. We utilize the backbone to
+augment new points along static structures to overcome sparsity. Newly
+introduced points could correspond to existing static structures or to static
+points that were earlier obstructed by dynamic objects. To the best of our
+knowledge, we are the first to use such a strategy for sparse LiDAR point
+clouds. Existing solutions close to our approach fail to identify and preserve
+the global static LiDAR topology and generate sub-optimal points. We propose
+GLiDR, a Graph Generative network that is topologically regularized using
+0-dimensional Persistent Homology ($\mathcal{PH}$) constraints. This enables
+GLiDR to introduce newer static points along a topologically consistent global
+static LiDAR backbone. GLiDR generates precise static points using $32\times$
+sparser dynamic scans and performs better than the baselines across three
+datasets. GLiDR generates a valuable byproduct - an accurate binary
+segmentation mask of static and dynamic objects that are helpful for navigation
+planning and safety in constrained environments. The newly introduced static
+points allow GLiDR to outperform LiDAR-based navigation using SLAM in several
+settings. Source code is available at https://kshitijbhat.github.io/glidr",cs.RO,"['cs.RO', 'cs.CV']"
+The STVchrono Dataset: Towards Continuous Change Recognition in Time,Yanjun Sun · Yue Qiu · Mariia Khan · Fumiya Matsuzawa · Kenji Iwata, ,,https://www.youtube.com/watch?v=44o-Xl60ipI,,,,,nan
+NECA: Neural Customizable Human Avatar,Junjin Xiao · Qing Zhang · Zhan Xu · Wei-Shi Zheng,https://github.com/iSEE-Laboratory/NECA,https://arxiv.org/abs/2403.10335,,2403.10335.pdf,NECA: Neural Customizable Human Avatar,"Human avatar has become a novel type of 3D asset with various applications.
+Ideally, a human avatar should be fully customizable to accommodate different
+settings and environments. In this work, we introduce NECA, an approach capable
+of learning versatile human representation from monocular or sparse-view
+videos, enabling granular customization across aspects such as pose, shadow,
+shape, lighting and texture. The core of our approach is to represent humans in
+complementary dual spaces and predict disentangled neural fields of geometry,
+albedo, shadow, as well as an external lighting, from which we are able to
+derive realistic rendering with high-frequency details via volumetric
+rendering. Extensive experiments demonstrate the advantage of our method over
+the state-of-the-art methods in photorealistic rendering, as well as various
+editing tasks such as novel pose synthesis and relighting. The code is
+available at https://github.com/iSEE-Laboratory/NECA.",cs.CV,['cs.CV']
+Continual Segmentation with Disentangled Objectness Learning and Class Recognition,Yizheng Gong · Siyue Yu · Xiaoyang Wang · Jimin Xiao, ,https://arxiv.org/abs/2403.03477,,2403.03477.pdf,Continual Segmentation with Disentangled Objectness Learning and Class Recognition,"Most continual segmentation methods tackle the problem as a per-pixel
+classification task. However, such a paradigm is very challenging, and we find
+query-based segmenters with built-in objectness have inherent advantages
+compared with per-pixel ones, as objectness has strong transfer ability and
+forgetting resistance. Based on these findings, we propose CoMasTRe by
+disentangling continual segmentation into two stages: forgetting-resistant
+continual objectness learning and well-researched continual classification.
+CoMasTRe uses a two-stage segmenter learning class-agnostic mask proposals at
+the first stage and leaving recognition to the second stage. During continual
+learning, a simple but effective distillation is adopted to strengthen
+objectness. To further mitigate the forgetting of old classes, we design a
+multi-label class distillation strategy suited for segmentation. We assess the
+effectiveness of CoMasTRe on PASCAL VOC and ADE20K. Extensive experiments show
+that our method outperforms per-pixel and query-based methods on both datasets.
+Code will be available at https://github.com/jordangong/CoMasTRe.",cs.CV,['cs.CV']
+Text2Loc: 3D Point Cloud Localization from Natural Language,Yan Xia · Letian Shi · Zifeng Ding · João F. Henriques · Daniel Cremers, ,https://arxiv.org/abs/2311.15977,,2311.15977.pdf,Text2Loc: 3D Point Cloud Localization from Natural Language,"We tackle the problem of 3D point cloud localization based on a few natural
+linguistic descriptions and introduce a novel neural network, Text2Loc, that
+fully interprets the semantic relationship between points and text. Text2Loc
+follows a coarse-to-fine localization pipeline: text-submap global place
+recognition, followed by fine localization. In global place recognition,
+relational dynamics among each textual hint are captured in a hierarchical
+transformer with max-pooling (HTM), whereas a balance between positive and
+negative pairs is maintained using text-submap contrastive learning. Moreover,
+we propose a novel matching-free fine localization method to further refine the
+location predictions, which completely removes the need for complicated
+text-instance matching and is lighter, faster, and more accurate than previous
+methods. Extensive experiments show that Text2Loc improves the localization
+accuracy by up to $2\times$ over the state-of-the-art on the KITTI360Pose
+dataset. Our project page is publicly available at
+\url{https://yan-xia.github.io/projects/text2loc/}.",cs.CV,['cs.CV']
+OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation,Ganlong Zhao · Guanbin Li · Weikai Chen · Yizhou Yu, ,https://arxiv.org/abs/2403.17334,,2403.17334.pdf,OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation,"Recent advances in Iterative Vision-and-Language Navigation (IVLN) introduce
+a more meaningful and practical paradigm of VLN by maintaining the agent's
+memory across tours of scenes. Although the long-term memory aligns better with
+the persistent nature of the VLN task, it poses more challenges on how to
+utilize the highly unstructured navigation memory with extremely sparse
+supervision. Towards this end, we propose OVER-NAV, which aims to go over and
+beyond the current arts of IVLN techniques. In particular, we propose to
+incorporate LLMs and open-vocabulary detectors to distill key information and
+establish correspondence between multi-modal signals. Such a mechanism
+introduces reliable cross-modal supervision and enables on-the-fly
+generalization to unseen scenes without the need of extra annotation and
+re-training. To fully exploit the interpreted navigation data, we further
+introduce a structured representation, coded Omnigraph, to effectively
+integrate multi-modal information along the tour. Accompanied with a novel
+omnigraph fusion mechanism, OVER-NAV is able to extract the most relevant
+knowledge from omnigraph for a more accurate navigating action. In addition,
+OVER-NAV seamlessly supports both discrete and continuous environments under a
+unified framework. We demonstrate the superiority of OVER-NAV in extensive
+experiments.",cs.CV,['cs.CV']
+Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework,Ziyao Huang · Fan Tang · Yong Zhang · Xiaodong Cun · Juan Cao · Jintao Li · Tong-yee Lee,https://github.com/ICTMCG/Make-Your-Anchor,https://arxiv.org/abs/2403.16510,,2403.16510.pdf,Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework,"Despite the remarkable process of talking-head-based avatar-creating
+solutions, directly generating anchor-style videos with full-body motions
+remains challenging. In this study, we propose Make-Your-Anchor, a novel system
+necessitating only a one-minute video clip of an individual for training,
+subsequently enabling the automatic generation of anchor-style videos with
+precise torso and hand movements. Specifically, we finetune a proposed
+structure-guided diffusion model on input video to render 3D mesh conditions
+into human appearances. We adopt a two-stage training strategy for the
+diffusion model, effectively binding movements with specific appearances. To
+produce arbitrary long temporal video, we extend the 2D U-Net in the frame-wise
+diffusion model to a 3D style without additional training cost, and a simple
+yet effective batch-overlapped temporal denoising module is proposed to bypass
+the constraints on video length during inference. Finally, a novel
+identity-specific face enhancement module is introduced to improve the visual
+quality of facial regions in the output videos. Comparative experiments
+demonstrate the effectiveness and superiority of the system in terms of visual
+quality, temporal coherence, and identity preservation, outperforming SOTA
+diffusion/non-diffusion methods. Project page:
+\url{https://github.com/ICTMCG/Make-Your-Anchor}.",cs.CV,['cs.CV']
+SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking,Xiaojun Hou · Jiazheng Xing · Yijie Qian · Yaowei Guo · Shuo Xin · Junhao Chen · Kai Tang · Mengmeng Wang · Zhengkai Jiang · Liang Liu · Yong Liu,https://github.com/hoqolo/SDSTrack,https://arxiv.org/abs/2403.16002,,2403.16002.pdf,SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking,"Multimodal Visual Object Tracking (VOT) has recently gained significant
+attention due to its robustness. Early research focused on fully fine-tuning
+RGB-based trackers, which was inefficient and lacked generalized representation
+due to the scarcity of multimodal data. Therefore, recent studies have utilized
+prompt tuning to transfer pre-trained RGB-based trackers to multimodal data.
+However, the modality gap limits pre-trained knowledge recall, and the
+dominance of the RGB modality persists, preventing the full utilization of
+information from other modalities. To address these issues, we propose a novel
+symmetric multimodal tracking framework called SDSTrack. We introduce
+lightweight adaptation for efficient fine-tuning, which directly transfers the
+feature extraction ability from RGB to other domains with a small number of
+trainable parameters and integrates multimodal features in a balanced,
+symmetric manner. Furthermore, we design a complementary masked patch
+distillation strategy to enhance the robustness of trackers in complex
+environments, such as extreme weather, poor imaging, and sensor failure.
+Extensive experiments demonstrate that SDSTrack outperforms state-of-the-art
+methods in various multimodal tracking scenarios, including RGB+Depth,
+RGB+Thermal, and RGB+Event tracking, and exhibits impressive results in extreme
+conditions. Our source code is available at https://github.com/hoqolo/SDSTrack.",cs.CV,['cs.CV']
+"Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras",Huajian Huang · Longwei Li · Hui Cheng · Sai-Kit Yeung, ,https://arxiv.org/abs/2311.16728,,2311.16728.pdf,"Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras","The integration of neural rendering and the SLAM system recently showed
+promising results in joint localization and photorealistic view reconstruction.
+However, existing methods, fully relying on implicit representations, are so
+resource-hungry that they cannot run on portable devices, which deviates from
+the original intention of SLAM. In this paper, we present Photo-SLAM, a novel
+SLAM framework with a hyper primitives map. Specifically, we simultaneously
+exploit explicit geometric features for localization and learn implicit
+photometric features to represent the texture information of the observed
+environment. In addition to actively densifying hyper primitives based on
+geometric features, we further introduce a Gaussian-Pyramid-based training
+method to progressively learn multi-level features, enhancing photorealistic
+mapping performance. The extensive experiments with monocular, stereo, and
+RGB-D datasets prove that our proposed system Photo-SLAM significantly
+outperforms current state-of-the-art SLAM systems for online photorealistic
+mapping, e.g., PSNR is 30% higher and rendering speed is hundreds of times
+faster in the Replica dataset. Moreover, the Photo-SLAM can run at real-time
+speed using an embedded platform such as Jetson AGX Orin, showing the potential
+of robotics applications.",cs.CV,['cs.CV']
+Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation,Wenhao Li · Mengyuan Liu · Hong Liu · Pichao Wang · Jialun Cai · Nicu Sebe,https://github.com/NationalGAILab/HoT,,https://paperswithcode.com/paper/hourglass-tokenizer-for-efficient-transformer,,,,,nan
+Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection,Jiaming Li · Jiacheng Zhang · Jichang Li · Ge Li · Si Liu · Liang Lin · Guanbin Li, ,https://arxiv.org/abs/2404.09216,,2404.09216.pdf,DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection,"Existing open-vocabulary object detectors typically require a predefined set
+of categories from users, significantly confining their application scenarios.
+In this paper, we introduce DetCLIPv3, a high-performing detector that excels
+not only at both open-vocabulary object detection, but also generating
+hierarchical labels for detected objects. DetCLIPv3 is characterized by three
+core designs: 1. Versatile model architecture: we derive a robust open-set
+detection framework which is further empowered with generation ability via the
+integration of a caption head. 2. High information density data: we develop an
+auto-annotation pipeline leveraging visual large language model to refine
+captions for large-scale image-text pairs, providing rich, multi-granular
+object labels to enhance the training. 3. Efficient training strategy: we
+employ a pre-training stage with low-resolution inputs that enables the object
+captioner to efficiently learn a broad spectrum of visual concepts from
+extensive image-text paired data. This is followed by a fine-tuning stage that
+leverages a small number of high-resolution samples to further enhance
+detection performance. With these effective designs, DetCLIPv3 demonstrates
+superior open-vocabulary detection performance, \eg, our Swin-T backbone model
+achieves a notable 47.0 zero-shot fixed AP on the LVIS minival benchmark,
+outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP,
+respectively. DetCLIPv3 also achieves a state-of-the-art 19.7 AP in dense
+captioning task on VG dataset, showcasing its strong generative capability.",cs.CV,['cs.CV']
+3D Feature Tracking via Event Camera,Siqi Li · Zhou Zhikuan · Zhou Xue · Yipeng Li · Shaoyi Du · Yue Gao, ,https://cvpr.thecvf.com/Conferences/2023/AuthorQAEventCameras,,,,,,nan
+Multiagent Multitraversal Multimodal Self-Driving: Open MARS Dataset,Yiming Li · Zhiheng Li · Nuo Chen · Moonjun Gong · Zonglin Lyu · Zehong Wang · Peili Jiang · Chen Feng, ,https://ar5iv.labs.arxiv.org/html/2202.08449,,2202.08449.pdf,V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving,"Vehicle-to-everything (V2X) communication techniques enable the collaboration
+between vehicles and many other entities in the neighboring environment, which
+could fundamentally improve the perception system for autonomous driving.
+However, the lack of a public dataset significantly restricts the research
+progress of collaborative perception. To fill this gap, we present V2X-Sim, a
+comprehensive simulated multi-agent perception dataset for V2X-aided autonomous
+driving. V2X-Sim provides: (1) \hl{multi-agent} sensor recordings from the
+road-side unit (RSU) and multiple vehicles that enable collaborative
+perception, (2) multi-modality sensor streams that facilitate multi-modality
+perception, and (3) diverse ground truths that support various perception
+tasks. Meanwhile, we build an open-source testbed and provide a benchmark for
+the state-of-the-art collaborative perception algorithms on three tasks,
+including detection, tracking and segmentation. V2X-Sim seeks to stimulate
+collaborative perception research for autonomous driving before realistic
+datasets become widely available. Our dataset and code are available at
+\url{https://ai4ce.github.io/V2X-Sim/}.",cs.CV,['cs.CV']
+Taming Stable Diffusion for Text to 360$^{\circ}$ Panorama Image Generation,Cheng Zhang · Qianyi Wu · Camilo Cruz Gambardella · Xiaoshui Huang · Dinh Phung · Wanli Ouyang · Jianfei Cai, ,https://arxiv.org/abs/2404.07949,,2404.07949.pdf,Taming Stable Diffusion for Text to 360° Panorama Image Generation,"Generative models, e.g., Stable Diffusion, have enabled the creation of
+photorealistic images from text prompts. Yet, the generation of 360-degree
+panorama images from text remains a challenge, particularly due to the dearth
+of paired text-panorama data and the domain gap between panorama and
+perspective images. In this paper, we introduce a novel dual-branch diffusion
+model named PanFusion to generate a 360-degree image from a text prompt. We
+leverage the stable diffusion model as one branch to provide prior knowledge in
+natural image generation and register it to another panorama branch for
+holistic image generation. We propose a unique cross-attention mechanism with
+projection awareness to minimize distortion during the collaborative denoising
+process. Our experiments validate that PanFusion surpasses existing methods
+and, thanks to its dual-branch structure, can integrate additional constraints
+like room layout for customized panorama outputs. Code is available at
+https://chengzhag.github.io/publication/panfusion.",cs.CV,['cs.CV']
+Frequency-aware Event-based Video Deblurring for Real-World Motion Blur,Taewoo Kim · Hoonhee Cho · Kuk-Jin Yoon, ,https://arxiv.org/abs/2404.12168,,,Real-World Efficient Blind Motion Deblurring via Blur Pixel Discretization,"As recent advances in mobile camera technology have enabled the capability to
+capture high-resolution images, such as 4K images, the demand for an efficient
+deblurring model handling large motion has increased. In this paper, we
+discover that the image residual errors, i.e., blur-sharp pixel differences,
+can be grouped into some categories according to their motion blur type and how
+complex their neighboring pixels are. Inspired by this, we decompose the
+deblurring (regression) task into blur pixel discretization (pixel-level blur
+classification) and discrete-to-continuous conversion (regression with blur
+class map) tasks. Specifically, we generate the discretized image residual
+errors by identifying the blur pixels and then transform them to a continuous
+form, which is computationally more efficient than naively solving the original
+regression problem with continuous values. Here, we found that the
+discretization result, i.e., blur segmentation map, remarkably exhibits visual
+similarity with the image residual errors. As a result, our efficient model
+shows comparable performance to state-of-the-art methods in realistic
+benchmarks, while our method is up to 10 times computationally more efficient.",cs.CV,"['cs.CV', 'cs.AI']"
+Snapshot Lidar: Fourier embedding of amplitude and phase for single-image depth reconstruction,Sarah Friday · Yunzi Shi · Yaswanth Kumar Cherivirala · Vishwanath Saragadam · Adithya Pediredla, ,https://arxiv.org/abs/2311.10950,,2311.10950.pdf,Single-shot Phase Retrieval from a Fractional Fourier Transform Perspective,"The realm of classical phase retrieval concerns itself with the arduous task
+of recovering a signal from its Fourier magnitude measurements, which are
+fraught with inherent ambiguities. A single-exposure intensity measurement is
+commonly deemed insufficient for the reconstruction of the primal signal, given
+that the absent phase component is imperative for the inverse transformation.
+In this work, we present a novel single-shot phase retrieval paradigm from a
+fractional Fourier transform (FrFT) perspective, which involves integrating the
+FrFT-based physical measurement model within a self-supervised reconstruction
+scheme. Specifically, the proposed FrFT-based measurement model addresses the
+aliasing artifacts problem in the numerical calculation of Fresnel diffraction,
+featuring adaptability to both short-distance and long-distance propagation
+scenarios. Moreover, the intensity measurement in the FrFT domain proves highly
+effective in alleviating the ambiguities of phase retrieval and relaxing the
+previous conditions on oversampled or multiple measurements in the Fourier
+domain. Furthermore, the proposed self-supervised reconstruction approach
+harnesses the fast discrete algorithm of FrFT alongside untrained neural
+network priors, thereby attaining preeminent results. Through numerical
+simulations, we demonstrate that both amplitude and phase objects can be
+effectively retrieved from a single-shot intensity measurement using the
+proposed approach and provide a promising technique for support-free coherent
+diffraction imaging.",cs.CV,"['cs.CV', 'physics.optics']"
+ODCR: Orthogonal Decoupling Contrastive Regularization for Unpaired Image Dehazing,Zhongze Wang · Haitao Zhao · Jingchao Peng · Lujian Yao · Kaijie Zhao, ,https://arxiv.org/abs/2404.17825,,2404.17825.pdf,ODCR: Orthogonal Decoupling Contrastive Regularization for Unpaired Image Dehazing,"Unpaired image dehazing (UID) holds significant research importance due to
+the challenges in acquiring haze/clear image pairs with identical backgrounds.
+This paper proposes a novel method for UID named Orthogonal Decoupling
+Contrastive Regularization (ODCR). Our method is grounded in the assumption
+that an image consists of both haze-related features, which influence the
+degree of haze, and haze-unrelated features, such as texture and semantic
+information. ODCR aims to ensure that the haze-related features of the dehazing
+result closely resemble those of the clear image, while the haze-unrelated
+features align with the input hazy image. To accomplish the motivation,
+Orthogonal MLPs optimized geometrically on the Stiefel manifold are proposed,
+which can project image features into an orthogonal space, thereby reducing the
+relevance between different features. Furthermore, a task-driven Depth-wise
+Feature Classifier (DWFC) is proposed, which assigns weights to the orthogonal
+features based on the contribution of each channel's feature in predicting
+whether the feature source is hazy or clear in a self-supervised fashion.
+Finally, a Weighted PatchNCE (WPNCE) loss is introduced to achieve the pulling
+of haze-related features in the output image toward those of clear images,
+while bringing haze-unrelated features close to those of the hazy input.
+Extensive experiments demonstrate the superior performance of our ODCR method
+on UID.",cs.CV,['cs.CV']
+MaxQ: Multi-Axis Query for N:M Sparsity Network,Jingyang Xiang · Siqi Li · Junhao Chen · Zhuangzhi Chen · Tianxin Huang · Linpeng Peng · Yong Liu,https://github.com/JingyangXiang/MaxQ,https://arxiv.org/abs/2312.07061,,2312.07061.pdf,MaxQ: Multi-Axis Query for N:M Sparsity Network,"N:M sparsity has received increasing attention due to its remarkable
+performance and latency trade-off compared with structured and unstructured
+sparsity. However, existing N:M sparsity methods do not differentiate the
+relative importance of weights among blocks and leave important weights
+underappreciated. Besides, they directly apply N:M sparsity to the whole
+network, which will cause severe information loss. Thus, they are still
+sub-optimal. In this paper, we propose an efficient and effective Multi-Axis
+Query methodology, dubbed as MaxQ, to rectify these problems. During the
+training, MaxQ employs a dynamic approach to generate soft N:M masks,
+considering the weight importance across multiple axes. This method enhances
+the weights with more importance and ensures more effective updates. Meanwhile,
+a sparsity strategy that gradually increases the percentage of N:M weight
+blocks is applied, which allows the network to heal from the pruning-induced
+damage progressively. During the runtime, the N:M soft masks can be precomputed
+as constants and folded into weights without causing any distortion to the
+sparse pattern and incurring additional computational overhead. Comprehensive
+experiments demonstrate that MaxQ achieves consistent improvements across
+diverse CNN architectures in various computer vision tasks, including image
+classification, object detection and instance segmentation. For ResNet50 with
+1:16 sparse pattern, MaxQ can achieve 74.6\% top-1 accuracy on ImageNet and
+improve by over 2.8\% over the state-of-the-art. Codes and checkpoints are
+available at \url{https://github.com/JingyangXiang/MaxQ}.",cs.CV,['cs.CV']
+Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework,Vu Minh Hieu Phan · Yutong Xie · Yuankai Qi · Lingqiao Liu · Liyang Liu · Bowen Zhang · Zhibin Liao · Qi Wu · Minh-Son To · Johan Verjans, ,https://arxiv.org/abs/2403.07636v2,,2403.07636v2.pdf,Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework,"Medical vision language pre-training (VLP) has emerged as a frontier of
+research, enabling zero-shot pathological recognition by comparing the query
+image with the textual descriptions for each disease. Due to the complex
+semantics of biomedical texts, current methods struggle to align medical images
+with key pathological findings in unstructured reports. This leads to the
+misalignment with the target disease's textual representation. In this paper,
+we introduce a novel VLP framework designed to dissect disease descriptions
+into their fundamental aspects, leveraging prior knowledge about the visual
+manifestations of pathologies. This is achieved by consulting a large language
+model and medical experts. Integrating a Transformer module, our approach
+aligns an input image with the diverse elements of a disease, generating
+aspect-centric image representations. By consolidating the matches from each
+aspect, we improve the compatibility between an image and its associated
+disease. Additionally, capitalizing on the aspect-oriented representations, we
+present a dual-head Transformer tailored to process known and unknown diseases,
+optimizing the comprehensive detection efficacy. Conducting experiments on
+seven downstream datasets, ours improves the accuracy of recent methods by up
+to 8.56% and 17.0% for seen and unseen categories, respectively. Our code is
+released at https://github.com/HieuPhan33/MAVL.",cs.CV,['cs.CV']
+EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation,Chanyoung Kim · Woojung Han · Dayun Ju · Seong Jae Hwang,https://micv-yonsei.github.io/eagle2024/,https://arxiv.org/abs/2403.01482,,2403.01482.pdf,EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation,"Semantic segmentation has innately relied on extensive pixel-level annotated
+data, leading to the emergence of unsupervised methodologies. Among them,
+leveraging self-supervised Vision Transformers for unsupervised semantic
+segmentation (USS) has been making steady progress with expressive deep
+features. Yet, for semantically segmenting images with complex objects, a
+predominant challenge remains: the lack of explicit object-level semantic
+encoding in patch-level features. This technical limitation often leads to
+inadequate segmentation of complex objects with diverse structures. To address
+this gap, we present a novel approach, EAGLE, which emphasizes object-centric
+representation learning for unsupervised semantic segmentation. Specifically,
+we introduce EiCue, a spectral technique providing semantic and structural cues
+through an eigenbasis derived from the semantic similarity matrix of deep image
+features and color affinity from an image. Further, by incorporating our
+object-centric contrastive loss with EiCue, we guide our model to learn
+object-level representations with intra- and inter-image object-feature
+consistency, thereby enhancing semantic accuracy. Extensive experiments on
+COCO-Stuff, Cityscapes, and Potsdam-3 datasets demonstrate the state-of-the-art
+USS results of EAGLE with accurate and consistent semantic segmentation across
+complex scenes.",cs.CV,['cs.CV']
+StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On,Jeongho Kim · Gyojung Gu · Minho Park · Sunghyun Park · Jaegul Choo,https://rlawjdghek.github.io/StableVITON/,https://arxiv.org/abs/2312.01725,,2312.01725.pdf,StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On,"Given a clothing image and a person image, an image-based virtual try-on aims
+to generate a customized image that appears natural and accurately reflects the
+characteristics of the clothing image. In this work, we aim to expand the
+applicability of the pre-trained diffusion model so that it can be utilized
+independently for the virtual try-on task.The main challenge is to preserve the
+clothing details while effectively utilizing the robust generative capability
+of the pre-trained model. In order to tackle these issues, we propose
+StableVITON, learning the semantic correspondence between the clothing and the
+human body within the latent space of the pre-trained diffusion model in an
+end-to-end manner. Our proposed zero cross-attention blocks not only preserve
+the clothing details by learning the semantic correspondence but also generate
+high-fidelity images by utilizing the inherent knowledge of the pre-trained
+model in the warping process. Through our proposed novel attention total
+variation loss and applying augmentation, we achieve the sharp attention map,
+resulting in a more precise representation of clothing details. StableVITON
+outperforms the baselines in qualitative and quantitative evaluation, showing
+promising quality in arbitrary person images. Our code is available at
+https://github.com/rlawjdghek/StableVITON.",cs.CV,['cs.CV']
+Towards Robust 3D Object Detection with LiDAR and 4D Radar Fusion in Various Weather Conditions,Yujeong Chae · Hyeonseong Kim · Kuk-Jin Yoon, ,https://arxiv.org/abs/2310.00944,,2310.00944.pdf,Towards Robust 3D Object Detection In Rainy Conditions,"LiDAR sensors are used in autonomous driving applications to accurately
+perceive the environment. However, they are affected by adverse weather
+conditions such as snow, fog, and rain. These everyday phenomena introduce
+unwanted noise into the measurements, severely degrading the performance of
+LiDAR-based perception systems. In this work, we propose a framework for
+improving the robustness of LiDAR-based 3D object detectors against road spray.
+Our approach uses a state-of-the-art adverse weather detection network to
+filter out spray from the LiDAR point cloud, which is then used as input for
+the object detector. In this way, the detected objects are less affected by the
+adverse weather in the scene, resulting in a more accurate perception of the
+environment. In addition to adverse weather filtering, we explore the use of
+radar targets to further filter false positive detections. Tests on real-world
+data show that our approach improves the robustness to road spray of several
+popular 3D object detectors.",cs.CV,"['cs.CV', 'cs.LG']"
+ConCon-Chi: Concept-Context Chimera Benchmark for Personalized Vision-Language Tasks,Andrea Rosasco · Stefano Berti · Giulia Pasquale · Damiano Malafronte · Shogo Sato · Hiroyuki Segawa · Tetsugo Inada · Lorenzo Natale, ,,https://paperswithcode.com/paper/open-ended-vqa-benchmarking-of-vision,,,,,nan
+Honeybee: Locality-enhanced Projector for Multimodal LLM,Junbum Cha · Woo-Young Kang · Jonghwan Mun · Byungseok Roh, ,https://arxiv.org/abs/2312.06742,,2312.06742.pdf,Honeybee: Locality-enhanced Projector for Multimodal LLM,"In Multimodal Large Language Models (MLLMs), a visual projector plays a
+crucial role in bridging pre-trained vision encoders with LLMs, enabling
+profound visual understanding while harnessing the LLMs' robust capabilities.
+Despite the importance of the visual projector, it has been relatively less
+explored. In this study, we first identify two essential projector properties:
+(i) flexibility in managing the number of visual tokens, crucial for MLLMs'
+overall efficiency, and (ii) preservation of local context from visual
+features, vital for spatial understanding. Based on these findings, we propose
+a novel projector design that is both flexible and locality-enhanced,
+effectively satisfying the two desirable properties. Additionally, we present
+comprehensive strategies to effectively utilize multiple and multifaceted
+instruction datasets. Through extensive experiments, we examine the impact of
+individual design choices. Finally, our proposed MLLM, Honeybee, remarkably
+outperforms previous state-of-the-art methods across various benchmarks,
+including MME, MMBench, SEED-Bench, and LLaVA-Bench, achieving significantly
+higher efficiency. Code and models are available at
+https://github.com/kakaobrain/honeybee.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search,Junghyup Lee · Bumsub Ham, ,https://arxiv.org/abs/2403.19232,,2403.19232.pdf,AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search,"Training-free network architecture search (NAS) aims to discover
+high-performing networks with zero-cost proxies, capturing network
+characteristics related to the final performance. However, network rankings
+estimated by previous training-free NAS methods have shown weak correlations
+with the performance. To address this issue, we propose AZ-NAS, a novel
+approach that leverages the ensemble of various zero-cost proxies to enhance
+the correlation between a predicted ranking of networks and the ground truth
+substantially in terms of the performance. To achieve this, we introduce four
+novel zero-cost proxies that are complementary to each other, analyzing
+distinct traits of architectures in the views of expressivity, progressivity,
+trainability, and complexity. The proxy scores can be obtained simultaneously
+within a single forward and backward pass, making an overall NAS process highly
+efficient. In order to integrate the rankings predicted by our proxies
+effectively, we introduce a non-linear ranking aggregation method that
+highlights the networks highly-ranked consistently across all the proxies.
+Experimental results conclusively demonstrate the efficacy and efficiency of
+AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all
+while maintaining a reasonable runtime cost.",cs.CV,"['cs.CV', 'cs.LG']"
+Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching,Xianqi Wang · Gangwei Xu · Hao Jia · Xin Yang,https://github.com/Windsrain/Selective-Stereo,https://arxiv.org/abs/2403.00486,,2403.00486.pdf,Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching,"Stereo matching methods based on iterative optimization, like RAFT-Stereo and
+IGEV-Stereo, have evolved into a cornerstone in the field of stereo matching.
+However, these methods struggle to simultaneously capture high-frequency
+information in edges and low-frequency information in smooth regions due to the
+fixed receptive field. As a result, they tend to lose details, blur edges, and
+produce false matches in textureless areas. In this paper, we propose Selective
+Recurrent Unit (SRU), a novel iterative update operator for stereo matching.
+The SRU module can adaptively fuse hidden disparity information at multiple
+frequencies for edge and smooth regions. To perform adaptive fusion, we
+introduce a new Contextual Spatial Attention (CSA) module to generate attention
+maps as fusion weights. The SRU empowers the network to aggregate hidden
+disparity information across multiple frequencies, mitigating the risk of vital
+hidden disparity information loss during iterative processes. To verify SRU's
+universality, we apply it to representative iterative stereo matching methods,
+collectively referred to as Selective-Stereo. Our Selective-Stereo ranks
+$1^{st}$ on KITTI 2012, KITTI 2015, ETH3D, and Middlebury leaderboards among
+all published methods. Code is available at
+https://github.com/Windsrain/Selective-Stereo.",cs.CV,['cs.CV']
+Learning the 3D Fauna of the Web,Zizhang Li · Dor Litvak · Ruining Li · Yunzhi Zhang · Tomas Jakab · Christian Rupprecht · Shangzhe Wu · Andrea Vedaldi · Jiajun Wu, ,https://arxiv.org/abs/2401.02400,,2401.02400.pdf,Learning the 3D Fauna of the Web,"Learning 3D models of all animals on the Earth requires massively scaling up
+existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an
+approach that learns a pan-category deformable 3D animal model for more than
+100 animal species jointly. One crucial bottleneck of modeling animals is the
+limited availability of training data, which we overcome by simply learning
+from 2D Internet images. We show that prior category-specific attempts fail to
+generalize to rare species with limited training images. We address this
+challenge by introducing the Semantic Bank of Skinned Models (SBSM), which
+automatically discovers a small set of base animal shapes by combining
+geometric inductive priors with semantic knowledge implicitly captured by an
+off-the-shelf self-supervised feature extractor. To train such a model, we also
+contribute a new large-scale dataset of diverse animal species. At inference
+time, given a single image of any quadruped animal, our model reconstructs an
+articulated 3D mesh in a feed-forward fashion within seconds.",cs.CV,['cs.CV']
+LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking,Jialin Li · Qiang Nie · Weifu Fu · Yuhuan Lin · Guangpin Tao · Yong Liu · Chengjie Wang, ,https://arxiv.org/abs/2403.04303,,2403.04303.pdf,LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking,"Deep learning models, particularly those based on transformers, often employ
+numerous stacked structures, which possess identical architectures and perform
+similar functions. While effective, this stacking paradigm leads to a
+substantial increase in the number of parameters, posing challenges for
+practical applications. In today's landscape of increasingly large models,
+stacking depth can even reach dozens, further exacerbating this issue. To
+mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS
+allows stacked modules to share the majority of parameters, requiring a much
+smaller number of unique ones per module to match or even surpass the
+performance of using entirely distinct ones, thereby significantly reducing
+parameter usage. We validate our method by applying it to the stacked decoders
+of a query-based object detector, and conduct extensive experiments on the
+widely used MS COCO dataset. Experimental results demonstrate the effectiveness
+of our method, as even with a 70\% reduction in the parameters of the decoder,
+our method still enables the model to achieve comparable or",cs.CV,['cs.CV']
+VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation,Yang Chen · Yingwei Pan · haibo yang · Ting Yao · Tao Mei,https://vp3d-cvpr24.github.io/,https://arxiv.org/abs/2403.17001,,2403.17001.pdf,VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation,"Recent innovations on text-to-3D generation have featured Score Distillation
+Sampling (SDS), which enables the zero-shot learning of implicit 3D models
+(NeRF) by directly distilling prior knowledge from 2D diffusion models.
+However, current SDS-based models still struggle with intricate text prompts
+and commonly result in distorted 3D models with unrealistic textures or
+cross-view inconsistency issues. In this work, we introduce a novel Visual
+Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
+visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
+Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
+2D diffusion model to generate a high-quality image from input text, which
+subsequently acts as visual prompt to strengthen SDS optimization with explicit
+visual appearance. Meanwhile, we couple the SDS optimization with additional
+differentiable reward function that encourages rendering images of 3D models to
+better visually align with 2D visual prompt and semantically match with text
+prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
+VP3D significantly eases the learning of visual appearance of 3D models and
+thus leads to higher visual fidelity with more detailed textures. It is also
+appealing in view that when replacing the self-generating visual prompt with a
+given reference image, VP3D is able to trigger a new task of stylized
+text-to-3D generation. Our project page is available at
+https://vp3d-cvpr24.github.io.",cs.CV,"['cs.CV', 'cs.MM']"
+Vlogger: Make Your Dream A Vlog,Shaobin Zhuang · Kunchang Li · Xinyuan Chen · Yaohui Wang · Ziwei Liu · Yu Qiao · Yali Wang,https://github.com/zhuangshaobin/Vlogger,https://arxiv.org/abs/2401.09414,,2401.09414.pdf,Vlogger: Make Your Dream A Vlog,"In this work, we present Vlogger, a generic AI system for generating a
+minute-level video blog (i.e., vlog) of user descriptions. Different from short
+videos with a few seconds, vlog often contains a complex storyline with
+diversified scenes, which is challenging for most existing video generation
+approaches. To break through this bottleneck, our Vlogger smartly leverages
+Large Language Model (LLM) as Director and decomposes a long video generation
+task of vlog into four key stages, where we invoke various foundation models to
+play the critical roles of vlog professionals, including (1) Script, (2) Actor,
+(3) ShowMaker, and (4) Voicer. With such a design of mimicking human beings,
+our Vlogger can generate vlogs through explainable cooperation of top-down
+planning and bottom-up shooting. Moreover, we introduce a novel video diffusion
+model, ShowMaker, which serves as a videographer in our Vlogger for generating
+the video snippet of each shooting scene. By incorporating Script and Actor
+attentively as textual and visual prompts, it can effectively enhance
+spatial-temporal coherence in the snippet. Besides, we design a concise mixed
+training paradigm for ShowMaker, boosting its capacity for both T2V generation
+and prediction. Finally, the extensive experiments show that our method
+achieves state-of-the-art performance on zero-shot T2V generation and
+prediction tasks. More importantly, Vlogger can generate over 5-minute vlogs
+from open-world descriptions, without loss of video coherence on script and
+actor. The code and model is all available at
+https://github.com/zhuangshaobin/Vlogger.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation,Ruida Zhang · Chenyangguang Zhang · Yan Di · Fabian Manhardt · Xingyu Liu · Federico Tombari · Xiangyang Ji, ,https://arxiv.org/abs/2403.10099,,2403.10099.pdf,KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation,"In this paper, we present KP-RED, a unified KeyPoint-driven REtrieval and
+Deformation framework that takes object scans as input and jointly retrieves
+and deforms the most geometrically similar CAD models from a pre-processed
+database to tightly match the target. Unlike existing dense matching based
+methods that typically struggle with noisy partial scans, we propose to
+leverage category-consistent sparse keypoints to naturally handle both full and
+partial object scans. Specifically, we first employ a lightweight retrieval
+module to establish a keypoint-based embedding space, measuring the similarity
+among objects by dynamically aggregating deformation-aware local-global
+features around extracted keypoints. Objects that are close in the embedding
+space are considered similar in geometry. Then we introduce the neural
+cage-based deformation module that estimates the influence vector of each
+keypoint upon cage vertices inside its local support region to control the
+deformation of the retrieved shape. Extensive experiments on the synthetic
+dataset PartNet and the real-world dataset Scan2CAD demonstrate that KP-RED
+surpasses existing state-of-the-art approaches by a large margin. Codes and
+trained models will be released in https://github.com/lolrudy/KP-RED.",cs.CV,['cs.CV']
+AssistGUI: Task-Oriented PC Graphical User Interface Automation,Difei Gao · Lei Ji · Zechen Bai · Mingyu Ouyang · Peiran Li · Dongxing Mao · Qin WU · Weichen Zhang · Peiyi Wang · Xiangwu Guo · Hengxu Wang · Luowei Zhou · Mike Zheng Shou, ,https://arxiv.org/abs/2312.13108,,2312.13108.pdf,ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation,"Graphical User Interface (GUI) automation holds significant promise for
+assisting users with complex tasks, thereby boosting human productivity.
+Existing works leveraging Large Language Model (LLM) or LLM-based AI agents
+have shown capabilities in automating tasks on Android and Web platforms.
+However, these tasks are primarily aimed at simple device usage and
+entertainment operations. This paper presents a novel benchmark, AssistGUI, to
+evaluate whether models are capable of manipulating the mouse and keyboard on
+the Windows platform in response to user-requested tasks. We carefully
+collected a set of 100 tasks from nine widely-used software applications, such
+as, After Effects and MS Word, each accompanied by the necessary project files
+for better evaluation. Moreover, we propose an advanced Actor-Critic Embodied
+Agent framework, which incorporates a sophisticated GUI parser driven by an
+LLM-agent and an enhanced reasoning mechanism adept at handling lengthy
+procedural tasks. Our experimental results reveal that our GUI Parser and
+Reasoning mechanism outshine existing methods in performance. Nevertheless, the
+potential remains substantial, with the best model attaining only a 46% success
+rate on our benchmark. We conclude with a thorough analysis of the current
+methods' limitations, setting the stage for future breakthroughs in this
+domain.",cs.CV,['cs.CV']
+MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision,Chenyangguang Zhang · Guanlong Jiao · Yan Di · Gu Wang · Ziqin Huang · Ruida Zhang · Fabian Manhardt · Bowen Fu · Federico Tombari · Xiangyang Ji, ,https://arxiv.org/abs/2310.11696,,2310.11696.pdf,MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision,"Previous works concerning single-view hand-held object reconstruction
+typically rely on supervision from 3D ground-truth models, which are hard to
+collect in real world. In contrast, readily accessible hand-object videos offer
+a promising training data source, but they only give heavily occluded object
+observations. In this paper, we present a novel synthetic-to-real framework to
+exploit Multi-view Occlusion-aware supervision from hand-object videos for
+Hand-held Object reconstruction (MOHO) from a single image, tackling two
+predominant challenges in such setting: hand-induced occlusion and object's
+self-occlusion. First, in the synthetic pre-training stage, we render a
+large-scaled synthetic dataset SOMVideo with hand-object images and multi-view
+occlusion-free supervisions, adopted to address hand-induced occlusion in both
+2D and 3D spaces. Second, in the real-world finetuning stage, MOHO leverages
+the amodal-mask-weighted geometric supervision to mitigate the unfaithful
+guidance caused by the hand-occluded supervising views in real world. Moreover,
+domain-consistent occlusion-aware features are amalgamated in MOHO to resist
+object's self-occlusion for inferring the complete object shape. Extensive
+experiments on HO3D and DexYCB datasets demonstrate 2D-supervised MOHO gains
+superior results against 3D-supervised methods by a large margin.",cs.CV,['cs.CV']
+Text-Guided 3D Face Synthesis - From Generation to Editing,Yunjie Wu · Yapeng Meng · Zhipeng Hu · Lincheng Li · Haoqian Wu · Kun Zhou · Weiwei Xu · Xin Yu, ,https://arxiv.org/abs/2312.00375,,2312.00375.pdf,Text-Guided 3D Face Synthesis -- From Generation to Editing,"Text-guided 3D face synthesis has achieved remarkable results by leveraging
+text-to-image (T2I) diffusion models. However, most existing works focus solely
+on the direct generation, ignoring the editing, restricting them from
+synthesizing customized 3D faces through iterative adjustments. In this paper,
+we propose a unified text-guided framework from face generation to editing. In
+the generation stage, we propose a geometry-texture decoupled generation to
+mitigate the loss of geometric details caused by coupling. Besides, decoupling
+enables us to utilize the generated geometry as a condition for texture
+generation, yielding highly geometry-texture aligned results. We further employ
+a fine-tuned texture diffusion model to enhance texture quality in both RGB and
+YUV space. In the editing stage, we first employ a pre-trained diffusion model
+to update facial geometry or texture based on the texts. To enable sequential
+editing, we introduce a UV domain consistency preservation regularization,
+preventing unintentional changes to irrelevant facial attributes. Besides, we
+propose a self-guided consistency weight strategy to improve editing efficacy
+while preserving consistency. Through comprehensive experiments, we showcase
+our method's superiority in face synthesis. Project page:
+https://faceg2e.github.io/.",cs.CV,['cs.CV']
+Characteristics Matching Based Hash Codes Generation for Efficient Fine-grained Image Retrieval,Zhen-Duo Chen · Li-Jun Zhao · Zi-Chao Zhang · Xin Luo · Xin-Shun Xu, ,https://arxiv.org/abs/2311.06067,,2311.06067.pdf,Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval,"In recent years, hashing methods have been popular in the large-scale media
+search for low storage and strong representation capabilities. To describe
+objects with similar overall appearance but subtle differences, more and more
+studies focus on hashing-based fine-grained image retrieval. Existing hashing
+networks usually generate both local and global features through attention
+guidance on the same deep activation tensor, which limits the diversity of
+feature representations. To handle this limitation, we substitute convolutional
+descriptors for attention-guided features and propose an Attributes Grouping
+and Mining Hashing (AGMH), which groups and embeds the category-specific visual
+attributes in multiple descriptors to generate a comprehensive feature
+representation for efficient fine-grained image retrieval. Specifically, an
+Attention Dispersion Loss (ADL) is designed to force the descriptors to attend
+to various local regions and capture diverse subtle details. Moreover, we
+propose a Stepwise Interactive External Attention (SIEA) to mine critical
+attributes in each descriptor and construct correlations between fine-grained
+attributes and objects. The attention mechanism is dedicated to learning
+discrete attributes, which will not cost additional computations in hash codes
+generation. Finally, the compact binary codes are learned by preserving
+pairwise similarities. Experimental results demonstrate that AGMH consistently
+yields the best performance against state-of-the-art methods on fine-grained
+benchmark datasets.",cs.IR,"['cs.IR', 'cs.AI', 'cs.CV']"
+VOODOO 3D: VOlumetric pOrtrait Disentanglement fOr Online 3D head reenactment,Phong Tran · Egor Zakharov · Long Nhat Ho · Anh Tran · Liwen Hu · Hao Li, ,https://arxiv.org/abs/2312.04651,,2312.04651.pdf,VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment,"We present a 3D-aware one-shot head reenactment method based on a fully
+volumetric neural disentanglement framework for source appearance and driver
+expressions. Our method is real-time and produces high-fidelity and
+view-consistent output, suitable for 3D teleconferencing systems based on
+holographic displays. Existing cutting-edge 3D-aware reenactment methods often
+use neural radiance fields or 3D meshes to produce view-consistent appearance
+encoding, but, at the same time, they rely on linear face models, such as 3DMM,
+to achieve its disentanglement with facial expressions. As a result, their
+reenactment results often exhibit identity leakage from the driver or have
+unnatural expressions. To address these problems, we propose a neural
+self-supervised disentanglement approach that lifts both the source image and
+driver video frame into a shared 3D volumetric representation based on
+tri-planes. This representation can then be freely manipulated with expression
+tri-planes extracted from the driving images and rendered from an arbitrary
+view using neural radiance fields. We achieve this disentanglement via
+self-supervised learning on a large in-the-wild video dataset. We further
+introduce a highly effective fine-tuning approach to improve the
+generalizability of the 3D lifting using the same real-world data. We
+demonstrate state-of-the-art performance on a wide range of datasets, and also
+showcase high-quality 3D-aware head reenactment on highly challenging and
+diverse subjects, including non-frontal head poses and complex expressions for
+both source and driver.",cs.CV,['cs.CV']
+Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence,Junyi Zhang · Charles Herrmann · Junhwa Hur · Eric Chen · Varun Jampani · Deqing Sun · Ming-Hsuan Yang,telling-left-from-right.github.io,https://arxiv.org/abs/2311.17034,,2311.17034.pdf,Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence,"While pre-trained large-scale vision models have shown significant promise
+for semantic correspondence, their features often struggle to grasp the
+geometry and orientation of instances. This paper identifies the importance of
+being geometry-aware for semantic correspondence and reveals a limitation of
+the features of current foundation models under simple post-processing. We show
+that incorporating this information can markedly enhance semantic
+correspondence performance with simple but effective solutions in both
+zero-shot and supervised settings. We also construct a new challenging
+benchmark for semantic correspondence built from an existing animal pose
+estimation dataset, for both pre-training validating models. Our method
+achieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on the
+challenging SPair-71k dataset, outperforming the state of the art by 5.5p and
+11.0p absolute gains, respectively. Our code and datasets are publicly
+available at: https://telling-left-from-right.github.io/.",cs.CV,['cs.CV']
+Federated Generalized Category Discovery,Nan Pu · Wenjing Li · Xinyuan Ji · Yalan Qin · Nicu Sebe · Zhun Zhong, ,https://arxiv.org/abs/2403.07369,,2403.07369.pdf,Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discovery,"In this paper, we study the problem of Generalized Category Discovery (GCD),
+which aims to cluster unlabeled data from both known and unknown categories
+using the knowledge of labeled data from known categories. Current GCD methods
+rely on only visual cues, which however neglect the multi-modality perceptive
+nature of human cognitive processes in discovering novel visual categories. To
+address this, we propose a two-phase TextGCD framework to accomplish
+multi-modality GCD by exploiting powerful Visual-Language Models. TextGCD
+mainly includes a retrieval-based text generation (RTG) phase and a
+cross-modality co-teaching (CCT) phase. First, RTG constructs a visual lexicon
+using category tags from diverse datasets and attributes from Large Language
+Models, generating descriptive texts for images in a retrieval manner. Second,
+CCT leverages disparities between textual and visual modalities to foster
+mutual learning, thereby enhancing visual GCD. In addition, we design an
+adaptive class aligning strategy to ensure the alignment of category
+perceptions between modalities as well as a soft-voting mechanism to integrate
+multi-modality cues. Experiments on eight datasets show the large superiority
+of our approach over state-of-the-art methods. Notably, our approach
+outperforms the best competitor, by 7.7% and 10.8% in All accuracy on
+ImageNet-1k and CUB, respectively.",cs.CV,['cs.CV']
+LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes,Shanlin Sun · Bingbing Zhuang · Ziyu Jiang · Buyu Liu · Xiaohui Xie · Manmohan Chandraker, ,https://arxiv.org/abs/2405.00900,,2405.00900.pdf,LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes,"Photorealistic simulation plays a crucial role in applications such as
+autonomous driving, where advances in neural radiance fields (NeRFs) may allow
+better scalability through the automatic creation of digital 3D assets.
+However, reconstruction quality suffers on street scenes due to largely
+collinear camera motions and sparser samplings at higher speeds. On the other
+hand, the application often demands rendering from camera views that deviate
+from the inputs to accurately simulate behaviors like lane changes. In this
+paper, we propose several insights that allow a better utilization of Lidar
+data to improve NeRF quality on street scenes. First, our framework learns a
+geometric scene representation from Lidar, which is fused with the implicit
+grid-based representation for radiance decoding, thereby supplying stronger
+geometric information offered by explicit point cloud. Second, we put forth a
+robust occlusion-aware depth supervision scheme, which allows utilizing
+densified Lidar points by accumulation. Third, we generate augmented training
+views from Lidar points for further improvement. Our insights translate to
+largely improved novel view synthesis under real driving scenes.",cs.CV,['cs.CV']
+Learning Occupancy for Monocular 3D Object Detection,Liang Peng · Junkai Xu · Haoran Cheng · Zheng Yang · Xiaopei Wu · Wei Qian · Wenxiao Wang · Boxi Wu · Deng Cai, ,https://arxiv.org/abs/2308.09421,,2308.09421.pdf,MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection,"In the field of monocular 3D detection, it is common practice to utilize
+scene geometric clues to enhance the detector's performance. However, many
+existing works adopt these clues explicitly such as estimating a depth map and
+back-projecting it into 3D space. This explicit methodology induces sparsity in
+3D representations due to the increased dimensionality from 2D to 3D, and leads
+to substantial information loss, especially for distant and occluded objects.
+To alleviate this issue, we propose MonoNeRD, a novel detection framework that
+can infer dense 3D geometry and occupancy. Specifically, we model scenes with
+Signed Distance Functions (SDF), facilitating the production of dense 3D
+representations. We treat these representations as Neural Radiance Fields
+(NeRF) and then employ volume rendering to recover RGB images and depth maps.
+To the best of our knowledge, this work is the first to introduce volume
+rendering for M3D, and demonstrates the potential of implicit reconstruction
+for image-based 3D perception. Extensive experiments conducted on the KITTI-3D
+benchmark and Waymo Open Dataset demonstrate the effectiveness of MonoNeRD.
+Codes are available at https://github.com/cskkxjk/MonoNeRD.",cs.CV,['cs.CV']
+CaDeT: a Causal Disentanglement Approach for Robust Trajectory Prediction in Autonomous Driving,Mozhgan Pourkeshavarz · Junrui Zhang · Amir Rasouli, ,https://arxiv.org/abs/2404.12538,,2404.12538.pdf,TrACT: A Training Dynamics Aware Contrastive Learning Framework for Long-tail Trajectory Prediction,"As a safety critical task, autonomous driving requires accurate predictions
+of road users' future trajectories for safe motion planning, particularly under
+challenging conditions. Yet, many recent deep learning methods suffer from a
+degraded performance on the challenging scenarios, mainly because these
+scenarios appear less frequently in the training data. To address such a
+long-tail issue, existing methods force challenging scenarios closer together
+in the feature space during training to trigger information sharing among them
+for more robust learning. These methods, however, primarily rely on the motion
+patterns to characterize scenarios, omitting more informative contextual
+information, such as interactions and scene layout. We argue that exploiting
+such information not only improves prediction accuracy but also scene
+compliance of the generated trajectories. In this paper, we propose to
+incorporate richer training dynamics information into a prototypical
+contrastive learning framework. More specifically, we propose a two-stage
+process. First, we generate rich contextual features using a baseline
+encoder-decoder framework. These features are split into clusters based on the
+model's output errors, using the training dynamics information, and a prototype
+is computed within each cluster. Second, we retrain the model using the
+prototypes in a contrastive learning framework. We conduct empirical
+evaluations of our approach using two large-scale naturalistic datasets and
+show that our method achieves state-of-the-art performance by improving
+accuracy and scene compliance on the long-tail samples. Furthermore, we perform
+experiments on a subset of the clusters to highlight the additional benefit of
+our approach in reducing training bias.",cs.CV,"['cs.CV', 'cs.LG']"
+Towards HDR and HFR Video from Rolling-Mixed-Bit Spikings,Yakun Chang · Yeliduosi Xiaokaiti · Yujia Liu · Bin Fan · Zhaojun Huang · Tiejun Huang · Boxin Shi, ,https://arxiv.org/abs/2405.00244,,,Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network,"As an important and practical way to obtain high dynamic range (HDR) video,
+HDR video reconstruction from sequences with alternating exposures is still
+less explored, mainly due to the lack of large-scale real-world datasets.
+Existing methods are mostly trained on synthetic datasets, which perform poorly
+in real scenes. In this work, to facilitate the development of real-world HDR
+video reconstruction, we present Real-HDRV, a large-scale real-world benchmark
+dataset for HDR video reconstruction, featuring various scenes, diverse motion
+patterns, and high-quality labels. Specifically, our dataset contains 500
+LDRs-HDRs video pairs, comprising about 28,000 LDR frames and 4,000 HDR labels,
+covering daytime, nighttime, indoor, and outdoor scenes. To our best knowledge,
+our dataset is the largest real-world HDR video reconstruction dataset.
+Correspondingly, we propose an end-to-end network for HDR video reconstruction,
+where a novel two-stage strategy is designed to perform alignment sequentially.
+Specifically, the first stage performs global alignment with the adaptively
+estimated global offsets, reducing the difficulty of subsequent alignment. The
+second stage implicitly performs local alignment in a coarse-to-fine manner at
+the feature level using the adaptive separable convolution. Extensive
+experiments demonstrate that: (1) models trained on our dataset can achieve
+better performance on real scenes than those trained on synthetic datasets; (2)
+our method outperforms previous state-of-the-art methods. Our dataset is
+available at https://github.com/yungsyu99/Real-HDRV.",cs.CV,['cs.CV']
+Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression,Hancheng Ye · Chong Yu · Peng Ye · Renqiu Xia · Bo Zhang · Yansong Tang · Jiwen Lu · Tao Chen, ,https://arxiv.org/abs/2403.15835,,2403.15835.pdf,Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression,"Recent Vision Transformer Compression (VTC) works mainly follow a two-stage
+scheme, where the importance score of each model unit is first evaluated or
+preset in each submodule, followed by the sparsity score evaluation according
+to the target sparsity constraint. Such a separate evaluation process induces
+the gap between importance and sparsity score distributions, thus causing high
+search costs for VTC. In this work, for the first time, we investigate how to
+integrate the evaluations of importance and sparsity scores into a single
+stage, searching the optimal subnets in an efficient manner. Specifically, we
+present OFB, a cost-efficient approach that simultaneously evaluates both
+importance and sparsity scores, termed Once for Both (OFB), for VTC. First, a
+bi-mask scheme is developed by entangling the importance score and the
+differentiable sparsity score to jointly determine the pruning potential
+(prunability) of each unit. Such a bi-mask search strategy is further used
+together with a proposed adaptive one-hot loss to realize the
+progressive-and-efficient search for the most important subnet. Finally,
+Progressive Masked Image Modeling (PMIM) is proposed to regularize the feature
+space to be more representative during the search process, which may be
+degraded by the dimension reduction. Extensive experiments demonstrate that OFB
+can achieve superior compression performance over state-of-the-art
+searching-based and pruning-based methods under various Vision Transformer
+architectures, meanwhile promoting search efficiency significantly, e.g.,
+costing one GPU search day for the compression of DeiT-S on ImageNet-1K.",cs.CV,['cs.CV']
+GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians,Liangxiao Hu · Hongwen Zhang · Yuxiang Zhang · Boyao ZHOU · Boning Liu · Shengping Zhang · Liqiang Nie,https://huliangxiao.github.io/GaussianAvatar,https://arxiv.org/abs/2312.02134,,2312.02134.pdf,GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians,"We present GaussianAvatar, an efficient approach to creating realistic human
+avatars with dynamic 3D appearances from a single video. We start by
+introducing animatable 3D Gaussians to explicitly represent humans in various
+poses and clothing styles. Such an explicit and animatable representation can
+fuse 3D appearances more efficiently and consistently from 2D observations. Our
+representation is further augmented with dynamic properties to support
+pose-dependent appearance modeling, where a dynamic appearance network along
+with an optimizable feature tensor is designed to learn the
+motion-to-appearance mapping. Moreover, by leveraging the differentiable motion
+condition, our method enables a joint optimization of motions and appearances
+during avatar modeling, which helps to tackle the long-standing issue of
+inaccurate motion estimation in monocular settings. The efficacy of
+GaussianAvatar is validated on both the public dataset and our collected
+dataset, demonstrating its superior performances in terms of appearance quality
+and rendering efficiency.",cs.CV,['cs.CV']
+OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning,Lingyi Hong · Shilin Yan · Renrui Zhang · Wanyun Li · Xinyu Zhou · Pinxue Guo · Kaixun Jiang · Yiting Cheng · Jinglun Li · Zhaoyu Chen · Wenqiang Zhang, ,https://arxiv.org/abs/2403.09634,,2403.09634.pdf,OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning,"Visual object tracking aims to localize the target object of each frame based
+on its initial appearance in the first frame. Depending on the input modility,
+tracking tasks can be divided into RGB tracking and RGB+X (e.g. RGB+N, and
+RGB+D) tracking. Despite the different input modalities, the core aspect of
+tracking is the temporal matching. Based on this common ground, we present a
+general framework to unify various tracking tasks, termed as OneTracker.
+OneTracker first performs a large-scale pre-training on a RGB tracker called
+Foundation Tracker. This pretraining phase equips the Foundation Tracker with a
+stable ability to estimate the location of the target object. Then we regard
+other modality information as prompt and build Prompt Tracker upon Foundation
+Tracker. Through freezing the Foundation Tracker and only adjusting some
+additional trainable parameters, Prompt Tracker inhibits the strong
+localization ability from Foundation Tracker and achieves parameter-efficient
+finetuning on downstream RGB+X tracking tasks. To evaluate the effectiveness of
+our general framework OneTracker, which is consisted of Foundation Tracker and
+Prompt Tracker, we conduct extensive experiments on 6 popular tracking tasks
+across 11 benchmarks and our OneTracker outperforms other models and achieves
+state-of-the-art performance.",cs.CV,['cs.CV']
+TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation,Hoonhee Cho · Taewoo Kim · Yuhwan Jeong · Kuk-Jin Yoon, ,https://arxiv.org/abs/2404.18156,,,Event-based Video Frame Interpolation with Edge Guided Motion Refinement,"Video frame interpolation, the process of synthesizing intermediate frames
+between sequential video frames, has made remarkable progress with the use of
+event cameras. These sensors, with microsecond-level temporal resolution, fill
+information gaps between frames by providing precise motion cues. However,
+contemporary Event-Based Video Frame Interpolation (E-VFI) techniques often
+neglect the fact that event data primarily supply high-confidence features at
+scene edges during multi-modal feature fusion, thereby diminishing the role of
+event signals in optical flow (OF) estimation and warping refinement. To
+address this overlooked aspect, we introduce an end-to-end E-VFI learning
+method (referred to as EGMR) to efficiently utilize edge features from event
+signals for motion flow and warping enhancement. Our method incorporates an
+Edge Guided Attentive (EGA) module, which rectifies estimated video motion
+through attentive aggregation based on the local correlation of multi-modal
+features in a coarse-to-fine strategy. Moreover, given that event data can
+provide accurate visual references at scene edges between consecutive frames,
+we introduce a learned visibility map derived from event data to adaptively
+mitigate the occlusion problem in the warping refinement process. Extensive
+experiments on both synthetic and real datasets show the effectiveness of the
+proposed approach, demonstrating its potential for higher quality video frame
+interpolation.",cs.CV,['cs.CV']
+GigaTraj: Predicting Long-term Trajectories of Hundreds of Pedestrians in Gigapixel Complex Scenes,Haozhe Lin · Chunyu Wei · Li He · Yuchen Guo · Yuchy Zhao · Shanglong Li · Lu Fang, ,https://arxiv.org/abs/2402.19002,,2402.19002.pdf,GoalNet: Goal Areas Oriented Pedestrian Trajectory Prediction,"Predicting the future trajectories of pedestrians on the road is an important
+task for autonomous driving. The pedestrian trajectory prediction is affected
+by scene paths, pedestrian's intentions and decision-making, which is a
+multi-modal problem. Most recent studies use past trajectories to predict a
+variety of potential future trajectory distributions, which do not account for
+the scene context and pedestrian targets. Instead of predicting the future
+trajectory directly, we propose to use scene context and observed trajectory to
+predict the goal points first, and then reuse the goal points to predict the
+future trajectories. By leveraging the information from scene context and
+observed trajectory, the uncertainty can be limited to a few target areas,
+which represent the ""goals"" of the pedestrians. In this paper, we propose
+GoalNet, a new trajectory prediction neural network based on the goal areas of
+a pedestrian. Our network can predict both pedestrian's trajectories and
+bounding boxes. The overall model is efficient and modular, and its outputs can
+be changed according to the usage scenario. Experimental results show that
+GoalNet significantly improves the previous state-of-the-art performance by
+48.7% on the JAAD and 40.8% on the PIE dataset.",cs.CV,"['cs.CV', 'cs.AI']"
+Discovering Syntactic Interaction Clues for Human-Object Interaction Detection,Jinguo Luo · Weihong Ren · Weibo Jiang · Xi'ai Chen · Qiang Wang · Zhi Han · Honghai LIU, ,,https://www.youtube.com/watch?v=YxKgZAoqzpY,,,,,nan
+Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches,Qing Yu · Mikihiro Tanaka · Kent Fujiwara,https://yu1ut.com/MotionPatches-HP/,https://arxiv.org/abs/2405.04771,,2405.04771.pdf,Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches,"To build a cross-modal latent space between 3D human motion and language,
+acquiring large-scale and high-quality human motion data is crucial. However,
+unlike the abundance of image data, the scarcity of motion data has limited the
+performance of existing motion-language models. To counter this, we introduce
+""motion patches"", a new representation of motion sequences, and propose using
+Vision Transformers (ViT) as motion encoders via transfer learning, aiming to
+extract useful knowledge from the image domain and apply it to the motion
+domain. These motion patches, created by dividing and sorting skeleton joints
+based on body parts in motion sequences, are robust to varying skeleton
+structures, and can be regarded as color image patches in ViT. We find that
+transfer learning with pre-trained weights of ViT obtained through training
+with 2D image data can boost the performance of motion analysis, presenting a
+promising direction for addressing the issue of limited motion data. Our
+extensive experiments show that the proposed motion patches, used jointly with
+ViT, achieve state-of-the-art performance in the benchmarks of text-to-motion
+retrieval, and other novel challenging tasks, such as cross-skeleton
+recognition, zero-shot motion classification, and human interaction
+recognition, which are currently impeded by the lack of data.",cs.CV,['cs.CV']
+ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification,Jiangbo Shi · Chen Li · Tieliang Gong · Yefeng Zheng · Huazhu Fu, ,https://arxiv.org/abs/2312.01099,,2312.01099.pdf,Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Bag-Level Classifier is a Good Instance-Level Teacher,"Multiple Instance Learning (MIL) has demonstrated promise in Whole Slide
+Image (WSI) classification. However, a major challenge persists due to the high
+computational cost associated with processing these gigapixel images. Existing
+methods generally adopt a two-stage approach, comprising a non-learnable
+feature embedding stage and a classifier training stage. Though it can greatly
+reduce the memory consumption by using a fixed feature embedder pre-trained on
+other domains, such scheme also results in a disparity between the two stages,
+leading to suboptimal classification accuracy. To address this issue, we
+propose that a bag-level classifier can be a good instance-level teacher. Based
+on this idea, we design Iteratively Coupled Multiple Instance Learning (ICMIL)
+to couple the embedder and the bag classifier at a low cost. ICMIL initially
+fix the patch embedder to train the bag classifier, followed by fixing the bag
+classifier to fine-tune the patch embedder. The refined embedder can then
+generate better representations in return, leading to a more accurate
+classifier for the next iteration. To realize more flexible and more effective
+embedder fine-tuning, we also introduce a teacher-student framework to
+efficiently distill the category knowledge in the bag classifier to help the
+instance-level embedder fine-tuning. Thorough experiments were conducted on
+four distinct datasets to validate the effectiveness of ICMIL. The experimental
+results consistently demonstrate that our method significantly improves the
+performance of existing MIL backbones, achieving state-of-the-art results. The
+code is available at: https://github.com/Dootmaan/ICMIL/tree/confidence_based",cs.CV,['cs.CV']
+Neural Visibility Field for Uncertainty-Driven Active Mapping,Shangjie Xue · Jesse Dill · Pranay Mathur · Frank Dellaert · Panagiotis Tsiotras · Danfei Xu, ,http://export.arxiv.org/abs/2308.16246,,2308.16246.pdf,Active Neural Mapping,"We address the problem of active mapping with a continually-learned neural
+scene representation, namely Active Neural Mapping. The key lies in actively
+finding the target space to be explored with efficient agent movement, thus
+minimizing the map uncertainty on-the-fly within a previously unseen
+environment. In this paper, we examine the weight space of the
+continually-learned neural field, and show empirically that the neural
+variability, the prediction robustness against random weight perturbation, can
+be directly utilized to measure the instant uncertainty of the neural map.
+Together with the continuous geometric information inherited in the neural map,
+the agent can be guided to find a traversable path to gradually gain knowledge
+of the environment. We present for the first time an active mapping system with
+a coordinate-based implicit neural representation for online scene
+reconstruction. Experiments in the visually-realistic Gibson and Matterport3D
+environment demonstrate the efficacy of the proposed method.",cs.CV,['cs.CV']
+Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors,Haoxuanye Ji · Pengpeng Liang · Erkang Cheng, ,https://arxiv.org/abs/2403.06093,,2403.06093.pdf,Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors,"Multi-camera-based 3D object detection has made notable progress in the past
+several years. However, we observe that there are cases (e.g. faraway regions)
+in which popular 2D object detectors are more reliable than state-of-the-art 3D
+detectors. In this paper, to improve the performance of query-based 3D object
+detectors, we present a novel query generating approach termed QAF2D, which
+infers 3D query anchors from 2D detection results. A 2D bounding box of an
+object in an image is lifted to a set of 3D anchors by associating each sampled
+point within the box with depth, yaw angle, and size candidates. Then, the
+validity of each 3D anchor is verified by comparing its projection in the image
+with its corresponding 2D box, and only valid anchors are kept and used to
+construct queries. The class information of the 2D bounding box associated with
+each query is also utilized to match the predicted boxes with ground truth for
+the set-based loss. The image feature extraction backbone is shared between the
+3D detector and 2D detector by adding a small number of prompt parameters. We
+integrate QAF2D into three popular query-based 3D object detectors and carry
+out comprehensive evaluations on the nuScenes dataset. The largest improvement
+that QAF2D can bring about on the nuScenes validation subset is $2.3\%$ NDS and
+$2.7\%$ mAP. Code is available at https://github.com/nullmax-vision/QAF2D.",cs.CV,['cs.CV']
+Resolution Limit of Single-Photon LIDAR,Stanley H. Chan · Hashan K Weerasooriya · Weijian Zhang · Pamela Abshire · Istvan Gyongy · Robert Henderson, ,https://arxiv.org/abs/2403.17719,,2403.17719.pdf,Resolution Limit of Single-Photon LiDAR,"Single-photon Light Detection and Ranging (LiDAR) systems are often equipped
+with an array of detectors for improved spatial resolution and sensing speed.
+However, given a fixed amount of flux produced by the laser transmitter across
+the scene, the per-pixel Signal-to-Noise Ratio (SNR) will decrease when more
+pixels are packed in a unit space. This presents a fundamental trade-off
+between the spatial resolution of the sensor array and the SNR received at each
+pixel. Theoretical characterization of this fundamental limit is explored. By
+deriving the photon arrival statistics and introducing a series of new
+approximation techniques, the Mean Squared Error (MSE) of the
+maximum-likelihood estimator of the time delay is derived. The theoretical
+predictions align well with simulations and real data.",eess.SP,"['eess.SP', 'cs.CV']"
+VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Models,Xiang Li · Qianli Shen · Kenji Kawaguchi, ,https://arxiv.org/abs/2312.00057,,2312.00057.pdf,VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Models,"The booming use of text-to-image generative models has raised concerns about
+their high risk of producing copyright-infringing content. While probabilistic
+copyright protection methods provide a probabilistic guarantee against such
+infringement, in this paper, we introduce Virtually Assured Amplification
+Attack (VA3), a novel online attack framework that exposes the vulnerabilities
+of these protection mechanisms. The proposed framework significantly amplifies
+the probability of generating infringing content on the sustained interactions
+with generative models and a non-trivial lower-bound on the success probability
+of each engagement. Our theoretical and experimental results demonstrate the
+effectiveness of our approach under various scenarios. These findings highlight
+the potential risk of implementing probabilistic copyright protection in
+practical applications of text-to-image generative models. Code is available at
+https://github.com/South7X/VA3.",cs.CR,"['cs.CR', 'cs.AI', 'cs.CV', 'cs.MM']"
+Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion,Fan Zhang · Shaodi You · Yu Li · Ying Fu,https://github.com/zkawfanx/Atlantis,https://arxiv.org/abs/2312.12471,,2312.12471.pdf,Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion,"Monocular depth estimation has experienced significant progress on
+terrestrial images in recent years, largely due to deep learning advancements.
+However, it remains inadequate for underwater scenes, primarily because of data
+scarcity. Given the inherent challenges of light attenuation and backscattering
+in water, acquiring clear underwater images or precise depth information is
+notably difficult and costly. Consequently, learning-based approaches often
+rely on synthetic data or turn to unsupervised or self-supervised methods to
+mitigate this lack of data. Nonetheless, the performance of these methods is
+often constrained by the domain gap and looser constraints. In this paper, we
+propose a novel pipeline for generating photorealistic underwater images using
+accurate terrestrial depth data. This approach facilitates the training of
+supervised models for underwater depth estimation, effectively reducing the
+performance disparity between terrestrial and underwater environments. Contrary
+to prior synthetic datasets that merely apply style transfer to terrestrial
+images without altering the scene content, our approach uniquely creates
+vibrant, non-existent underwater scenes by leveraging terrestrial depth data
+through the innovative Stable Diffusion model. Specifically, we introduce a
+unique Depth2Underwater ControlNet, trained on specially prepared \{Underwater,
+Depth, Text\} data triplets, for this generation task. Our newly developed
+dataset enables terrestrial depth estimation models to achieve considerable
+improvements, both quantitatively and qualitatively, on unseen underwater
+images, surpassing their terrestrial pre-trained counterparts. Moreover, the
+enhanced depth accuracy for underwater scenes also aids underwater image
+restoration techniques that rely on depth maps, further demonstrating our
+dataset's utility. The dataset will be available at
+https://github.com/zkawfanx/Atlantis.",cs.CV,['cs.CV']
+ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations,Maitreya Patel · Changhoon Kim · Sheng Cheng · Chitta Baral · 'YZ' Yezhou Yang, ,https://arxiv.org/abs/2312.04655,,2312.04655.pdf,ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations,"Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
+DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
+T2I benchmarks, at the cost of significant computational resources. The unCLIP
+stack comprises T2I prior and diffusion image decoder. The T2I prior model
+alone adds a billion parameters compared to the Latent Diffusion Models, which
+increases the computational and high-quality data requirements. We introduce
+ECLIPSE, a novel contrastive learning method that is both parameter and
+data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
+CLIP) to distill the knowledge into the prior model. We demonstrate that the
+ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
+2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
+preference score under resource-limited setting. It also attains performance on
+par with SOTA big models, achieving an average of 63.36% preference score in
+terms of the ability to follow the text compositions. Extensive experiments on
+two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
+priors consistently deliver high performance while significantly reducing
+resource dependency.",cs.CV,['cs.CV']
+Hallucination Augmented Contrastive Learning for Multimodal Large Language Model,Chaoya Jiang · Haiyang Xu · Mengfan Dong · Jiaxing Chen · Wei Ye · Ming Yan · Qinghao Ye · Ji Zhang · Fei Huang · Fei Huang · Shikun Zhang, ,https://arxiv.org/abs/2312.06968,,2312.06968.pdf,Hallucination Augmented Contrastive Learning for Multimodal Large Language Model,"Multi-modal large language models (MLLMs) have been shown to efficiently
+integrate natural language with visual information to handle multi-modal tasks.
+However, MLLMs still face a fundamental limitation of hallucinations, where
+they tend to generate erroneous or fabricated information. In this paper, we
+address hallucinations in MLLMs from a novel perspective of representation
+learning. We first analyzed the representation distribution of textual and
+visual tokens in MLLM, revealing two important findings: 1) there is a
+significant gap between textual and visual representations, indicating
+unsatisfactory cross-modal representation alignment; 2) representations of
+texts that contain and do not contain hallucinations are entangled, making it
+challenging to distinguish them. These two observations inspire us with a
+simple yet effective method to mitigate hallucinations. Specifically, we
+introduce contrastive learning into MLLMs and use text with hallucination as
+hard negative examples, naturally bringing representations of non-hallucinative
+text and visual samples closer while pushing way representations of
+non-hallucinating and hallucinative text. We evaluate our method quantitatively
+and qualitatively, showing its effectiveness in reducing hallucination
+occurrences and improving performance across multiple benchmarks. On the
+MMhal-Bench benchmark, our method obtains a 34.66% /29.5% improvement over the
+baseline MiniGPT-4/LLaVA. Our code is available on
+https://github.com/X-PLUG/mPLUG-HalOwl/tree/main/hacl.",cs.CV,['cs.CV']
+Active Domain Adaptation with False Negative Prediction for Object Detection,Yuzuru Nakamura · Yasunori Ishii · Takayoshi Yamashita, ,https://arxiv.org/abs/2307.07944,,2307.07944.pdf,"Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labeling","Unsupervised domain adaptation (DA) with the aid of pseudo labeling
+techniques has emerged as a crucial approach for domain-adaptive 3D object
+detection. While effective, existing DA methods suffer from a substantial drop
+in performance when applied to a multi-class training setting, due to the
+co-existence of low-quality pseudo labels and class imbalance issues. In this
+paper, we address this challenge by proposing a novel ReDB framework tailored
+for learning to detect all classes at once. Our approach produces Reliable,
+Diverse, and class-Balanced pseudo 3D boxes to iteratively guide the
+self-training on a distributionally different target domain. To alleviate
+disruptions caused by the environmental discrepancy (e.g., beam numbers), the
+proposed cross-domain examination (CDE) assesses the correctness of pseudo
+labels by copy-pasting target instances into a source environment and measuring
+the prediction consistency. To reduce computational overhead and mitigate the
+object shift (e.g., scales and point densities), we design an overlapped boxes
+counting (OBC) metric that allows to uniformly downsample pseudo-labeled
+objects across different geometric characteristics. To confront the issue of
+inter-class imbalance, we progressively augment the target point clouds with a
+class-balanced set of pseudo-labeled target instances and source objects, which
+boosts recognition accuracies on both frequently appearing and rare classes.
+Experimental results on three benchmark datasets using both voxel-based (i.e.,
+SECOND) and point-based 3D detectors (i.e., PointRCNN) demonstrate that our
+proposed ReDB approach outperforms existing 3D domain adaptation methods by a
+large margin, improving 23.15% mAP on the nuScenes $\rightarrow$ KITTI task.
+The code is available at https://github.com/zhuoxiao-chen/ReDB-DA-3Ddet.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Material Palette: Extraction of Materials from a Single Image,Ivan Lopes · Fabio Pizzati · Raoul de Charette,https://astra-vision.github.io/MaterialPalette/,https://arxiv.org/abs/2311.17060v1,,2311.17060v1.pdf,Material Palette: Extraction of Materials from a Single Image,"In this paper, we propose a method to extract physically-based rendering
+(PBR) materials from a single real-world image. We do so in two steps: first,
+we map regions of the image to material concepts using a diffusion model, which
+allows the sampling of texture images resembling each material in the scene.
+Second, we benefit from a separate network to decompose the generated textures
+into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be
+used in rendering applications. Our approach builds on existing synthetic
+material libraries with SVBRDF ground truth, but also exploits a
+diffusion-generated RGB texture dataset to allow generalization to new samples
+using unsupervised domain adaptation (UDA). Our contributions are thoroughly
+evaluated on synthetic and real-world datasets. We further demonstrate the
+applicability of our method for editing 3D scenes with materials estimated from
+real photographs. The code and models will be made open-source. Project page:
+https://astra-vision.github.io/MaterialPalette/",cs.CV,"['cs.CV', 'cs.GR']"
+DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly,Gianluca Scarpellini · Stefano Fiorini · Francesco Giuliari · Pietro Morerio · Alessio Del Bue,https://iit-pavis.github.io/DiffAssemble/,https://arxiv.org/abs/2402.19302,,2402.19302.pdf,DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly,"Reassembly tasks play a fundamental role in many fields and multiple
+approaches exist to solve specific reassembly problems. In this context, we
+posit that a general unified model can effectively address them all,
+irrespective of the input data type (images, 3D, etc.). We introduce
+DiffAssemble, a Graph Neural Network (GNN)-based architecture that learns to
+solve reassembly tasks using a diffusion model formulation. Our method treats
+the elements of a set, whether pieces of 2D patch or 3D object fragments, as
+nodes of a spatial graph. Training is performed by introducing noise into the
+position and rotation of the elements and iteratively denoising them to
+reconstruct the coherent initial pose. DiffAssemble achieves state-of-the-art
+(SOTA) results in most 2D and 3D reassembly tasks and is the first
+learning-based approach that solves 2D puzzles for both rotation and
+translation. Furthermore, we highlight its remarkable reduction in run-time,
+performing 11 times faster than the quickest optimization-based method for
+puzzle solving. Code available at https://github.com/IIT-PAVIS/DiffAssemble",cs.CV,['cs.CV']
+Situational Awareness Matters in 3D Vision Language Reasoning,Yunze Man · Liang-Yan Gui · Yu-Xiong Wang, ,https://arxiv.org/abs/2401.09340,,2401.09340.pdf,SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding,"3D vision-language grounding, which focuses on aligning language with the 3D
+physical environment, stands as a cornerstone in the development of embodied
+agents. In comparison to recent advancements in the 2D domain, grounding
+language in 3D scenes faces several significant challenges: (i) the inherent
+complexity of 3D scenes due to the diverse object configurations, their rich
+attributes, and intricate relationships; (ii) the scarcity of paired 3D
+vision-language data to support grounded learning; and (iii) the absence of a
+unified learning framework to distill knowledge from grounded 3D data. In this
+work, we aim to address these three major challenges in 3D vision-language by
+examining the potential of systematically upscaling 3D vision-language learning
+in indoor environments. We introduce the first million-scale 3D vision-language
+dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
+2.5M vision-language pairs derived from both human annotations and our scalable
+scene-graph-based generation approach. We demonstrate that this scaling allows
+for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
+for 3D vision-language learning. Through extensive experiments, we showcase the
+effectiveness of GPS by achieving state-of-the-art performance on all existing
+3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
+unveiled through zero-shot transfer experiments in the challenging 3D
+vision-language tasks. Project website: https://scene-verse.github.io.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG', 'cs.RO']"
+PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness,Anh-Quan Cao · Angela Dai · Raoul de Charette,https://astra-vision.github.io/PaSCo/,https://arxiv.org/abs/2312.02158,,2312.02158.pdf,PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness,"We propose the task of Panoptic Scene Completion (PSC) which extends the
+recently popular Semantic Scene Completion (SSC) task with instance-level
+information to produce a richer understanding of the 3D scene. Our PSC proposal
+utilizes a hybrid mask-based technique on the non-empty voxels from sparse
+multi-scale completions. Whereas the SSC literature overlooks uncertainty which
+is critical for robotics applications, we instead propose an efficient
+ensembling to estimate both voxel-wise and instance-wise uncertainties along
+PSC. This is achieved by building on a multi-input multi-output (MIMO)
+strategy, while improving performance and yielding better uncertainty for
+little additional compute. Additionally, we introduce a technique to aggregate
+permutation-invariant mask predictions. Our experiments demonstrate that our
+method surpasses all baselines in both Panoptic Scene Completion and
+uncertainty estimation on three large-scale autonomous driving datasets. Our
+code and data are available at https://astra-vision.github.io/PaSCo .",cs.CV,"['cs.CV', 'cs.AI']"
+DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models,Yukang Cao · Yan-Pei Cao · Kai Han · Ying Shan · Kwan-Yee K. Wong,https://yukangcao.github.io/DreamAvatar/,https://arxiv.org/html/2402.17292v1,,2402.17292v1.pdf,DivAvatar: Diverse 3D Avatar Generation with a Single Prompt,"Text-to-Avatar generation has recently made significant strides due to
+advancements in diffusion models. However, most existing work remains
+constrained by limited diversity, producing avatars with subtle differences in
+appearance for a given text prompt. We design DivAvatar, a novel framework that
+generates diverse avatars, empowering 3D creatives with a multitude of distinct
+and richly varied 3D avatars from a single text prompt. Different from most
+existing work that exploits scene-specific 3D representations such as NeRF,
+DivAvatar finetunes a 3D generative model (i.e., EVA3D), allowing diverse
+avatar generation from simply noise sampling in inference time. DivAvatar has
+two key designs that help achieve generation diversity and visual quality. The
+first is a noise sampling technique during training phase which is critical in
+generating diverse appearances. The second is a semantic-aware zoom mechanism
+and a novel depth loss, the former producing appearances of high textual
+fidelity by separate fine-tuning of specific body parts and the latter
+improving geometry quality greatly by smoothing the generated mesh in the
+features space. Extensive experiments show that DivAvatar is highly versatile
+in generating avatars of diverse appearances.",cs.CV,['cs.CV']
+Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities,AJ Piergiovanni · Isaac Noble · Dahun Kim · Michael Ryoo · Victor Gomes · Anelia Angelova, ,https://arxiv.org/abs/2311.05698,,2311.05698.pdf,Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities,"One of the main challenges of multimodal learning is the need to combine
+heterogeneous modalities (e.g., video, audio, text). For example, video and
+audio are obtained at much higher rates than text and are roughly aligned in
+time. They are often not synchronized with text, which comes as a global
+context, e.g., a title, or a description. Furthermore, video and audio inputs
+are of much larger volumes, and grow as the video length increases, which
+naturally requires more compute dedicated to these modalities and makes
+modeling of long-range dependencies harder.
+  We here decouple the multimodal modeling, dividing it into separate, focused
+autoregressive models, processing the inputs according to the characteristics
+of the modalities. We propose a multimodal model, called Mirasol3B, consisting
+of an autoregressive component for the time-synchronized modalities (audio and
+video), and an autoregressive component for the context modalities which are
+not necessarily aligned in time but are still sequential. To address the
+long-sequences of the video-audio inputs, we propose to further partition the
+video and audio sequences in consecutive snippets and autoregressively process
+their representations. To that end, we propose a Combiner mechanism, which
+models the audio-video information jointly within a timeframe. The Combiner
+learns to extract audio and video features from raw spatio-temporal signals,
+and then learns to fuse these features producing compact but expressive
+representations per snippet.
+  Our approach achieves the state-of-the-art on well established multimodal
+benchmarks, outperforming much larger models. It effectively addresses the high
+computational demand of media inputs by both learning compact representations,
+controlling the sequence length of the audio-video feature representations, and
+modeling their dependencies in time.",cs.CV,['cs.CV']
+Discontinuity-preserving Normal Integration with Auxiliary Edges,Hyomin Kim · Yucheol Jung · Seungyong Lee, ,https://arxiv.org/abs/2404.03138,,2404.03138.pdf,Discontinuity-preserving Normal Integration with Auxiliary Edges,"Many surface reconstruction methods incorporate normal integration, which is
+a process to obtain a depth map from surface gradients. In this process, the
+input may represent a surface with discontinuities, e.g., due to
+self-occlusion. To reconstruct an accurate depth map from the input normal map,
+hidden surface gradients occurring from the jumps must be handled. To model
+these jumps correctly, we design a novel discretization scheme for the domain
+of normal integration. Our key idea is to introduce auxiliary edges, which
+bridge between piecewise-smooth patches in the domain so that the magnitude of
+hidden jumps can be explicitly expressed. Using the auxiliary edges, we design
+a novel algorithm to optimize the discontinuity and the depth map from the
+input normal map. Our method optimizes discontinuities by using a combination
+of iterative re-weighted least squares and iterative filtering of the jump
+magnitudes on auxiliary edges to provide strong sparsity regularization.
+Compared to previous discontinuity-preserving normal integration methods, which
+model the magnitudes of jumps only implicitly, our method reconstructs subtle
+discontinuities accurately thanks to our explicit representation of jumps
+allowing for strong sparsity regularization.",cs.CV,"['cs.CV', 'cs.GR', 'I.4.5']"
+Depth  Information Assisted Collaborative Mutual Promotion Network for Single Image Dehazing,Yafei Zhang · Shen Zhou · Huafeng Li, ,https://arxiv.org/abs/2403.01105,,2403.01105.pdf,Depth Information Assisted Collaborative Mutual Promotion Network for Single Image Dehazing,"Recovering a clear image from a single hazy image is an open inverse problem.
+Although significant research progress has been made, most existing methods
+ignore the effect that downstream tasks play in promoting upstream dehazing.
+From the perspective of the haze generation mechanism, there is a potential
+relationship between the depth information of the scene and the hazy image.
+Based on this, we propose a dual-task collaborative mutual promotion framework
+to achieve the dehazing of a single image. This framework integrates depth
+estimation and dehazing by a dual-task interaction mechanism and achieves
+mutual enhancement of their performance. To realize the joint optimization of
+the two tasks, an alternative implementation mechanism with the difference
+perception is developed. On the one hand, the difference perception between the
+depth maps of the dehazing result and the ideal image is proposed to promote
+the dehazing network to pay attention to the non-ideal areas of the dehazing.
+On the other hand, by improving the depth estimation performance in the
+difficult-to-recover areas of the hazy image, the dehazing network can
+explicitly use the depth information of the hazy image to assist the clear
+image recovery. To promote the depth estimation, we propose to use the
+difference between the dehazed image and the ground truth to guide the depth
+estimation network to focus on the dehazed unideal areas. It allows dehazing
+and depth estimation to leverage their strengths in a mutually reinforcing
+manner. Experimental results show that the proposed method can achieve better
+performance than that of the state-of-the-art approaches.",cs.CV,['cs.CV']
+ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction,Zhicheng Zhang · Junyao Hu · Wentao Cheng · Danda Paudel · Jufeng Yang,https://zzcheng.top/ExtDM/,,https://junyaohu.github.io/publication/,,,,,nan
+VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis,Linshan Wu · Linshan Wu · Jia-Xin Zhuang · Hao Chen, ,https://arxiv.org/abs/2402.17300v1,,2402.17300v1.pdf,VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis,"Self-Supervised Learning (SSL) has demonstrated promising results in 3D
+medical image analysis. However, the lack of high-level semantics in
+pre-training still heavily hinders the performance of downstream tasks. We
+observe that 3D medical images contain relatively consistent contextual
+position information, i.e., consistent geometric relations between different
+organs, which leads to a potential way for us to learn consistent semantic
+representations in pre-training. In this paper, we propose a
+simple-yet-effective Volume Contrast (VoCo) framework to leverage the
+contextual position priors for pre-training. Specifically, we first generate a
+group of base crops from different regions while enforcing feature discrepancy
+among them, where we employ them as class assignments of different regions.
+Then, we randomly crop sub-volumes and predict them belonging to which class
+(located at which region) by contrasting their similarity to different base
+crops, which can be seen as predicting contextual positions of different
+sub-volumes. Through this pretext task, VoCo implicitly encodes the contextual
+position priors into model representations without the guidance of annotations,
+enabling us to effectively improve the performance of downstream tasks that
+require high-level semantics. Extensive experimental results on six downstream
+tasks demonstrate the superior effectiveness of VoCo. Code will be available at
+https://github.com/Luffy03/VoCo.",eess.IV,['eess.IV']
+JoAPR: Cleaning the Lens of Prompt Learning for Vision-Language Models,YUNCHENG GUO · Xiaodong Gu, ,https://arxiv.org/abs/2312.01564,,2312.01564.pdf,APoLLo: Unified Adapter and Prompt Learning for Vision Language Models,"The choice of input text prompt plays a critical role in the performance of
+Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a
+unified multi-modal approach that combines Adapter and Prompt learning for
+Vision-Language models. Our method is designed to substantially improve the
+generalization capabilities of VLP models when they are fine-tuned in a
+few-shot setting. We introduce trainable cross-attention-based adapter layers
+in conjunction with vision and language encoders to strengthen the alignment
+between the two modalities. We enforce consistency between the respective
+encoder branches (receiving augmented inputs) to prevent overfitting in
+downstream tasks. Our method is evaluated on three representative tasks:
+generalization to novel classes, cross-dataset evaluation, and unseen domain
+shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe
+(SOTA) on novel classes for 10 diverse image recognition datasets.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CL', 'cs.CV']"
+F$^3$Loc: Fusion and Filtering for Floorplan Localization,Changan Chen · Rui Wang · Christoph Vogel · Marc Pollefeys, ,https://arxiv.org/abs/2403.03370,,2403.03370.pdf,F$^3$Loc: Fusion and Filtering for Floorplan Localization,"In this paper we propose an efficient data-driven solution to
+self-localization within a floorplan. Floorplan data is readily available,
+long-term persistent and inherently robust to changes in the visual appearance.
+Our method does not require retraining per map and location or demand a large
+database of images of the area of interest. We propose a novel probabilistic
+model consisting of an observation and a novel temporal filtering module.
+Operating internally with an efficient ray-based representation, the
+observation module consists of a single and a multiview module to predict
+horizontal depth from images and fuses their results to benefit from advantages
+offered by either methodology. Our method operates on conventional consumer
+hardware and overcomes a common limitation of competing methods that often
+demand upright images. Our full system meets real-time requirements, while
+outperforming the state-of-the-art by a significant margin.",cs.CV,"['cs.CV', 'cs.RO']"
+Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance,Phuc Nguyen · Tuan Duc Ngo · Evangelos Kalogerakis · Chuang Gan · Anh Tran · Cuong Pham · Khoi Nguyen,https://open3dis.github.io/,https://arxiv.org/abs/2312.10671,,2312.10671.pdf,Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance,"We introduce Open3DIS, a novel solution designed to tackle the problem of
+Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D
+environments exhibit diverse shapes, scales, and colors, making precise
+instance-level identification a challenging task. Recent advancements in
+Open-Vocabulary scene understanding have made significant strides in this area
+by employing class-agnostic 3D instance proposal networks for object
+localization and learning queryable features for each 3D mask. While these
+methods produce high-quality instance proposals, they struggle with identifying
+small-scale and geometrically ambiguous objects. The key idea of our method is
+a new module that aggregates 2D instance masks across frames and maps them to
+geometrically coherent point cloud regions as high-quality object proposals
+addressing the above limitations. These are then combined with 3D
+class-agnostic instance proposals to include a wide range of objects in the
+real world. To validate our approach, we conducted experiments on three
+prominent datasets, including ScanNet200, S3DIS, and Replica, demonstrating
+significant performance gains in segmenting objects with diverse categories
+over the state-of-the-art approaches.",cs.CV,['cs.CV']
+Binarized Low-light Raw Video Enhancement,Gengchen Zhang · Yulun Zhang · Xin Yuan · Ying Fu, ,https://arxiv.org/abs/2403.19944,,2403.19944.pdf,Binarized Low-light Raw Video Enhancement,"Recently, deep neural networks have achieved excellent performance on
+low-light raw video enhancement. However, they often come with high
+computational complexity and large memory costs, which hinder their
+applications on resource-limited devices. In this paper, we explore the
+feasibility of applying the extremely compact binary neural network (BNN) to
+low-light raw video enhancement. Nevertheless, there are two main issues with
+binarizing video enhancement models. One is how to fuse the temporal
+information to improve low-light denoising without complex modules. The other
+is how to narrow the performance gap between binary convolutions with the full
+precision ones. To address the first issue, we introduce a spatial-temporal
+shift operation, which is easy-to-binarize and effective. The temporal shift
+efficiently aggregates the features of neighbor frames and the spatial shift
+handles the misalignment caused by the large motion in videos. For the second
+issue, we present a distribution-aware binary convolution, which captures the
+distribution characteristics of real-valued input and incorporates them into
+plain binary convolutions to alleviate the degradation in performance.
+Extensive quantitative and qualitative experiments have shown our
+high-efficiency binarized low-light raw video enhancement method can attain a
+promising performance.",cs.CV,"['cs.CV', 'eess.IV']"
+Generating Non-Stationary Textures using Self-Rectification,Yang Zhou · Rongjun Xiao · Dani Lischinski · Daniel Cohen-Or · Hui Huang,https://vcc.tech/research/2024/TexRec,https://arxiv.org/abs/2401.02847,,2401.02847.pdf,Generating Non-Stationary Textures using Self-Rectification,"This paper addresses the challenge of example-based non-stationary texture
+synthesis. We introduce a novel twostep approach wherein users first modify a
+reference texture using standard image editing tools, yielding an initial rough
+target for the synthesis. Subsequently, our proposed method, termed
+""self-rectification"", automatically refines this target into a coherent,
+seamless texture, while faithfully preserving the distinct visual
+characteristics of the reference exemplar. Our method leverages a pre-trained
+diffusion network, and uses self-attention mechanisms, to gradually align the
+synthesized texture with the reference, ensuring the retention of the
+structures in the provided target. Through experimental validation, our
+approach exhibits exceptional proficiency in handling non-stationary textures,
+demonstrating significant advancements in texture synthesis when compared to
+existing state-of-the-art techniques. Code is available at
+https://github.com/xiaorongjun000/Self-Rectification",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering,Tao Hu · Fangzhou Hong · Ziwei Liu, ,https://arxiv.org/abs/2404.01225,,2404.01225.pdf,SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering,"Dynamic human rendering from video sequences has achieved remarkable progress
+by formulating the rendering as a mapping from static poses to human images.
+However, existing methods focus on the human appearance reconstruction of every
+single frame while the temporal motion relations are not fully explored. In
+this paper, we propose a new 4D motion modeling paradigm, SurMo, that jointly
+models the temporal dynamics and human appearances in a unified framework with
+three key designs: 1) Surface-based motion encoding that models 4D human
+motions with an efficient compact surface-based triplane. It encodes both
+spatial and temporal motion relations on the dense surface manifold of a
+statistical body template, which inherits body topology priors for
+generalizable novel view synthesis with sparse training observations. 2)
+Physical motion decoding that is designed to encourage physical motion learning
+by decoding the motion triplane features at timestep t to predict both spatial
+derivatives and temporal derivatives at the next timestep t+1 in the training
+stage. 3) 4D appearance decoding that renders the motion triplanes into images
+by an efficient volumetric surface-conditioned renderer that focuses on the
+rendering of body surfaces with motion learning conditioning. Extensive
+experiments validate the state-of-the-art performance of our new paradigm and
+illustrate the expressiveness of surface-based motion triplanes for rendering
+high-fidelity view-consistent humans with fast motions and even
+motion-dependent shadows. Our project page is at:
+https://taohuumd.github.io/projects/SurMo/",cs.CV,['cs.CV']
+MultiDiff: Consistent Novel View Synthesis from a Single Image,Norman Müller · Katja Schwarz · Katja Schwarz · Barbara Roessle · Lorenzo Porzi · Samuel Rota Bulò · Matthias Nießner · Peter Kontschieder, ,,https://sirwyver.github.io/publications/,,,,,nan
+Vector Graphics Generation via Mutually Impulsed Dual-domain Diffusion,Zhongyin Zhao · Ye Chen · Zhangli Hu · Xuanhong Chen · Bingbing Ni, ,https://arxiv.org/abs/2312.10540,,,VecFusion: Vector Font Generation with Diffusion,"We present VecFusion, a new neural architecture that can generate vector
+fonts with varying topological structures and precise control point positions.
+Our approach is a cascaded diffusion model which consists of a raster diffusion
+model followed by a vector diffusion model. The raster model generates
+low-resolution, rasterized fonts with auxiliary control point information,
+capturing the global style and shape of the font, while the vector model
+synthesizes vector fonts conditioned on the low-resolution raster fonts from
+the first stage. To synthesize long and complex curves, our vector diffusion
+model uses a transformer architecture and a novel vector representation that
+enables the modeling of diverse vector geometry and the precise prediction of
+control points. Our experiments show that, in contrast to previous generative
+models for vector graphics, our new cascaded vector diffusion model generates
+higher quality vector fonts, with complex structures and diverse styles.",cs.CV,"['cs.CV', 'cs.GR']"
+Equivariant plug-and-play image reconstruction,Matthieu Terris · Thomas Moreau · Nelly Pustelnik · Julián Tachella, ,https://arxiv.org/html/2312.01831v2,,2312.01831v2.pdf,Equivariant plug-and-play image reconstruction,"Plug-and-play algorithms constitute a popular framework for solving inverse
+imaging problems that rely on the implicit definition of an image prior via a
+denoiser. These algorithms can leverage powerful pre-trained denoisers to solve
+a wide range of imaging tasks, circumventing the necessity to train models on a
+per-task basis. Unfortunately, plug-and-play methods often show unstable
+behaviors, hampering their promise of versatility and leading to suboptimal
+quality of reconstructed images. In this work, we show that enforcing
+equivariance to certain groups of transformations (rotations, reflections,
+and/or translations) on the denoiser strongly improves the stability of the
+algorithm as well as its reconstruction quality. We provide a theoretical
+analysis that illustrates the role of equivariance on better performance and
+stability. We present a simple algorithm that enforces equivariance on any
+existing denoiser by simply applying a random transformation to the input of
+the denoiser and the inverse transformation to the output at each iteration of
+the algorithm. Experiments on multiple imaging modalities and denoising
+networks show that the equivariant plug-and-play algorithm improves both the
+reconstruction performance and the stability compared to their non-equivariant
+counterparts.",eess.IV,"['eess.IV', 'cs.CV']"
+"SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM",Nikhil Keetha · Jay Karhade · Krishna Murthy Jatavallabhula · Gengshan Yang · Sebastian Scherer · Deva Ramanan · Jonathon Luiten,https://spla-tam.github.io/,https://arxiv.org/abs/2312.02126,,2312.02126.pdf,"SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM","Dense simultaneous localization and mapping (SLAM) is crucial for robotics
+and augmented reality applications. However, current methods are often hampered
+by the non-volumetric or implicit way they represent a scene. This work
+introduces SplaTAM, an approach that, for the first time, leverages explicit
+volumetric representations, i.e., 3D Gaussians, to enable high-fidelity
+reconstruction from a single unposed RGB-D camera, surpassing the capabilities
+of existing methods. SplaTAM employs a simple online tracking and mapping
+system tailored to the underlying Gaussian representation. It utilizes a
+silhouette mask to elegantly capture the presence of scene density. This
+combination enables several benefits over prior representations, including fast
+rendering and dense optimization, quickly determining if areas have been
+previously mapped, and structured map expansion by adding more Gaussians.
+Extensive experiments show that SplaTAM achieves up to 2x superior performance
+in camera pose estimation, map construction, and novel-view synthesis over
+existing methods, paving the way for more immersive high-fidelity SLAM
+applications.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring,Chengxu Liu · Xuan Wang · Xiangyu Xu · Ruhao Tian · Shuai Li · Xueming Qian · Ming-Hsuan Yang, ,https://arxiv.org/abs/2404.13153,,2404.13153.pdf,Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring,"Eliminating image blur produced by various kinds of motion has been a
+challenging problem. Dominant approaches rely heavily on model capacity to
+remove blurring by reconstructing residual from blurry observation in feature
+space. These practices not only prevent the capture of spatially variable
+motion in the real world but also ignore the tailored handling of various
+motions in image space. In this paper, we propose a novel real-world deblurring
+filtering model called the Motion-adaptive Separable Collaborative (MISC)
+Filter. In particular, we use a motion estimation network to capture motion
+information from neighborhoods, thereby adaptively estimating spatially-variant
+motion flow, mask, kernels, weights, and offsets to obtain the MISC Filter. The
+MISC Filter first aligns the motion-induced blurring patterns to the motion
+middle along the predicted flow direction, and then collaboratively filters the
+aligned image through the predicted kernels, weights, and offsets to generate
+the output. This design can handle more generalized and complex motion in a
+spatially differentiated manner. Furthermore, we analyze the relationships
+between the motion estimation network and the residual reconstruction network.
+Extensive experiments on four widely used benchmarks demonstrate that our
+method provides an effective solution for real-world motion blur removal and
+achieves state-of-the-art performance. Code is available at
+https://github.com/ChengxuLiu/MISCFilter",eess.IV,"['eess.IV', 'cs.CV']"
+BoQ: A Place is Worth a Bag of Learnable Queries,Amar Ali-bey · Brahim Chaib-draa · Philippe Giguère, ,https://arxiv.org/abs/2405.07364,,2405.07364.pdf,BoQ: A Place is Worth a Bag of Learnable Queries,"In visual place recognition, accurately identifying and matching images of
+locations under varying environmental conditions and viewpoints remains a
+significant challenge. In this paper, we introduce a new technique, called
+Bag-of-Queries (BoQ), which learns a set of global queries designed to capture
+universal place-specific attributes. Unlike existing methods that employ
+self-attention and generate the queries directly from the input features, BoQ
+employs distinct learnable global queries, which probe the input features via
+cross-attention, ensuring consistent information aggregation. In addition, our
+technique provides an interpretable attention mechanism and integrates with
+both CNN and Vision Transformer backbones. The performance of BoQ is
+demonstrated through extensive experiments on 14 large-scale benchmarks. It
+consistently outperforms current state-of-the-art techniques including NetVLAD,
+MixVPR and EigenPlaces. Moreover, as a global retrieval technique (one-stage),
+BoQ surpasses two-stage retrieval methods, such as Patch-NetVLAD, TransVPR and
+R2Former, all while being orders of magnitude faster and more efficient. The
+code and model weights are publicly available at
+https://github.com/amaralibey/Bag-of-Queries.",cs.CV,['cs.CV']
+Deformable One-shot Face Stylization via DINO Semantic Guidance,Yang Zhou · Zichong Chen · Hui Huang,https://vcc.tech/research/2024/DoesFS,https://arxiv.org/abs/2403.00459,,2403.00459.pdf,Deformable One-shot Face Stylization via DINO Semantic Guidance,"This paper addresses the complex issue of one-shot face stylization, focusing
+on the simultaneous consideration of appearance and structure, where previous
+methods have fallen short. We explore deformation-aware face stylization that
+diverges from traditional single-image style reference, opting for a real-style
+image pair instead. The cornerstone of our method is the utilization of a
+self-supervised vision transformer, specifically DINO-ViT, to establish a
+robust and consistent facial structure representation across both real and
+style domains. Our stylization process begins by adapting the StyleGAN
+generator to be deformation-aware through the integration of spatial
+transformers (STN). We then introduce two innovative constraints for generator
+fine-tuning under the guidance of DINO semantics: i) a directional deformation
+loss that regulates directional vectors in DINO space, and ii) a relative
+structural consistency constraint based on DINO token self-similarities,
+ensuring diverse generation. Additionally, style-mixing is employed to align
+the color generation with the reference, minimizing inconsistent
+correspondences. This framework delivers enhanced deformability for general
+one-shot face stylization, achieving notable efficiency with a fine-tuning
+duration of approximately 10 minutes. Extensive qualitative and quantitative
+comparisons demonstrate our superiority over state-of-the-art one-shot face
+stylization methods. Code is available at https://github.com/zichongc/DoesFS",cs.CV,['cs.CV']
+Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion,Yuanxun Lu · Jingyang Zhang · Shiwei Li · Tian Fang · David McKinnon · Yanghai Tsin · Long Quan · Xun Cao · Yao Yao,https://nju-3dv.github.io/projects/direct25/,https://arxiv.org/abs/2311.15980,,2311.15980.pdf,Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion,"Recent advances in generative AI have unveiled significant potential for the
+creation of 3D content. However, current methods either apply a pre-trained 2D
+diffusion model with the time-consuming score distillation sampling (SDS), or a
+direct 3D diffusion model trained on limited 3D data losing generation
+diversity. In this work, we approach the problem by employing a multi-view 2.5D
+diffusion fine-tuned from a pre-trained 2D diffusion model. The multi-view 2.5D
+diffusion directly models the structural distribution of 3D data, while still
+maintaining the strong generalization ability of the original 2D diffusion
+model, filling the gap between 2D diffusion-based and direct 3D diffusion-based
+methods for 3D content generation. During inference, multi-view normal maps are
+generated using the 2.5D diffusion, and a novel differentiable rasterization
+scheme is introduced to fuse the almost consistent multi-view normal maps into
+a consistent 3D model. We further design a normal-conditioned multi-view image
+generation module for fast appearance generation given the 3D geometry. Our
+method is a one-pass diffusion process and does not require any SDS
+optimization as post-processing. We demonstrate through extensive experiments
+that, our direct 2.5D generation with the specially-designed fusion scheme can
+achieve diverse, mode-seeking-free, and high-fidelity 3D content generation in
+only 10 seconds. Project page: https://nju-3dv.github.io/projects/direct25.",cs.CV,['cs.CV']
+Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation,Ba Hung Ngo · Nhat-Tuong Do-Tran · Tuan-Ngoc Nguyen · Hae-Gon Jeon · Tae Jong Choi,https://dotrannhattuong.github.io/ECB/website/,https://arxiv.org/abs/2403.18360,,2403.18360.pdf,Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation,"Most domain adaptation (DA) methods are based on either a convolutional
+neural networks (CNNs) or a vision transformers (ViTs). They align the
+distribution differences between domains as encoders without considering their
+unique characteristics. For instance, ViT excels in accuracy due to its
+superior ability to capture global representations, while CNN has an advantage
+in capturing local representations. This fact has led us to design a hybrid
+method to fully take advantage of both ViT and CNN, called Explicitly
+Class-specific Boundaries (ECB). ECB learns CNN on ViT to combine their
+distinct strengths. In particular, we leverage ViT's properties to explicitly
+find class-specific decision boundaries by maximizing the discrepancy between
+the outputs of the two classifiers to detect target samples far from the source
+support. In contrast, the CNN encoder clusters target features based on the
+previously defined class-specific boundaries by minimizing the discrepancy
+between the probabilities of the two classifiers. Finally, ViT and CNN mutually
+exchange knowledge to improve the quality of pseudo labels and reduce the
+knowledge discrepancies of these models. Compared to conventional DA methods,
+our ECB achieves superior performance, which verifies its effectiveness in this
+hybrid model. The project website can be found
+https://dotrannhattuong.github.io/ECB/website.",cs.CV,['cs.CV']
+Versatile Navigation under Partial Observability via Value-Guided Diffusion Policy,Gengyu Zhang · Hao Tang · Yan Yan, ,https://arxiv.org/abs/2404.02176,,2404.02176.pdf,Versatile Navigation under Partial Observability via Value-guided Diffusion Policy,"Route planning for navigation under partial observability plays a crucial
+role in modern robotics and autonomous driving. Existing route planning
+approaches can be categorized into two main classes: traditional autoregressive
+and diffusion-based methods. The former often fails due to its myopic nature,
+while the latter either assumes full observability or struggles to adapt to
+unfamiliar scenarios, due to strong couplings with behavior cloning from
+experts. To address these deficiencies, we propose a versatile diffusion-based
+approach for both 2D and 3D route planning under partial observability.
+Specifically, our value-guided diffusion policy first generates plans to
+predict actions across various timesteps, providing ample foresight to the
+planning. It then employs a differentiable planner with state estimations to
+derive a value function, directing the agent's exploration and goal-seeking
+behaviors without seeking experts while explicitly addressing partial
+observability. During inference, our policy is further enhanced by a
+best-plan-selection strategy, substantially boosting the planning success rate.
+Moreover, we propose projecting point clouds, derived from RGB-D inputs, onto
+2D grid-based bird-eye-view maps via semantic segmentation, generalizing to 3D
+environments. This simple yet effective adaption enables zero-shot transfer
+from 2D-trained policy to 3D, cutting across the laborious training for 3D
+policy, and thus certifying our versatility. Experimental results demonstrate
+our superior performance, particularly in navigating situations beyond expert
+demonstrations, surpassing state-of-the-art autoregressive and diffusion-based
+baselines for both 2D and 3D scenarios.",cs.RO,"['cs.RO', 'cs.AI']"
+PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees,Chulin Xie · De-An Huang · Wenda Chu · Daguang Xu · Chaowei Xiao · Bo Li · Anima Anandkumar, ,https://arxiv.org/abs/2405.09771,,2405.09771.pdf,Harmonizing Generalization and Personalization in Federated Prompt Learning,"Federated Prompt Learning (FPL) incorporates large pre-trained
+Vision-Language models (VLM) into federated learning through prompt tuning. The
+transferable representations and remarkable generalization capacity of VLM make
+them highly compatible with the integration of federated learning. Addressing
+data heterogeneity in federated learning requires personalization, but
+excessive focus on it across clients could compromise the model's ability to
+generalize effectively. To preserve the impressive generalization capability of
+VLM, it is crucial to strike a balance between personalization and
+generalization in FPL. To tackle this challenge, we proposed Federated Prompt
+Learning with CLIP Generalization and low-rank Personalization (FedPGP), which
+employs pre-trained CLIP to provide knowledge-guidance on the global prompt for
+improved generalization and incorporates a low-rank adaptation term to
+personalize the global prompt. Further, FedPGP integrates a prompt-wise
+contrastive loss to achieve knowledge guidance and personalized adaptation
+simultaneously, enabling a harmonious balance between personalization and
+generalization in FPL. We conduct extensive experiments on various datasets to
+explore base-to-novel generalization in both category-level and domain-level
+scenarios with heterogeneous data, showing the superiority of FedPGP in
+balancing generalization and personalization.",cs.LG,['cs.LG']
+Bi-Causal: Group Activity Recognition via Bidirectional Causality,Youliang Zhang · Wenxuan Liu · danni xu · Zhuo Zhou · Zheng Wang, ,https://arxiv.org/html/2312.00404v1,,2312.00404v1.pdf,A Causality-Aware Pattern Mining Scheme for Group Activity Recognition in a Pervasive Sensor Space,"Human activity recognition (HAR) is a key challenge in pervasive computing
+and its solutions have been presented based on various disciplines.
+Specifically, for HAR in a smart space without privacy and accessibility
+issues, data streams generated by deployed pervasive sensors are leveraged. In
+this paper, we focus on a group activity by which a group of users perform a
+collaborative task without user identification and propose an efficient group
+activity recognition scheme which extracts causality patterns from pervasive
+sensor event sequences generated by a group of users to support as good
+recognition accuracy as the state-of-the-art graphical model. To filter out
+irrelevant noise events from a given data stream, a set of rules is leveraged
+to highlight causally related events. Then, a pattern-tree algorithm extracts
+frequent causal patterns by means of a growing tree structure. Based on the
+extracted patterns, a weighted sum-based pattern matching algorithm computes
+the likelihoods of stored group activities to the given test event sequence by
+means of matched event pattern counts for group activity recognition. We
+evaluate the proposed scheme using the data collected from our testbed and
+CASAS datasets where users perform their tasks on a daily basis and validate
+its effectiveness in a real environment. Experiment results show that the
+proposed scheme performs higher recognition accuracy and with a small amount of
+runtime overhead than the existing schemes.",cs.LG,"['cs.LG', 'cs.DB']"
+MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning,Yixin Liu · Chenrui Fan · Yutong Dai · Xun Chen · Pan Zhou · Lichao Sun, ,https://arxiv.org/abs/2311.13127v3,,2311.13127v3.pdf,MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning,"Text-to-image diffusion models allow seamless generation of personalized
+images from scant reference photos. Yet, these tools, in the wrong hands, can
+fabricate misleading or harmful content, endangering individuals. To address
+this problem, existing poisoning-based approaches perturb user images in an
+imperceptible way to render them ""unlearnable"" from malicious uses. We identify
+two limitations of these defending approaches: i) sub-optimal due to the
+hand-crafted heuristics for solving the intractable bilevel optimization and
+ii) lack of robustness against simple data transformations like Gaussian
+filtering. To solve these challenges, we propose MetaCloak, which solves the
+bi-level poisoning problem with a meta-learning framework with an additional
+transformation sampling process to craft transferable and robust perturbation.
+Specifically, we employ a pool of surrogate diffusion models to craft
+transferable and model-agnostic perturbation. Furthermore, by incorporating an
+additional transformation process, we design a simple denoising-error
+maximization loss that is sufficient for causing transformation-robust semantic
+distortion and degradation in a personalized generation. Extensive experiments
+on the VGGFace2 and CelebA-HQ datasets show that MetaCloak outperforms existing
+approaches. Notably, MetaCloak can successfully fool online training services
+like Replicate, in a black-box manner, demonstrating the effectiveness of
+MetaCloak in real-world scenarios. Our code is available at
+https://github.com/liuyixin-louis/MetaCloak.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CR']"
+BIVDiff: A Training-free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models,Fengyuan Shi · Jiaxi Gu · Hang Xu · Songcen Xu · Wei Zhang · Limin Wang,https://github.com/MCG-NJU/BIVDiff,https://arxiv.org/abs/2312.02813,,2312.02813.pdf,BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models,"Diffusion models have made tremendous progress in text-driven image and video
+generation. Now text-to-image foundation models are widely applied to various
+downstream image synthesis tasks, such as controllable image generation and
+image editing, while downstream video synthesis tasks are less explored for
+several reasons. First, it requires huge memory and computation overhead to
+train a video generation foundation model. Even with video foundation models,
+additional costly training is still required for downstream video synthesis
+tasks. Second, although some works extend image diffusion models into videos in
+a training-free manner, temporal consistency cannot be well preserved. Finally,
+these adaption methods are specifically designed for one task and fail to
+generalize to different tasks. To mitigate these issues, we propose a
+training-free general-purpose video synthesis framework, coined as {\bf
+BIVDiff}, via bridging specific image diffusion models and general
+text-to-video foundation diffusion models. Specifically, we first use a
+specific image diffusion model (e.g., ControlNet and Instruct Pix2Pix) for
+frame-wise video generation, then perform Mixed Inversion on the generated
+video, and finally input the inverted latents into the video diffusion models
+(e.g., VidRD and ZeroScope) for temporal smoothing. This decoupled framework
+enables flexible image model selection for different purposes with strong task
+generalization and high efficiency. To validate the effectiveness and general
+use of BIVDiff, we perform a wide range of video synthesis tasks, including
+controllable video generation, video editing, video inpainting, and
+outpainting.",cs.CV,"['cs.CV', 'cs.AI']"
+A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames,Pinelopi Papalampidi · Skanda Koppula · Shreya Pathak · Justin Chiu · Joseph Heyward · Viorica Patraucean · Jiajun Shen · Antoine Miech · Andrew Zisserman · Aida Nematzadeh, ,https://arxiv.org/abs/2312.07395,,2312.07395.pdf,A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames,"Understanding long, real-world videos requires modeling of long-range visual
+dependencies. To this end, we explore video-first architectures, building on
+the common paradigm of transferring large-scale, image--text models to video
+via shallow temporal fusion. However, we expose two limitations to the
+approach: (1) decreased spatial capabilities, likely due to poor
+video--language alignment in standard video datasets, and (2) higher memory
+consumption, bottlenecking the number of frames that can be processed. To
+mitigate the memory bottleneck, we systematically analyze the memory/accuracy
+trade-off of various efficient methods: factorized attention,
+parameter-efficient image-to-video adaptation, input masking, and
+multi-resolution patchification. Surprisingly, simply masking large portions of
+the video (up to 75%) during contrastive pre-training proves to be one of the
+most robust ways to scale encoders to videos up to 4.3 minutes at 1 FPS. Our
+simple approach for training long video-to-text models, which scales to 1B
+parameters, does not add new architectural complexity and is able to outperform
+the popular paradigm of using much larger LLMs as an information aggregator
+over segment-based information on benchmarks with long-range temporal
+dependencies (YouCook2, EgoSchema).",cs.CV,"['cs.CV', 'cs.CL']"
+DaReNeRF: Direction-aware Representation for Dynamic Scenes,Ange Lou · Benjamin Planche · Zhongpai Gao · Yamin Li · Tianyu Luan · Hao Ding · Terrence Chen · Jack Noble · Ziyan Wu, ,https://arxiv.org/abs/2403.02265v1,,2403.02265v1.pdf,DaReNeRF: Direction-aware Representation for Dynamic Scenes,"Addressing the intricate challenge of modeling and re-rendering dynamic
+scenes, most recent approaches have sought to simplify these complexities using
+plane-based explicit representations, overcoming the slow training time issues
+associated with methods like Neural Radiance Fields (NeRF) and implicit
+representations. However, the straightforward decomposition of 4D dynamic
+scenes into multiple 2D plane-based representations proves insufficient for
+re-rendering high-fidelity scenes with complex motions. In response, we present
+a novel direction-aware representation (DaRe) approach that captures scene
+dynamics from six different directions. This learned representation undergoes
+an inverse dual-tree complex wavelet transformation (DTCWT) to recover
+plane-based information. DaReNeRF computes features for each space-time point
+by fusing vectors from these recovered planes. Combining DaReNeRF with a tiny
+MLP for color regression and leveraging volume rendering in training yield
+state-of-the-art performance in novel view synthesis for complex dynamic
+scenes. Notably, to address redundancy introduced by the six real and six
+imaginary direction-aware wavelet coefficients, we introduce a trainable
+masking approach, mitigating storage issues without significant performance
+decline. Moreover, DaReNeRF maintains a 2x reduction in training time compared
+to prior art while delivering superior performance.",cs.CV,"['cs.CV', 'cs.GR']"
+Learning without Exact Guidance: Updating Large-scale High-resolution Land Cover Maps from Low-resolution Historical Labels,Zhuohong Li · Wei He · Jiepan Li · Fangxiao Lu · Hongyan Zhang, ,https://arxiv.org/abs/2403.02746,,2403.02746.pdf,Learning without Exact Guidance: Updating Large-scale High-resolution Land Cover Maps from Low-resolution Historical Labels,"Large-scale high-resolution (HR) land-cover mapping is a vital task to survey
+the Earth's surface and resolve many challenges facing humanity. However, it is
+still a non-trivial task hindered by complex ground details, various landforms,
+and the scarcity of accurate training labels over a wide-span geographic area.
+In this paper, we propose an efficient, weakly supervised framework
+(Paraformer) to guide large-scale HR land-cover mapping with easy-access
+historical land-cover data of low resolution (LR). Specifically, existing
+land-cover mapping approaches reveal the dominance of CNNs in preserving local
+ground details but still suffer from insufficient global modeling in various
+landforms. Therefore, we design a parallel CNN-Transformer feature extractor in
+Paraformer, consisting of a downsampling-free CNN branch and a Transformer
+branch, to jointly capture local and global contextual information. Besides,
+facing the spatial mismatch of training data, a pseudo-label-assisted training
+(PLAT) module is adopted to reasonably refine LR labels for weakly supervised
+semantic segmentation of HR images. Experiments on two large-scale datasets
+demonstrate the superiority of Paraformer over other state-of-the-art methods
+for automatically updating HR land-cover maps from LR historical labels.",cs.CV,"['cs.CV', 'cs.LG']"
+SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks,Yaxu Xie · Alain Pagani · Didier Stricker, ,https://arxiv.org/abs/2403.19474,,2403.19474.pdf,SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks,"Scene graphs have been recently introduced into 3D spatial understanding as a
+comprehensive representation of the scene. The alignment between 3D scene
+graphs is the first step of many downstream tasks such as scene graph aided
+point cloud registration, mosaicking, overlap checking, and robot navigation.
+In this work, we treat 3D scene graph alignment as a partial graph-matching
+problem and propose to solve it with a graph neural network. We reuse the
+geometric features learned by a point cloud registration method and associate
+the clustered point-level geometric features with the node-level semantic
+feature via our designed feature fusion module. Partial matching is enabled by
+using a learnable method to select the top-k similar node pairs. Subsequent
+downstream tasks such as point cloud registration are achieved by running a
+pre-trained registration network within the matched regions. We further propose
+a point-matching rescoring method, that uses the node-wise alignment of the 3D
+scene graph to reweight the matching candidates from a pre-trained point cloud
+registration method. It reduces the false point correspondences estimated
+especially in low-overlapping cases. Experiments show that our method improves
+the alignment accuracy by 10~20% in low-overlap and random transformation
+scenarios and outperforms the existing work in multiple downstream tasks.",cs.CV,"['cs.CV', 'cs.RO']"
+Frequency-Adaptive Dilated Convolution for Semantic Segmentation,Linwei Chen · Lin Gu · Dezhi Zheng · Ying Fu,https://github.com/Linwei-Chen/FADC,https://arxiv.org/abs/2403.05369,,2403.05369.pdf,Frequency-Adaptive Dilated Convolution for Semantic Segmentation,"Dilated convolution, which expands the receptive field by inserting gaps
+between its consecutive elements, is widely employed in computer vision. In
+this study, we propose three strategies to improve individual phases of dilated
+convolution from the view of spectrum analysis. Departing from the conventional
+practice of fixing a global dilation rate as a hyperparameter, we introduce
+Frequency-Adaptive Dilated Convolution (FADC), which dynamically adjusts
+dilation rates spatially based on local frequency components. Subsequently, we
+design two plug-in modules to directly enhance effective bandwidth and
+receptive field size. The Adaptive Kernel (AdaKern) module decomposes
+convolution weights into low-frequency and high-frequency components,
+dynamically adjusting the ratio between these components on a per-channel
+basis. By increasing the high-frequency part of convolution weights, AdaKern
+captures more high-frequency components, thereby improving effective bandwidth.
+The Frequency Selection (FreqSelect) module optimally balances high- and
+low-frequency components in feature representations through spatially variant
+reweighting. It suppresses high frequencies in the background to encourage FADC
+to learn a larger dilation, thereby increasing the receptive field for an
+expanded scope. Extensive experiments on segmentation and object detection
+consistently validate the efficacy of our approach. The code is publicly
+available at https://github.com/Linwei-Chen/FADC.",cs.CV,['cs.CV']
+Distilled Datamodel with Reverse Gradient Matching,Jingwen Ye · Ruonan Yu · Songhua Liu · Xinchao Wang, ,https://arxiv.org/abs/2404.14006,,2404.14006.pdf,Distilled Datamodel with Reverse Gradient Matching,"The proliferation of large-scale AI models trained on extensive datasets has
+revolutionized machine learning. With these models taking on increasingly
+central roles in various applications, the need to understand their behavior
+and enhance interpretability has become paramount. To investigate the impact of
+changes in training data on a pre-trained model, a common approach is
+leave-one-out retraining. This entails systematically altering the training
+dataset by removing specific samples to observe resulting changes within the
+model. However, retraining the model for each altered dataset presents a
+significant computational challenge, given the need to perform this operation
+for every dataset variation. In this paper, we introduce an efficient framework
+for assessing data impact, comprising offline training and online evaluation
+stages. During the offline training phase, we approximate the influence of
+training data on the target model through a distilled synset, formulated as a
+reversed gradient matching problem. For online evaluation, we expedite the
+leave-one-out process using the synset, which is then utilized to compute the
+attribution matrix based on the evaluation objective. Experimental evaluations,
+including training data attribution and assessments of data quality,
+demonstrate that our proposed method achieves comparable model behavior
+evaluation while significantly speeding up the process compared to the direct
+retraining method.",cs.LG,"['cs.LG', 'cs.CV']"
+Memory-based Adapters for Online 3D Scene Perception,Xiuwei Xu · Chong Xia · Ziwei Wang · Linqing Zhao · Linqing Zhao · Yueqi Duan · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2403.06974,,2403.06974.pdf,Memory-based Adapters for Online 3D Scene Perception,"In this paper, we propose a new framework for online 3D scene perception.
+Conventional 3D scene perception methods are offline, i.e., take an already
+reconstructed 3D scene geometry as input, which is not applicable in robotic
+applications where the input data is streaming RGB-D videos rather than a
+complete 3D scene reconstructed from pre-collected RGB-D videos. To deal with
+online 3D scene perception tasks where data collection and perception should be
+performed simultaneously, the model should be able to process 3D scenes frame
+by frame and make use of the temporal information. To this end, we propose an
+adapter-based plug-and-play module for the backbone of 3D scene perception
+model, which constructs memory to cache and aggregate the extracted RGB-D
+features to empower offline models with temporal learning ability.
+Specifically, we propose a queued memory mechanism to cache the supporting
+point cloud and image features. Then we devise aggregation modules which
+directly perform on the memory and pass temporal information to current frame.
+We further propose 3D-to-2D adapter to enhance image features with strong
+global context. Our adapters can be easily inserted into mainstream offline
+architectures of different tasks and significantly boost their performance on
+online tasks. Extensive experiments on ScanNet and SceneNN datasets demonstrate
+our approach achieves leading performance on three 3D scene perception tasks
+compared with state-of-the-art online methods by simply finetuning existing
+offline models, without any model and task-specific designs.
+\href{https://xuxw98.github.io/Online3D/}{Project page}.",cs.CV,['cs.CV']
+Ungeneralizable Examples,Jingwen Ye · Xinchao Wang, ,https://arxiv.org/abs/2404.14016,,2404.14016.pdf,Ungeneralizable Examples,"The training of contemporary deep learning models heavily relies on publicly
+available data, posing a risk of unauthorized access to online data and raising
+concerns about data privacy. Current approaches to creating unlearnable data
+involve incorporating small, specially designed noises, but these methods
+strictly limit data usability, overlooking its potential usage in authorized
+scenarios. In this paper, we extend the concept of unlearnable data to
+conditional data learnability and introduce \textbf{U}n\textbf{G}eneralizable
+\textbf{E}xamples (UGEs). UGEs exhibit learnability for authorized users while
+maintaining unlearnability for potential hackers. The protector defines the
+authorized network and optimizes UGEs to match the gradients of the original
+data and its ungeneralizable version, ensuring learnability. To prevent
+unauthorized learning, UGEs are trained by maximizing a designated distance
+loss in a common feature space. Additionally, to further safeguard the
+authorized side from potential attacks, we introduce additional undistillation
+optimization. Experimental results on multiple datasets and various networks
+demonstrate that the proposed UGEs framework preserves data usability while
+reducing training performance on hacker networks, even under different types of
+attacks.",cs.LG,"['cs.LG', 'cs.CV']"
+ColorPCR: Color Point Cloud Registration with Multi-Stage Geometric-Color Fusion,Juncheng Mu · Lin Bie · Shaoyi Du · Yue Gao, ,,https://www.mdpi.com/2072-4292/16/5/743,,,,,nan
+IIRP-Net: Iterative Inference Residual Pyramid Network for Enhanced Image Registration,Tai Ma · zhangsuwei · Jiafeng Li · Ying Wen, ,https://arxiv.org/html/2312.13396v1,,2312.13396v1.pdf,EPNet: An Efficient Pyramid Network for Enhanced Single-Image Super-Resolution with Reduced Computational Requirements,"Single-image super-resolution (SISR) has seen significant advancements
+through the integration of deep learning. However, the substantial
+computational and memory requirements of existing methods often limit their
+practical application. This paper introduces a new Efficient Pyramid Network
+(EPNet) that harmoniously merges an Edge Split Pyramid Module (ESPM) with a
+Panoramic Feature Extraction Module (PFEM) to overcome the limitations of
+existing methods, particularly in terms of computational efficiency. The ESPM
+applies a pyramid-based channel separation strategy, boosting feature
+extraction while maintaining computational efficiency. The PFEM, a novel fusion
+of CNN and Transformer structures, enables the concurrent extraction of local
+and global features, thereby providing a panoramic view of the image landscape.
+Our architecture integrates the PFEM in a manner that facilitates the
+streamlined exchange of feature information and allows for the further
+refinement of image texture details. Experimental results indicate that our
+model outperforms existing state-of-the-art methods in image resolution
+quality, while considerably decreasing computational and memory costs. This
+research contributes to the ongoing evolution of efficient and practical SISR
+methodologies, bearing broader implications for the field of computer vision.",cs.CV,['cs.CV']
+Towards Efficient Replay in Federated Incremental Learning,Yichen Li · Qunwei Li · Haozhao Wang · Ruixuan Li · Wenliang Zhong · Guannan Zhang, ,https://arxiv.org/abs/2403.05890,,2403.05890.pdf,Towards Efficient Replay in Federated Incremental Learning,"In Federated Learning (FL), the data in each client is typically assumed
+fixed or static. However, data often comes in an incremental manner in
+real-world applications, where the data domain may increase dynamically. In
+this work, we study catastrophic forgetting with data heterogeneity in
+Federated Incremental Learning (FIL) scenarios where edge clients may lack
+enough storage space to retain full data. We propose to employ a simple,
+generic framework for FIL named Re-Fed, which can coordinate each client to
+cache important samples for replay. More specifically, when a new task arrives,
+each client first caches selected previous samples based on their global and
+local importance. Then, the client trains the local model with both the cached
+samples and the samples from the new task. Theoretically, we analyze the
+ability of Re-Fed to discover important samples for replay thus alleviating the
+catastrophic forgetting problem. Moreover, we empirically show that Re-Fed
+achieves competitive performance compared to state-of-the-art methods.",cs.LG,"['cs.LG', 'cs.DC']"
+Disentangled Pre-training for Human-Object Interaction Detection,Zhuolong Li · Xingao Li · Changxing Ding · Xiangmin Xu,https://github.com/xingaoli/DP-HOI,https://arxiv.org/abs/2404.01725,,2404.01725.pdf,Disentangled Pre-training for Human-Object Interaction Detection,"Detecting human-object interaction (HOI) has long been limited by the amount
+of supervised data available. Recent approaches address this issue by
+pre-training according to pseudo-labels, which align object regions with HOI
+triplets parsed from image captions. However, pseudo-labeling is tricky and
+noisy, making HOI pre-training a complex process. Therefore, we propose an
+efficient disentangled pre-training method for HOI detection (DP-HOI) to
+address this problem. First, DP-HOI utilizes object detection and action
+recognition datasets to pre-train the detection and interaction decoder layers,
+respectively. Then, we arrange these decoder layers so that the pre-training
+architecture is consistent with the downstream HOI detection task. This
+facilitates efficient knowledge transfer. Specifically, the detection decoder
+identifies reliable human instances in each action recognition dataset image,
+generates one corresponding query, and feeds it into the interaction decoder
+for verb classification. Next, we combine the human instance verb predictions
+in the same image and impose image-level supervision. The DP-HOI structure can
+be easily adapted to the HOI detection task, enabling effective model parameter
+initialization. Therefore, it significantly enhances the performance of
+existing HOI detection models on a broad range of rare categories. The code and
+pre-trained weight are available at https://github.com/xingaoli/DP-HOI.",cs.CV,['cs.CV']
+RegionGPT: Towards Region Understanding Vision Language Model,Qiushan Guo · Shalini De Mello · Danny Yin · Wonmin Byeon · Ka Chun Cheung · Yizhou Yu · Ping Luo · Sifei Liu,https://guoqiushan.github.io/regiongpt.github.io/,https://arxiv.org/abs/2403.02330v1,,2403.02330v1.pdf,RegionGPT: Towards Region Understanding Vision Language Model,"Vision language models (VLMs) have experienced rapid advancements through the
+integration of large language models (LLMs) with image-text pairs, yet they
+struggle with detailed regional visual understanding due to limited spatial
+awareness of the vision encoder, and the use of coarse-grained training data
+that lacks detailed, region-specific captions. To address this, we introduce
+RegionGPT (short as RGPT), a novel framework designed for complex region-level
+captioning and understanding. RGPT enhances the spatial awareness of regional
+representation with simple yet effective modifications to existing visual
+encoders in VLMs. We further improve performance on tasks requiring a specific
+output scope by integrating task-guided instruction prompts during both
+training and inference phases, while maintaining the model's versatility for
+general-purpose tasks. Additionally, we develop an automated region caption
+data generation pipeline, enriching the training set with detailed region-level
+captions. We demonstrate that a universal RGPT model can be effectively applied
+and significantly enhancing performance across a range of region-level tasks,
+including but not limited to complex region descriptions, reasoning, object
+classification, and referring expressions comprehension.",cs.CV,['cs.CV']
+Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence,Ripon Saha · Dehao Qin · Nianyi Li · Jinwei Ye · Suren Jayasuriya, ,https://arxiv.org/abs/2404.13605,,2404.13605.pdf,Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence,"Tackling image degradation due to atmospheric turbulence, particularly in
+dynamic environment, remains a challenge for long-range imaging systems.
+Existing techniques have been primarily designed for static scenes or scenes
+with small motion. This paper presents the first segment-then-restore pipeline
+for restoring the videos of dynamic scenes in turbulent environment. We
+leverage mean optical flow with an unsupervised motion segmentation method to
+separate dynamic and static scene components prior to restoration. After camera
+shake compensation and segmentation, we introduce foreground/background
+enhancement leveraging the statistics of turbulence strength and a transformer
+model trained on a novel noise-based procedural turbulence generator for fast
+dataset augmentation. Benchmarked against existing restoration methods, our
+approach restores most of the geometric distortion and enhances sharpness for
+videos. We make our code, simulator, and data publicly available to advance the
+field of video restoration from turbulence: riponcs.github.io/TurbSegRes",cs.CV,"['cs.CV', 'eess.IV']"
+Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo,Zongrui Li · Zhan Lu · Haojie Yan · Boxin Shi · Gang Pan · Qian Zheng · Xudong Jiang, ,https://arxiv.org/abs/2404.01612,,2404.01612.pdf,Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo,"Natural Light Uncalibrated Photometric Stereo (NaUPS) relieves the strict
+environment and light assumptions in classical Uncalibrated Photometric Stereo
+(UPS) methods. However, due to the intrinsic ill-posedness and high-dimensional
+ambiguities, addressing NaUPS is still an open question. Existing works impose
+strong assumptions on the environment lights and objects' material, restricting
+the effectiveness in more general scenarios. Alternatively, some methods
+leverage supervised learning with intricate models while lacking
+interpretability, resulting in a biased estimation. In this work, we proposed
+Spin Light Uncalibrated Photometric Stereo (Spin-UP), an unsupervised method to
+tackle NaUPS in various environment lights and objects. The proposed method
+uses a novel setup that captures the object's images on a rotatable platform,
+which mitigates NaUPS's ill-posedness by reducing unknowns and provides
+reliable priors to alleviate NaUPS's ambiguities. Leveraging neural inverse
+rendering and the proposed training strategies, Spin-UP recovers surface
+normals, environment light, and isotropic reflectance under complex natural
+light with low computational cost. Experiments have shown that Spin-UP
+outperforms other supervised / unsupervised NaUPS methods and achieves
+state-of-the-art performance on synthetic and real-world datasets. Codes and
+data are available at https://github.com/LMozart/CVPR2024-SpinUP.",cs.CV,['cs.CV']
+Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training,Arun Reddy · William Paul · Corban Rivera · Ketul Shah · Celso M. de Melo · Rama Chellappa, ,https://arxiv.org/abs/2312.02914,,2312.02914.pdf,Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training,"In this work, we tackle the problem of unsupervised domain adaptation (UDA)
+for video action recognition. Our approach, which we call UNITE, uses an image
+teacher model to adapt a video student model to the target domain. UNITE first
+employs self-supervised pre-training to promote discriminative feature learning
+on target domain videos using a teacher-guided masked distillation objective.
+We then perform self-training on masked target data, using the video student
+model and image teacher model together to generate improved pseudolabels for
+unlabeled target videos. Our self-training process successfully leverages the
+strengths of both models to achieve strong transfer performance across domains.
+We evaluate our approach on multiple video domain adaptation benchmarks and
+observe significant improvements upon previously reported results.",cs.CV,"['cs.CV', 'cs.LG']"
+Would Deep Generative Models Amplify Bias in Future Models?,Tianwei Chen · Yusuke Hirota · Mayu Otani · Noa Garcia · Yuta Nakashima, ,https://arxiv.org/abs/2404.03242,,2404.03242.pdf,Would Deep Generative Models Amplify Bias in Future Models?,"We investigate the impact of deep generative models on potential social
+biases in upcoming computer vision models. As the internet witnesses an
+increasing influx of AI-generated images, concerns arise regarding inherent
+biases that may accompany them, potentially leading to the dissemination of
+harmful content. This paper explores whether a detrimental feedback loop,
+resulting in bias amplification, would occur if generated images were used as
+the training data for future models. We conduct simulations by progressively
+substituting original images in COCO and CC3M datasets with images generated
+through Stable Diffusion. The modified datasets are used to train OpenCLIP and
+image captioning models, which we evaluate in terms of quality and bias.
+Contrary to expectations, our findings indicate that introducing generated
+images during training does not uniformly amplify bias. Instead, instances of
+bias mitigation across specific tasks are observed. We further explore the
+factors that may influence these phenomena, such as artifacts in image
+generation (e.g., blurry faces) or pre-existing biases in the original
+datasets.",cs.CV,['cs.CV']
+Prompt-Free Diffusion: Taking “Text” out of Text-to-Image Diffusion Models,Xingqian Xu · Jiayi Guo · Zhangyang Wang · Gao Huang · Irfan Essa · Humphrey Shi, ,,https://openreview.net/forum?id=QL3Zuth6E7,,,,,nan
+Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models,Jiayi Guo · Xingqian Xu · Yifan Pu · Zanlin Ni · Chaofei Wang · Manushree Vasu · Shiji Song · Gao Huang · Humphrey Shi,https://shi-labs.github.io/Smooth-Diffusion/,https://arxiv.org/abs/2312.04410,,2312.04410.pdf,Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models,"Recently, diffusion models have made remarkable progress in text-to-image
+(T2I) generation, synthesizing images with high fidelity and diverse contents.
+Despite this advancement, latent space smoothness within diffusion models
+remains largely unexplored. Smooth latent spaces ensure that a perturbation on
+an input latent corresponds to a steady change in the output image. This
+property proves beneficial in downstream tasks, including image interpolation,
+inversion, and editing. In this work, we expose the non-smoothness of diffusion
+latent spaces by observing noticeable visual fluctuations resulting from minor
+latent variations. To tackle this issue, we propose Smooth Diffusion, a new
+category of diffusion models that can be simultaneously high-performing and
+smooth. Specifically, we introduce Step-wise Variation Regularization to
+enforce the proportion between the variations of an arbitrary input latent and
+that of the output image is a constant at any diffusion training step. In
+addition, we devise an interpolation standard deviation (ISTD) metric to
+effectively assess the latent space smoothness of a diffusion model. Extensive
+quantitative and qualitative experiments demonstrate that Smooth Diffusion
+stands out as a more desirable solution not only in T2I generation but also
+across various downstream tasks. Smooth Diffusion is implemented as a
+plug-and-play Smooth-LoRA to work with various community models. Code is
+available at https://github.com/SHI-Labs/Smooth-Diffusion.",cs.CV,['cs.CV']
+PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor,Vidit Goel · Elia Peruzzo · Yifan Jiang · Dejia Xu · Xingqian Xu · Nicu Sebe · Trevor Darrell · Zhangyang Wang · Humphrey Shi,https://vidit98.github.io/publication/conference-paper/pair_diff.html,,https://openreview.net/forum?id=cI5j8tEPNU,,,,,nan
+Large Language Models are Good Prompt Learners for Low-Shot Image Classification,Zhaoheng Zheng · Jingmin Wei · Xuefeng Hu · Haidong Zhu · Ram Nevatia, ,https://arxiv.org/abs/2312.04076,,2312.04076.pdf,Large Language Models are Good Prompt Learners for Low-Shot Image Classification,"Low-shot image classification, where training images are limited or
+inaccessible, has benefited from recent progress on pre-trained vision-language
+(VL) models with strong generalizability, e.g. CLIP. Prompt learning methods
+built with VL models generate text features from the class names that only have
+confined class-specific information. Large Language Models (LLMs), with their
+vast encyclopedic knowledge, emerge as the complement. Thus, in this paper, we
+discuss the integration of LLMs to enhance pre-trained VL models, specifically
+on low-shot classification. However, the domain gap between language and vision
+blocks the direct application of LLMs. Thus, we propose LLaMP, Large Language
+Models as Prompt learners, that produces adaptive prompts for the CLIP text
+encoder, establishing it as the connecting bridge. Experiments show that,
+compared with other state-of-the-art prompt learning methods, LLaMP yields
+better performance on both zero-shot generalization and few-shot image
+classification, over a spectrum of 11 datasets. Code will be made available at:
+https://github.com/zhaohengz/LLaMP.",cs.CV,['cs.CV']
+SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream,Lin Zhu · Kangmin Jia · Yifan Zhao · Yunshan Qi · Lizhi Wang · Hua Huang, ,https://arxiv.org/abs/2403.11222,,2403.11222.pdf,SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream,"Spike cameras, leveraging spike-based integration sampling and high temporal
+resolution, offer distinct advantages over standard cameras. However, existing
+approaches reliant on spike cameras often assume optimal illumination, a
+condition frequently unmet in real-world scenarios. To address this, we
+introduce SpikeNeRF, the first work that derives a NeRF-based volumetric scene
+representation from spike camera data. Our approach leverages NeRF's multi-view
+consistency to establish robust self-supervision, effectively eliminating
+erroneous measurements and uncovering coherent structures within exceedingly
+noisy input amidst diverse real-world illumination scenarios. The framework
+comprises two core elements: a spike generation model incorporating an
+integrate-and-fire neuron layer and parameters accounting for non-idealities,
+such as threshold variation, and a spike rendering loss capable of generalizing
+across varying illumination conditions. We describe how to effectively optimize
+neural radiance fields to render photorealistic novel views from the novel
+continuous spike stream, demonstrating advantages over other vision sensors in
+certain scenes. Empirical evaluations conducted on both real and novel
+realistically simulated sequences affirm the efficacy of our methodology. The
+dataset and source code are released at
+https://github.com/BIT-Vision/SpikeNeRF.",cs.CV,['cs.CV']
+UVEB: A Large-scale Benchmark and Baseline Towards Real-World Underwater Video Enhancement,yaofeng xie · Lingwei Kong · Kai Chen · Zheng Ziqiang · Xiao Yu · Zhibin Yu · Bing Zheng,https://github.com/yzbouc/UVEB,https://arxiv.org/abs/2404.14542,,2404.14542.pdf,UVEB: A Large-scale Benchmark and Baseline Towards Real-World Underwater Video Enhancement,"Learning-based underwater image enhancement (UIE) methods have made great
+progress. However, the lack of large-scale and high-quality paired training
+samples has become the main bottleneck hindering the development of UIE. The
+inter-frame information in underwater videos can accelerate or optimize the UIE
+process. Thus, we constructed the first large-scale high-resolution underwater
+video enhancement benchmark (UVEB) to promote the development of underwater
+vision.It contains 1,308 pairs of video sequences and more than 453,000
+high-resolution with 38\% Ultra-High-Definition (UHD) 4K frame pairs. UVEB
+comes from multiple countries, containing various scenes and video degradation
+types to adapt to diverse and complex underwater environments. We also propose
+the first supervised underwater video enhancement method, UVE-Net. UVE-Net
+converts the current frame information into convolutional kernels and passes
+them to adjacent frames for efficient inter-frame information exchange. By
+fully utilizing the redundant degraded information of underwater videos,
+UVE-Net completes video enhancement better. Experiments show the effective
+network design and good performance of UVE-Net.",cs.CV,"['cs.CV', 'I.4']"
+Single-View Scene Point Cloud Human Grasp Generation,Yan-Kang Wang · Chengyi Xing · Yi-Lin Wei · Xiao-Ming Wu · Wei-Shi Zheng, ,https://arxiv.org/abs/2404.15815,,2404.15815.pdf,Single-View Scene Point Cloud Human Grasp Generation,"In this work, we explore a novel task of generating human grasps based on
+single-view scene point clouds, which more accurately mirrors the typical
+real-world situation of observing objects from a single viewpoint. Due to the
+incompleteness of object point clouds and the presence of numerous scene
+points, the generated hand is prone to penetrating into the invisible parts of
+the object and the model is easily affected by scene points. Thus, we introduce
+S2HGrasp, a framework composed of two key modules: the Global Perception module
+that globally perceives partial object point clouds, and the DiffuGrasp module
+designed to generate high-quality human grasps based on complex inputs that
+include scene points. Additionally, we introduce S2HGD dataset, which comprises
+approximately 99,000 single-object single-view scene point clouds of 1,668
+unique objects, each annotated with one human grasp. Our extensive experiments
+demonstrate that S2HGrasp can not only generate natural human grasps regardless
+of scene points, but also effectively prevent penetration between the hand and
+invisible parts of the object. Moreover, our model showcases strong
+generalization capability when applied to unseen objects. Our code and dataset
+are available at https://github.com/iSEE-Laboratory/S2HGrasp.",cs.CV,['cs.CV']
+MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception,Yiran Qin · Enshen Zhou · Qichang Liu · Zhenfei Yin · Lu Sheng · Ruimao Zhang · Yu Qiao · Jing Shao,https://iranqin.github.io/MP5.github.io/,https://arxiv.org/abs/2312.07472,,2312.07472.pdf,MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception,"It is a long-lasting goal to design an embodied system that can solve
+long-horizon open-world tasks in human-like ways. However, existing approaches
+usually struggle with compound difficulties caused by the logic-aware
+decomposition and context-aware execution of these tasks. To this end, we
+introduce MP5, an open-ended multimodal embodied system built upon the
+challenging Minecraft simulator, which can decompose feasible sub-objectives,
+design sophisticated situation-aware plans, and perform embodied action
+control, with frequent communication with a goal-conditioned active perception
+scheme. Specifically, MP5 is developed on top of recent advances in Multimodal
+Large Language Models (MLLMs), and the system is modulated into functional
+modules that can be scheduled and collaborated to ultimately solve pre-defined
+context- and process-dependent tasks. Extensive experiments prove that MP5 can
+achieve a 22% success rate on difficult process-dependent tasks and a 91%
+success rate on tasks that heavily depend on the context. Moreover, MP5
+exhibits a remarkable ability to address many open-ended tasks that are
+entirely novel.",cs.CV,['cs.CV']
+Scaling Up Video Summarization Pretraining with Large Language Models,Dawit Argaw Argaw · Seunghyun Yoon · Fabian Caba Heilbron · Hanieh Deilamsalehy · Trung Bui · Zhaowen Wang · Franck Dernoncourt · Joon Chung, ,https://arxiv.org/abs/2404.03398,,2404.03398.pdf,Scaling Up Video Summarization Pretraining with Large Language Models,"Long-form video content constitutes a significant portion of internet
+traffic, making automated video summarization an essential research problem.
+However, existing video summarization datasets are notably limited in their
+size, constraining the effectiveness of state-of-the-art methods for
+generalization. Our work aims to overcome this limitation by capitalizing on
+the abundance of long-form videos with dense speech-to-video alignment and the
+remarkable capabilities of recent large language models (LLMs) in summarizing
+long text. We introduce an automated and scalable pipeline for generating a
+large-scale video summarization dataset using LLMs as Oracle summarizers. By
+leveraging the generated dataset, we analyze the limitations of existing
+approaches and propose a new video summarization model that effectively
+addresses them. To facilitate further research in the field, our work also
+presents a new benchmark dataset that contains 1200 long videos each with
+high-quality summaries annotated by professionals. Extensive experiments
+clearly indicate that our proposed approach sets a new state-of-the-art in
+video summarization across several benchmarks.",cs.CV,['cs.CV']
+CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification,Haoran Lai · Qingsong Yao · Zihang Jiang · Rongsheng Wang · Zhiyang He · Xiaodong Tao · S Kevin Zhou, ,https://arxiv.org/abs/2402.17417,,2402.17417.pdf,CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification,"The advancement of Zero-Shot Learning in the medical domain has been driven
+forward by using pre-trained models on large-scale image-text pairs, focusing
+on image-text alignment. However, existing methods primarily rely on cosine
+similarity for alignment, which may not fully capture the complex relationship
+between medical images and reports. To address this gap, we introduce a novel
+approach called Cross-Attention Alignment for Radiology Zero-Shot
+Classification (CARZero). Our approach innovatively leverages cross-attention
+mechanisms to process image and report features, creating a Similarity
+Representation that more accurately reflects the intricate relationships in
+medical semantics. This representation is then linearly projected to form an
+image-text similarity matrix for cross-modality alignment. Additionally,
+recognizing the pivotal role of prompt selection in zero-shot learning, CARZero
+incorporates a Large Language Model-based prompt alignment strategy. This
+strategy standardizes diverse diagnostic expressions into a unified format for
+both training and inference phases, overcoming the challenges of manual prompt
+design. Our approach is simple yet effective, demonstrating state-of-the-art
+performance in zero-shot classification on five official chest radiograph
+diagnostic test sets, including remarkable results on datasets with long-tail
+distributions of rare diseases. This achievement is attributed to our new
+image-text alignment strategy, which effectively addresses the complex
+relationship between medical images and reports. Code and models are available
+at https://github.com/laihaoran/CARZero.",cs.CV,['cs.CV']
+LiDAR-Net: A Real-scanned 3D Point Cloud Dataset for Indoor Scenes,Yanwen Guo · Yuanqi Li · Dayong Ren · Xiaohong Zhang · Jiawei Li · Liang Pu · Changfeng Ma · xiaoyu zhan · Jie Guo · Mingqiang Wei · Yan Zhang · Piaopiao Yu · Shuangyu Yang · Donghao Ji · Huisheng Ye · Hao Sun · Yansong Liu · Yinuo Chen · Jiaqi Zhu · Hongyu Liu, ,https://arxiv.org/html/2309.13596v2,,2309.13596v2.pdf,Advancements in 3D Lane Detection Using LiDAR Point Clouds: From Data Collection to Model Development,"Advanced Driver-Assistance Systems (ADAS) have successfully integrated
+learning-based techniques into vehicle perception and decision-making. However,
+their application in 3D lane detection for effective driving environment
+perception is hindered by the lack of comprehensive LiDAR datasets. The sparse
+nature of LiDAR point cloud data prevents an efficient manual annotation
+process. To solve this problem, we present LiSV-3DLane, a large-scale 3D lane
+dataset that comprises 20k frames of surround-view LiDAR point clouds with
+enriched semantic annotation. Unlike existing datasets confined to a frontal
+perspective, LiSV-3DLane provides a full 360-degree spatial panorama around the
+ego vehicle, capturing complex lane patterns in both urban and highway
+environments. We leverage the geometric traits of lane lines and the intrinsic
+spatial attributes of LiDAR data to design a simple yet effective automatic
+annotation pipeline for generating finer lane labels. To propel future
+research, we propose a novel LiDAR-based 3D lane detection model, LiLaDet,
+incorporating the spatial geometry learning of the LiDAR point cloud into
+Bird's Eye View (BEV) based lane identification. Experimental results indicate
+that LiLaDet outperforms existing camera- and LiDAR-based approaches in the 3D
+lane detection task on the K-Lane dataset and our LiSV-3DLane.",cs.CV,['cs.CV']
+Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection,Chuangchuang Tan · Huan Liu · Yao Zhao · Shikui Wei · Guanghua Gu · Ping Liu · Yunchao Wei,https://github.com/chuangchuangtan/NPR-DeepfakeDetection,https://arxiv.org/abs/2312.10461,,2312.10461.pdf,Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection,"Recently, the proliferation of highly realistic synthetic images, facilitated
+through a variety of GANs and Diffusions, has significantly heightened the
+susceptibility to misuse. While the primary focus of deepfake detection has
+traditionally centered on the design of detection algorithms, an investigative
+inquiry into the generator architectures has remained conspicuously absent in
+recent years. This paper contributes to this lacuna by rethinking the
+architectures of CNN-based generators, thereby establishing a generalized
+representation of synthetic artifacts. Our findings illuminate that the
+up-sampling operator can, beyond frequency-based artifacts, produce generalized
+forgery artifacts. In particular, the local interdependence among image pixels
+caused by upsampling operators is significantly demonstrated in synthetic
+images generated by GAN or diffusion. Building upon this observation, we
+introduce the concept of Neighboring Pixel Relationships(NPR) as a means to
+capture and characterize the generalized structural artifacts stemming from
+up-sampling operations. A comprehensive analysis is conducted on an open-world
+dataset, comprising samples generated by \tft{28 distinct generative models}.
+This analysis culminates in the establishment of a novel state-of-the-art
+performance, showcasing a remarkable \tft{11.6\%} improvement over existing
+methods. The code is available at
+https://github.com/chuangchuangtan/NPR-DeepfakeDetection.",cs.CV,['cs.CV']
+SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection,JUNSU KIM · Hoseong Cho · Jihyeon Kim · Yihalem Tiruneh · Seungryul Baek, ,https://arxiv.org/abs/2402.17323,,2402.17323.pdf,SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection,"In the field of class incremental learning (CIL), generative replay has
+become increasingly prominent as a method to mitigate the catastrophic
+forgetting, alongside the continuous improvements in generative models.
+However, its application in class incremental object detection (CIOD) has been
+significantly limited, primarily due to the complexities of scenes involving
+multiple labels. In this paper, we propose a novel approach called stable
+diffusion deep generative replay (SDDGR) for CIOD. Our method utilizes a
+diffusion-based generative model with pre-trained text-to-diffusion networks to
+generate realistic and diverse synthetic images. SDDGR incorporates an
+iterative refinement strategy to produce high-quality images encompassing old
+classes. Additionally, we adopt an L2 knowledge distillation technique to
+improve the retention of prior knowledge in synthetic images. Furthermore, our
+approach includes pseudo-labeling for old objects within new task images,
+preventing misclassification as background elements. Extensive experiments on
+the COCO 2017 dataset demonstrate that SDDGR significantly outperforms existing
+algorithms, achieving a new state-of-the-art in various CIOD scenarios. The
+source code will be made available to the public.",cs.CV,['cs.CV']
+Mean-Shift Feature Transformer,Takumi Kobayashi, ,https://arxiv.org/abs/2404.11062,,2404.11062.pdf,Generation of a precise time scale assisted by a near-continuously operating optical lattice clock,"We report on a reduced time variation of a time scale with respect to
+Coordinated Universal Time (UTC) by steering a hydrogen-maser-based time scale
+with a near-continuously operating optical lattice clock. The time scale is
+generated in a post-processing analysis for 230 days with a hydrogen maser with
+its fractional frequency stability limited by a flicker floor of
+$2\times10^{-15}$ and an Yb optical lattice clock operated with an uptime of
+81.6 $\%$. During the 230-day period, the root mean square time variation of
+our time scale with respect to UTC is 0.52 ns, which is a better performance
+compared with those of time scales steered by microwave fountain clocks that
+exhibit root mean square variations from 0.99 ns to 1.6 ns. With the high
+uptime achieved by the Yb optical lattice clock, our simulation implies the
+potential of generating a state-of-the-art time scale with a time variation of
+$<0.1$ ns over a month using a better hydrogen maser reaching the mid
+$10^{-16}$ level. This work demonstrates that a use of an optical clock with a
+high uptime enhances the stability of a time scale.",physics.atom-ph,['physics.atom-ph']
+TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding,Zhihao Zhang · Shengcao Cao · Yu-Xiong Wang, ,https://arxiv.org/abs/2402.18490,,2402.18490.pdf,TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding,"The limited scale of current 3D shape datasets hinders the advancements in 3D
+shape understanding, and motivates multi-modal learning approaches which
+transfer learned knowledge from data-abundant 2D image and language modalities
+to 3D shapes. However, even though the image and language representations have
+been aligned by cross-modal models like CLIP, we find that the image modality
+fails to contribute as much as the language in existing multi-modal 3D
+representation learning methods. This is attributed to the domain shift in the
+2D images and the distinct focus of each modality. To more effectively leverage
+both modalities in the pre-training, we introduce TriAdapter Multi-Modal
+Learning (TAMM) -- a novel two-stage learning approach based on three
+synergistic adapters. First, our CLIP Image Adapter mitigates the domain gap
+between 3D-rendered images and natural images, by adapting the visual
+representations of CLIP for synthetic image-text pairs. Subsequently, our Dual
+Adapters decouple the 3D shape representation space into two complementary
+sub-spaces: one focusing on visual attributes and the other for semantic
+understanding, which ensure a more comprehensive and effective multi-modal
+pre-training. Extensive experiments demonstrate that TAMM consistently enhances
+3D representations for a wide range of 3D encoder architectures, pre-training
+datasets, and downstream tasks. Notably, we boost the zero-shot classification
+accuracy on Objaverse-LVIS from 46.8\% to 50.7\%, and improve the 5-way 10-shot
+linear probing classification accuracy on ModelNet40 from 96.1\% to 99.0\%.
+Project page: https://alanzhangcs.github.io/tamm-page.",cs.CV,['cs.CV']
+Open-Vocabulary 3D Semantic Segmentation with Foundation Models,Li Jiang · Shaoshuai Shi · Bernt Schiele, ,https://arxiv.org/abs/2306.13631,,2306.13631.pdf,OpenMask3D: Open-Vocabulary 3D Instance Segmentation,"We introduce the task of open-vocabulary 3D instance segmentation. Current
+approaches for 3D instance segmentation can typically only recognize object
+categories from a pre-defined closed set of classes that are annotated in the
+training datasets. This results in important limitations for real-world
+applications where one might need to perform tasks guided by novel,
+open-vocabulary queries related to a wide variety of objects. Recently,
+open-vocabulary 3D scene understanding methods have emerged to address this
+problem by learning queryable features for each point in the scene. While such
+a representation can be directly employed to perform semantic segmentation,
+existing methods cannot separate multiple object instances. In this work, we
+address this limitation, and propose OpenMask3D, which is a zero-shot approach
+for open-vocabulary 3D instance segmentation. Guided by predicted
+class-agnostic 3D instance masks, our model aggregates per-mask features via
+multi-view fusion of CLIP-based image embeddings. Experiments and ablation
+studies on ScanNet200 and Replica show that OpenMask3D outperforms other
+open-vocabulary methods, especially on the long-tail distribution. Qualitative
+experiments further showcase OpenMask3D's ability to segment object properties
+based on free-form queries describing geometry, affordances, and materials.",cs.CV,['cs.CV']
+Multiplane Prior Guided Few-Shot Aerial Scene Rendering,Zihan Gao · Licheng Jiao · Lingling Li · Xu Liu · Fang Liu · Puhua Chen · Yuwei Guo, ,http://export.arxiv.org/abs/2402.16407,,2402.16407.pdf,CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency,"Neural Radiance Field (NeRF) has shown impressive results in novel view
+synthesis, particularly in Virtual Reality (VR) and Augmented Reality (AR),
+thanks to its ability to represent scenes continuously. However, when just a
+few input view images are available, NeRF tends to overfit the given views and
+thus make the estimated depths of pixels share almost the same value. Unlike
+previous methods that conduct regularization by introducing complex priors or
+additional supervisions, we propose a simple yet effective method that
+explicitly builds depth-aware consistency across input views to tackle this
+challenge. Our key insight is that by forcing the same spatial points to be
+sampled repeatedly in different input views, we are able to strengthen the
+interactions between views and therefore alleviate the overfitting problem. To
+achieve this, we build the neural networks on layered representations
+(\textit{i.e.}, multiplane images), and the sampling point can thus be
+resampled on multiple discrete planes. Furthermore, to regularize the unseen
+target views, we constrain the rendered colors and depths from different input
+views to be the same. Although simple, extensive experiments demonstrate that
+our proposed method can achieve better synthesis quality over state-of-the-art
+methods.",cs.CV,"['cs.CV', 'cs.GR']"
+One-step Diffusion with Distribution Matching Distillation,Tianwei Yin · Michaël Gharbi · Michaël Gharbi · Richard Zhang · Eli Shechtman · Fredo Durand · William Freeman · Taesung Park, ,https://arxiv.org/abs/2311.18828,,2311.18828.pdf,One-step Diffusion with Distribution Matching Distillation,"Diffusion models generate high-quality images but require dozens of forward
+passes. We introduce Distribution Matching Distillation (DMD), a procedure to
+transform a diffusion model into a one-step image generator with minimal impact
+on image quality. We enforce the one-step image generator match the diffusion
+model at distribution level, by minimizing an approximate KL divergence whose
+gradient can be expressed as the difference between 2 score functions, one of
+the target distribution and the other of the synthetic distribution being
+produced by our one-step generator. The score functions are parameterized as
+two diffusion models trained separately on each distribution. Combined with a
+simple regression loss matching the large-scale structure of the multi-step
+diffusion outputs, our method outperforms all published few-step diffusion
+approaches, reaching 2.62 FID on ImageNet 64x64 and 11.49 FID on zero-shot
+COCO-30k, comparable to Stable Diffusion but orders of magnitude faster.
+Utilizing FP16 inference, our model generates images at 20 FPS on modern
+hardware.",cs.CV,['cs.CV']
+Towards 3D Vision with Low-Cost Single-Photon Cameras,Fangzhou Mu · Carter Sifferman · Sacha Jungerman · Yiquan Li · Zhiyue Han · Michael Gleicher · Mohit Gupta · Yin Li,https://cpsiff.github.io/towards_3d_vision/,https://arxiv.org/abs/2403.17801,,2403.17801.pdf,Towards 3D Vision with Low-Cost Single-Photon Cameras,"We present a method for reconstructing 3D shape of arbitrary Lambertian
+objects based on measurements by miniature, energy-efficient, low-cost
+single-photon cameras. These cameras, operating as time resolved image sensors,
+illuminate the scene with a very fast pulse of diffuse light and record the
+shape of that pulse as it returns back from the scene at a high temporal
+resolution. We propose to model this image formation process, account for its
+non-idealities, and adapt neural rendering to reconstruct 3D geometry from a
+set of spatially distributed sensors with known poses. We show that our
+approach can successfully recover complex 3D shapes from simulated data. We
+further demonstrate 3D object reconstruction from real-world captures,
+utilizing measurements from a commodity proximity sensor. Our work draws a
+connection between image-based modeling and active range scanning and is a step
+towards 3D vision with single-photon cameras.",cs.CV,"['cs.CV', 'eess.IV']"
+RepKPU: Point Cloud Upsampling with Kernel Point Representation and Deformation,Yi Rong · Haoran Zhou · Kang Xia · Cheng Mei · Jiahao Wang · Tong Lu, ,,https://www.mdpi.com/2072-4292/16/3/450,,,,,nan
+UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence,Ruihai Wu · Haoran Lu · Yiyan Wang · Yubo Wang · Hao Dong, ,https://arxiv.org/abs/2405.06903,,2405.06903.pdf,UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence,"Garment manipulation (e.g., unfolding, folding and hanging clothes) is
+essential for future robots to accomplish home-assistant tasks, while highly
+challenging due to the diversity of garment configurations, geometries and
+deformations. Although able to manipulate similar shaped garments in a certain
+task, previous works mostly have to design different policies for different
+tasks, could not generalize to garments with diverse geometries, and often rely
+heavily on human-annotated data. In this paper, we leverage the property that,
+garments in a certain category have similar structures, and then learn the
+topological dense (point-level) visual correspondence among garments in the
+category level with different deformations in the self-supervised manner. The
+topological correspondence can be easily adapted to the functional
+correspondence to guide the manipulation policies for various downstream tasks,
+within only one or few-shot demonstrations. Experiments over garments in 3
+different categories on 3 representative tasks in diverse scenarios, using one
+or two arms, taking one or more steps, inputting flat or messy garments,
+demonstrate the effectiveness of our proposed method. Project page:
+https://warshallrho.github.io/unigarmentmanip.",cs.CV,['cs.CV']
+Learning Diffusion Texture Priors for Image Restoration,Tian Ye · Sixiang Chen · Wenhao Chai · Zhaohu Xing · Jing Qin · Ge lin · Lei Zhu, ,https://arxiv.org/abs/2312.08606,,2312.08606.pdf,VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook,"Night photography often struggles with challenges like low light and
+blurring, stemming from dark environments and prolonged exposures. Current
+methods either disregard priors and directly fitting end-to-end networks,
+leading to inconsistent illumination, or rely on unreliable handcrafted priors
+to constrain the network, thereby bringing the greater error to the final
+result. We believe in the strength of data-driven high-quality priors and
+strive to offer a reliable and consistent prior, circumventing the restrictions
+of manual priors. In this paper, we propose Clearer Night Image Restoration
+with Vector-Quantized Codebook (VQCNIR) to achieve remarkable and consistent
+restoration outcomes on real-world and synthetic benchmarks. To ensure the
+faithful restoration of details and illumination, we propose the incorporation
+of two essential modules: the Adaptive Illumination Enhancement Module (AIEM)
+and the Deformable Bi-directional Cross-Attention (DBCA) module. The AIEM
+leverages the inter-channel correlation of features to dynamically maintain
+illumination consistency between degraded features and high-quality codebook
+features. Meanwhile, the DBCA module effectively integrates texture and
+structural information through bi-directional cross-attention and deformable
+convolution, resulting in enhanced fine-grained detail and structural fidelity
+across parallel decoders. Extensive experiments validate the remarkable
+benefits of VQCNIR in enhancing image quality under low-light conditions,
+showcasing its state-of-the-art performance on both synthetic and real-world
+datasets. The code is available at https://github.com/AlexZou14/VQCNIR.",cs.CV,['cs.CV']
+Move Anything with Layered Scene Diffusion,Jiawei Ren · Mengmeng Xu · Jui-Chieh Wu · Ziwei Liu · Tao Xiang · Antoine Toisoul, ,https://arxiv.org/abs/2404.07178,,2404.07178.pdf,Move Anything with Layered Scene Diffusion,"Diffusion models generate images with an unprecedented level of quality, but
+how can we freely rearrange image layouts? Recent works generate controllable
+scenes via learning spatially disentangled latent codes, but these methods do
+not apply to diffusion models due to their fixed forward process. In this work,
+we propose SceneDiffusion to optimize a layered scene representation during the
+diffusion sampling process. Our key insight is that spatial disentanglement can
+be obtained by jointly denoising scene renderings at different spatial layouts.
+Our generated scenes support a wide range of spatial editing operations,
+including moving, resizing, cloning, and layer-wise appearance editing
+operations, including object restyling and replacing. Moreover, a scene can be
+generated conditioned on a reference image, thus enabling object moving for
+in-the-wild images. Notably, this approach is training-free, compatible with
+general text-to-image diffusion models, and responsive in less than a second.",cs.CV,['cs.CV']
+MoML: Online Meta Adaptation for 3D Human Motion Prediction,Xiaoning Sun · Huaijiang Sun · Bin Li · Dong Wei · Weiqing Li · Jianfeng Lu, ,https://arxiv.org/abs/2405.02911,,,Multimodal Sense-Informed Prediction of 3D Human Motions,"Predicting future human pose is a fundamental application for machine
+intelligence, which drives robots to plan their behavior and paths ahead of
+time to seamlessly accomplish human-robot collaboration in real-world 3D
+scenarios. Despite encouraging results, existing approaches rarely consider the
+effects of the external scene on the motion sequence, leading to pronounced
+artifacts and physical implausibilities in the predictions. To address this
+limitation, this work introduces a novel multi-modal sense-informed motion
+prediction approach, which conditions high-fidelity generation on two modal
+information: external 3D scene, and internal human gaze, and is able to
+recognize their salience for future human activity. Furthermore, the gaze
+information is regarded as the human intention, and combined with both motion
+and scene features, we construct a ternary intention-aware attention to
+supervise the generation to match where the human wants to reach. Meanwhile, we
+introduce semantic coherence-aware attention to explicitly distinguish the
+salient point clouds and the underlying ones, to ensure a reasonable
+interaction of the generated sequence with the 3D scene. On two real-world
+benchmarks, the proposed method achieves state-of-the-art performance both in
+3D human pose and trajectory prediction.",cs.CV,['cs.CV']
+Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark,Ziyang Chen · Israel D. Gebru · Christian Richardt · Anurag Kumar · William Laney · Andrew Owens · Alexander Richard, ,,https://openreview.net/forum?id=Mk0Uf3zHtU,,,,,nan
+Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling,Shentong Mo · Pedro Morgado, ,https://arxiv.org/abs/2312.01017,,2312.01017.pdf,Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling,"Humans possess a remarkable ability to integrate auditory and visual
+information, enabling a deeper understanding of the surrounding environment.
+This early fusion of audio and visual cues, demonstrated through cognitive
+psychology and neuroscience research, offers promising potential for developing
+multimodal perception models. However, training early fusion architectures
+poses significant challenges, as the increased model expressivity requires
+robust learning frameworks to harness their enhanced capabilities. In this
+paper, we address this challenge by leveraging the masked reconstruction
+framework, previously successful in unimodal settings, to train audio-visual
+encoders with early fusion. Additionally, we propose an attention-based fusion
+module that captures interactions between local audio and visual
+representations, enhancing the model's ability to capture fine-grained
+interactions. While effective, this procedure can become computationally
+intractable, as the number of local representations increases. Thus, to address
+the computational complexity, we propose an alternative procedure that
+factorizes the local representations before representing audio-visual
+interactions. Extensive evaluations on a variety of datasets demonstrate the
+superiority of our approach in audio-event classification, visual sound
+localization, sound separation, and audio-visual segmentation. These
+contributions enable the efficient training of deeply integrated audio-visual
+models and significantly advance the usefulness of early fusion architectures.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM', 'cs.SD']"
+HumanNeRF-SE: A Simple yet Effective Approach to Animate HumanNeRF with Diverse Poses,Caoyuan Ma · Yu-Lun Liu · Zhixiang Wang · Wu Liu · Xinchen Liu · Zheng Wang, ,https://arxiv.org/abs/2312.02232,,2312.02232.pdf,HumanNeRF-SE: A Simple yet Effective Approach to Animate HumanNeRF with Diverse Poses,"We present HumanNeRF-SE, a simple yet effective method that synthesizes
+diverse novel pose images with simple input. Previous HumanNeRF works require a
+large number of optimizable parameters to fit the human images. Instead, we
+reload these approaches by combining explicit and implicit human
+representations to design both generalized rigid deformation and specific
+non-rigid deformation. Our key insight is that explicit shape can reduce the
+sampling points used to fit implicit representation, and frozen blending
+weights from SMPL constructing a generalized rigid deformation can effectively
+avoid overfitting and improve pose generalization performance. Our architecture
+involving both explicit and implicit representation is simple yet effective.
+Experiments demonstrate our model can synthesize images under arbitrary poses
+with few-shot input and increase the speed of synthesizing images by 15 times
+through a reduction in computational complexity without using any existing
+acceleration modules. Compared to the state-of-the-art HumanNeRF studies,
+HumanNeRF-SE achieves better performance with fewer learnable parameters and
+less training time.",cs.CV,['cs.CV']
+GALA: Generating Animatable Layered Assets from a Single Scan,Taeksoo Kim · Byungjun Kim · Shunsuke Saito · Hanbyul Joo, ,https://arxiv.org/abs/2401.12979,,2401.12979.pdf,GALA: Generating Animatable Layered Assets from a Single Scan,"We present GALA, a framework that takes as input a single-layer clothed 3D
+human mesh and decomposes it into complete multi-layered 3D assets. The outputs
+can then be combined with other assets to create novel clothed human avatars
+with any pose. Existing reconstruction approaches often treat clothed humans as
+a single-layer of geometry and overlook the inherent compositionality of humans
+with hairstyles, clothing, and accessories, thereby limiting the utility of the
+meshes for downstream applications. Decomposing a single-layer mesh into
+separate layers is a challenging task because it requires the synthesis of
+plausible geometry and texture for the severely occluded regions. Moreover,
+even with successful decomposition, meshes are not normalized in terms of poses
+and body shapes, failing coherent composition with novel identities and poses.
+To address these challenges, we propose to leverage the general knowledge of a
+pretrained 2D diffusion model as geometry and appearance prior for humans and
+other assets. We first separate the input mesh using the 3D surface
+segmentation extracted from multi-view 2D segmentations. Then we synthesize the
+missing geometry of different layers in both posed and canonical spaces using a
+novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
+inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
+texture to obtain the complete appearance including the initially occluded
+regions. Through a series of decomposition steps, we obtain multiple layers of
+3D assets in a shared canonical space normalized in terms of poses and human
+shapes, hence supporting effortless composition to novel identities and
+reanimation with novel poses. Our experiments demonstrate the effectiveness of
+our approach for decomposition, canonicalization, and composition tasks
+compared to existing solutions.",cs.CV,['cs.CV']
+A Vision Check-up for Language Models,Pratyusha Sharma · Tamar Rott Shaham · Manel Baradad · Stephanie Fu · Adrian Rodriguez-Munoz · Shivam Duggal · Phillip Isola · Antonio Torralba, ,https://arxiv.org/abs/2401.01862,,2401.01862.pdf,A Vision Check-up for Language Models,"What does learning to model relationships between strings teach large
+language models (LLMs) about the visual world? We systematically evaluate LLMs'
+abilities to generate and recognize an assortment of visual concepts of
+increasing complexity and then demonstrate how a preliminary visual
+representation learning system can be trained using models of text. As language
+models lack the ability to consume or output visual information as pixels, we
+use code to represent images in our study. Although LLM-generated images do not
+look like natural images, results on image generation and the ability of models
+to correct these generated images indicate that precise modeling of strings can
+teach language models about numerous aspects of the visual world. Furthermore,
+experiments on self-supervised visual representation learning, utilizing images
+generated with text models, highlight the potential to train vision models
+capable of making semantic assessments of natural images using just LLMs.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Boosting Adversarial Transferability by Block Shuffle and Rotation,Kunyu Wang · he xuanran · Wenxuan Wang · Xiaosen Wang, ,https://arxiv.org/abs/2308.10299,,2308.10299.pdf,Boosting Adversarial Transferability by Block Shuffle and Rotation,"Adversarial examples mislead deep neural networks with imperceptible
+perturbations and have brought significant threats to deep learning. An
+important aspect is their transferability, which refers to their ability to
+deceive other models, thus enabling attacks in the black-box setting. Though
+various methods have been proposed to boost transferability, the performance
+still falls short compared with white-box attacks. In this work, we observe
+that existing input transformation based attacks, one of the mainstream
+transfer-based attacks, result in different attention heatmaps on various
+models, which might limit the transferability. We also find that breaking the
+intrinsic relation of the image can disrupt the attention heatmap of the
+original image. Based on this finding, we propose a novel input transformation
+based attack called block shuffle and rotation (BSR). Specifically, BSR splits
+the input image into several blocks, then randomly shuffles and rotates these
+blocks to construct a set of new images for gradient calculation. Empirical
+evaluations on the ImageNet dataset demonstrate that BSR could achieve
+significantly better transferability than the existing input transformation
+based methods under single-model and ensemble-model settings. Combining BSR
+with the current input transformation method can further improve the
+transferability, which significantly outperforms the state-of-the-art methods.
+Code is available at https://github.com/Trustworthy-AI-Group/BSR",cs.CV,"['cs.CV', 'eess.IV']"
+Residual Learning in Diffusion Models,Junyu Zhang · Daochang Liu · Eunbyung Park · Shichao Zhang · Chang Xu, ,https://arxiv.org/abs/2308.13712,,2308.13712.pdf,Residual Denoising Diffusion Models,"We propose residual denoising diffusion models (RDDM), a novel dual diffusion
+process that decouples the traditional single denoising diffusion process into
+residual diffusion and noise diffusion. This dual diffusion framework expands
+the denoising-based diffusion models, initially uninterpretable for image
+restoration, into a unified and interpretable model for both image generation
+and restoration by introducing residuals. Specifically, our residual diffusion
+represents directional diffusion from the target image to the degraded input
+image and explicitly guides the reverse generation process for image
+restoration, while noise diffusion represents random perturbations in the
+diffusion process. The residual prioritizes certainty, while the noise
+emphasizes diversity, enabling RDDM to effectively unify tasks with varying
+certainty or diversity requirements, such as image generation and restoration.
+We demonstrate that our sampling process is consistent with that of DDPM and
+DDIM through coefficient transformation, and propose a partially
+path-independent generation process to better understand the reverse process.
+Notably, our RDDM enables a generic UNet, trained with only an L1 loss and a
+batch size of 1, to compete with state-of-the-art image restoration methods. We
+provide code and pre-trained models to encourage further exploration,
+application, and development of our innovative framework
+(https://github.com/nachifur/RDDM).",cs.CV,"['cs.CV', 'cs.LG']"
+"What, How, and When Should Object Detectors Update in Continually Changing Test Domains?",Jayeon Yoo · Dongkwan Lee · Inseop Chung · Donghyun Kim · Nojun Kwak, ,https://arxiv.org/abs/2312.08875,,2312.08875.pdf,"What, How, and When Should Object Detectors Update in Continually Changing Test Domains?","It is a well-known fact that the performance of deep learning models
+deteriorates when they encounter a distribution shift at test time. Test-time
+adaptation (TTA) algorithms have been proposed to adapt the model online while
+inferring test data. However, existing research predominantly focuses on
+classification tasks through the optimization of batch normalization layers or
+classification heads, but this approach limits its applicability to various
+model architectures like Transformers and makes it challenging to apply to
+other tasks, such as object detection. In this paper, we propose a novel online
+adaption approach for object detection in continually changing test domains,
+considering which part of the model to update, how to update it, and when to
+perform the update. By introducing architecture-agnostic and lightweight
+adaptor modules and only updating these while leaving the pre-trained backbone
+unchanged, we can rapidly adapt to new test domains in an efficient way and
+prevent catastrophic forgetting. Furthermore, we present a practical and
+straightforward class-wise feature aligning method for object detection to
+resolve domain shifts. Additionally, we enhance efficiency by determining when
+the model is sufficiently adapted or when additional adaptation is needed due
+to changes in the test distribution. Our approach surpasses baselines on widely
+used benchmarks, achieving improvements of up to 4.9\%p and 7.9\%p in mAP for
+COCO $\rightarrow$ COCO-corrupted and SHIFT, respectively, while maintaining
+about 20 FPS or higher.",cs.CV,['cs.CV']
+Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement,Daiwei Yu · Zhuorong Li · Lina Wei · Canghong Jin · Yun Zhang · Sixian Chan, ,https://arxiv.org/abs/2403.09101,,2403.09101.pdf,Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement,"Adversarial training (AT) is currently one of the most effective ways to
+obtain the robustness of deep neural networks against adversarial attacks.
+However, most AT methods suffer from robust overfitting, i.e., a significant
+generalization gap in adversarial robustness between the training and testing
+curves. In this paper, we first identify a connection between robust
+overfitting and the excessive memorization of noisy labels in AT from a view of
+gradient norm. As such label noise is mainly caused by a distribution mismatch
+and improper label assignments, we are motivated to propose a label refinement
+approach for AT. Specifically, our Self-Guided Label Refinement first
+self-refines a more accurate and informative label distribution from
+over-confident hard labels, and then it calibrates the training by dynamically
+incorporating knowledge from self-distilled models into the current model and
+thus requiring no external teachers. Empirical results demonstrate that our
+method can simultaneously boost the standard accuracy and robust performance
+across multiple benchmark datasets, attack types, and architectures. In
+addition, we also provide a set of analyses from the perspectives of
+information theory to dive into our method and suggest the importance of soft
+labels for robust generalization.",cs.LG,"['cs.LG', 'cs.CR', 'cs.CV']"
+SAOR: Single-View Articulated Object Reconstruction,Mehmet Aygun · Oisin Mac Aodha, ,,https://synthical.com/article/e8c0baeb-d277-4528-b526-8a08fcc46a22,,,,,nan
+Infrared Adversarial Car Stickers,Xiaopei Zhu · Yuqiu Liu · Zhanhao Hu · Jianmin Li · Xiaolin Hu, ,https://arxiv.org/abs/2405.09924,,2405.09924.pdf,Infrared Adversarial Car Stickers,"Infrared physical adversarial examples are of great significance for studying
+the security of infrared AI systems that are widely used in our lives such as
+autonomous driving. Previous infrared physical attacks mainly focused on 2D
+infrared pedestrian detection which may not fully manifest its destructiveness
+to AI systems. In this work, we propose a physical attack method against
+infrared detectors based on 3D modeling, which is applied to a real car. The
+goal is to design a set of infrared adversarial stickers to make cars invisible
+to infrared detectors at various viewing angles, distances, and scenes. We
+build a 3D infrared car model with real infrared characteristics and propose an
+infrared adversarial pattern generation method based on 3D mesh shadow. We
+propose a 3D control points-based mesh smoothing algorithm and use a set of
+smoothness loss functions to enhance the smoothness of adversarial meshes and
+facilitate the sticker implementation. Besides, We designed the aluminum
+stickers and conducted physical experiments on two real Mercedes-Benz A200L
+cars. Our adversarial stickers hid the cars from Faster RCNN, an object
+detector, at various viewing angles, distances, and scenes. The attack success
+rate (ASR) was 91.49% for real cars. In comparison, the ASRs of random stickers
+and no sticker were only 6.21% and 0.66%, respectively. In addition, the ASRs
+of the designed stickers against six unseen object detectors such as YOLOv3 and
+Deformable DETR were between 73.35%-95.80%, showing good transferability of the
+attack performance across detectors.",cs.CV,['cs.CV']
+Effective Video Mirror Detection with Inconsistent Motion Cues,Alex Warren · Ke Xu · Jiaying Lin · Gary Tam · Rynson W.H. Lau, ,,https://cronfa.swan.ac.uk/Record/cronfa65886/Details,,,,,nan
+Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer,Hyeongjin Nam · Daniel Jung · Gyeongsik Moon · Kyoung Mu Lee,https://github.com/dqj5182/CONTHO_RELEASE,https://arxiv.org/abs/2404.04819,,2404.04819.pdf,Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer,"Human-object contact serves as a strong cue to understand how humans
+physically interact with objects. Nevertheless, it is not widely explored to
+utilize human-object contact information for the joint reconstruction of 3D
+human and object from a single image. In this work, we present a novel joint 3D
+human-object reconstruction method (CONTHO) that effectively exploits contact
+information between humans and objects. There are two core designs in our
+system: 1) 3D-guided contact estimation and 2) contact-based 3D human and
+object refinement. First, for accurate human-object contact estimation, CONTHO
+initially reconstructs 3D humans and objects and utilizes them as explicit 3D
+guidance for contact estimation. Second, to refine the initial reconstructions
+of 3D human and object, we propose a novel contact-based refinement Transformer
+that effectively aggregates human features and object features based on the
+estimated human-object contact. The proposed contact-based refinement prevents
+the learning of erroneous correlation between human and object, which enables
+accurate 3D reconstruction. As a result, our CONTHO achieves state-of-the-art
+performance in both human-object contact estimation and joint reconstruction of
+3D human and object. The code is publicly available at
+https://github.com/dqj5182/CONTHO_RELEASE.",cs.CV,['cs.CV']
+SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model,Inhwan Bae · Young-Jae Park · Hae-Gon Jeon,https://github.com/InhwanBae/SingularTrajectory,https://arxiv.org/abs/2403.18452v1,,2403.18452v1.pdf,SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model,"There are five types of trajectory prediction tasks: deterministic,
+stochastic, domain adaptation, momentary observation, and few-shot. These
+associated tasks are defined by various factors, such as the length of input
+paths, data split and pre-processing methods. Interestingly, even though they
+commonly take sequential coordinates of observations as input and infer future
+paths in the same coordinates as output, designing specialized architectures
+for each task is still necessary. For the other task, generality issues can
+lead to sub-optimal performances. In this paper, we propose SingularTrajectory,
+a diffusion-based universal trajectory prediction framework to reduce the
+performance gap across the five tasks. The core of SingularTrajectory is to
+unify a variety of human dynamics representations on the associated tasks. To
+do this, we first build a Singular space to project all types of motion
+patterns from each task into one embedding space. We next propose an adaptive
+anchor working in the Singular space. Unlike traditional fixed anchor methods
+that sometimes yield unacceptable paths, our adaptive anchor enables correct
+anchors, which are put into a wrong location, based on a traversability map.
+Finally, we adopt a diffusion-based predictor to further enhance the prototype
+paths using a cascaded denoising process. Our unified framework ensures the
+generality across various benchmark settings such as input modality, and
+trajectory lengths. Extensive experiments on five public benchmarks demonstrate
+that SingularTrajectory substantially outperforms existing models, highlighting
+its effectiveness in estimating general dynamics of human movements. Code is
+publicly available at https://github.com/inhwanbae/SingularTrajectory .",cs.CV,"['cs.CV', 'cs.LG', 'cs.RO']"
+Gradient Reweighting: Towards Imbalanced Class-Incremental Learning,Jiangpeng He,https://github.com/JiangpengHe/imbalanced_cil,https://arxiv.org/abs/2402.18528,,2402.18528.pdf,Gradient Reweighting: Towards Imbalanced Class-Incremental Learning,"Class-Incremental Learning (CIL) trains a model to continually recognize new
+classes from non-stationary data while retaining learned knowledge. A major
+challenge of CIL arises when applying to real-world data characterized by
+non-uniform distribution, which introduces a dual imbalance problem involving
+(i) disparities between stored exemplars of old tasks and new class data
+(inter-phase imbalance), and (ii) severe class imbalances within each
+individual task (intra-phase imbalance). We show that this dual imbalance issue
+causes skewed gradient updates with biased weights in FC layers, thus inducing
+over/under-fitting and catastrophic forgetting in CIL. Our method addresses it
+by reweighting the gradients towards balanced optimization and unbiased
+classifier learning. Additionally, we observe imbalanced forgetting where
+paradoxically the instance-rich classes suffer higher performance degradation
+during CIL due to a larger amount of training data becoming unavailable in
+subsequent learning phases. To tackle this, we further introduce a
+distribution-aware knowledge distillation loss to mitigate forgetting by
+aligning output logits proportionally with the distribution of lost training
+data. We validate our method on CIFAR-100, ImageNetSubset, and Food101 across
+various evaluation protocols and demonstrate consistent improvements compared
+to existing works, showing great potential to apply CIL in real-world scenarios
+with enhanced robustness and effectiveness.",cs.CV,['cs.CV']
+OpenEQA: Embodied Question Answering in the Era of Foundation Models,Arjun Majumdar · Anurag Ajay · Xiaohan Zhang · Sriram Yenamandra · Mikael Henaff · Alexander Sax · Sneha Silwal · Paul McVay · Oleksandr Maksymets · Sergio Arnaud · Pranav Putta · Karmesh Yadav · Qiyang Li · Benjamin Newman · Mohit Sharma · Mohit Sharma · Vincent-Pierre Berges · Shiqi Zhang · Pulkit Agrawal · Dhruv Batra · Yonatan Bisk · Mrinal Kalakrishnan · Franziska Meier · Chris Paxton · Aravind Rajeswaran, ,,https://openreview.net/forum?id=7JIW6e1UJX,,,,,nan
+Batch Normalization Alleviates the Spectral Bias in Coordinate Networks,Zhicheng Cai · Hao Zhu · Qiu Shen · Xinran Wang · Xun Cao, ,https://arxiv.org/abs/2306.16999,,2306.16999.pdf,Spectral Batch Normalization: Normalization in the Frequency Domain,"Regularization is a set of techniques that are used to improve the
+generalization ability of deep neural networks. In this paper, we introduce
+spectral batch normalization (SBN), a novel effective method to improve
+generalization by normalizing feature maps in the frequency (spectral) domain.
+The activations of residual networks without batch normalization (BN) tend to
+explode exponentially in the depth of the network at initialization. This leads
+to extremely large feature map norms even though the parameters are relatively
+small. These explosive dynamics can be very detrimental to learning. BN makes
+weight decay regularization on the scaling factors $\gamma, \beta$
+approximately equivalent to an additive penalty on the norm of the feature
+maps, which prevents extremely large feature map norms to a certain degree.
+However, we show experimentally that, despite the approximate additive penalty
+of BN, feature maps in deep neural networks (DNNs) tend to explode at the
+beginning of the network and that feature maps of DNNs contain large values
+during the whole training. This phenomenon also occurs in a weakened form in
+non-residual networks. SBN addresses large feature maps by normalizing them in
+the frequency domain. In our experiments, we empirically show that SBN prevents
+exploding feature maps at initialization and large feature map values during
+the training. Moreover, the normalization of feature maps in the frequency
+domain leads to more uniform distributed frequency components. This discourages
+the DNNs to rely on single frequency components of feature maps. These,
+together with other effects of SBN, have a regularizing effect on the training
+of residual and non-residual networks. We show experimentally that using SBN in
+addition to standard regularization methods improves the performance of DNNs by
+a relevant margin, e.g. ResNet50 on ImageNet by 0.71%.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning for Transductive Threshold Calibration in Open-World Recognition,Qin ZHANG · DONGSHENG An · Tianjun Xiao · Tong He · Qingming Tang · Ying Nian Wu · Joseph Tighe · Yifan Xing, ,,https://synthical.com/summary/ed7531f5-2d4e-43c1-95e3-15ec48a9b43d,,,,,nan
+MatSynth: A Modern PBR Materials Dataset,Giuseppe Vecchio · Valentin Deschaintre,https://gvecchio.com/matsynth/,https://arxiv.org/abs/2401.06056,,2401.06056.pdf,MatSynth: A Modern PBR Materials Dataset,"We introduce MatSynth, a dataset of 4,000+ CC0 ultra-high resolution PBR
+materials. Materials are crucial components of virtual relightable assets,
+defining the interaction of light at the surface of geometries. Given their
+importance, significant research effort was dedicated to their representation,
+creation and acquisition. However, in the past 6 years, most research in
+material acquisiton or generation relied either on the same unique dataset, or
+on company-owned huge library of procedural materials. With this dataset we
+propose a significantly larger, more diverse, and higher resolution set of
+materials than previously publicly available. We carefully discuss the data
+collection process and demonstrate the benefits of this dataset on material
+acquisition and generation applications. The complete data further contains
+metadata with each material's origin, license, category, tags, creation method
+and, when available, descriptions and physical size, as well as 3M+ renderings
+of the augmented materials, in 1K, under various environment lightings. The
+MatSynth dataset is released through the project page at:
+https://www.gvecchio.com/matsynth.",cs.CV,"['cs.CV', 'cs.GR']"
+Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation,Xiaoyang Chen · Hao Zheng · Yuemeng LI · Yuncong Ma · Liang Ma · Hongming Li · Yong Fan, ,https://arxiv.org/abs/2311.10696,,2311.10696.pdf,Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation,"A versatile medical image segmentation model applicable to images acquired
+with diverse equipment and protocols can facilitate model deployment and
+maintenance. However, building such a model typically demands a large, diverse,
+and fully annotated dataset, which is challenging to obtain due to the
+labor-intensive nature of data curation. To address this challenge, we propose
+a cost-effective alternative that harnesses multi-source data with only partial
+or sparse segmentation labels for training, substantially reducing the cost of
+developing a versatile model. We devise strategies for model
+self-disambiguation, prior knowledge incorporation, and imbalance mitigation to
+tackle challenges associated with inconsistently labeled multi-source data,
+including label ambiguity and modality, dataset, and class imbalances.
+Experimental results on a multi-modal dataset compiled from eight different
+sources for abdominal structure segmentation have demonstrated the
+effectiveness and superior performance of our method compared to
+state-of-the-art alternative approaches. We anticipate that its cost-saving
+features, which optimize the utilization of existing annotated data and reduce
+annotation efforts for new data, will have a significant impact in the field.",cs.CV,['cs.CV']
+ASAM: Boosting Segment Anything Model with Adversarial Tuning,Bo Li · Haoke Xiao · Lv Tang, ,https://arxiv.org/abs/2405.00256,,2405.00256.pdf,ASAM: Boosting Segment Anything Model with Adversarial Tuning,"In the evolving landscape of computer vision, foundation models have emerged
+as pivotal tools, exhibiting exceptional adaptability to a myriad of tasks.
+Among these, the Segment Anything Model (SAM) by Meta AI has distinguished
+itself in image segmentation. However, SAM, like its counterparts, encounters
+limitations in specific niche applications, prompting a quest for enhancement
+strategies that do not compromise its inherent capabilities. This paper
+introduces ASAM, a novel methodology that amplifies SAM's performance through
+adversarial tuning. We harness the potential of natural adversarial examples,
+inspired by their successful implementation in natural language processing. By
+utilizing a stable diffusion model, we augment a subset (1%) of the SA-1B
+dataset, generating adversarial instances that are more representative of
+natural variations rather than conventional imperceptible perturbations. Our
+approach maintains the photorealism of adversarial examples and ensures
+alignment with original mask annotations, thereby preserving the integrity of
+the segmentation task. The fine-tuned ASAM demonstrates significant
+improvements across a diverse range of segmentation tasks without necessitating
+additional data or architectural modifications. The results of our extensive
+evaluations confirm that ASAM establishes new benchmarks in segmentation tasks,
+thereby contributing to the advancement of foundational models in computer
+vision. Our project page is in https://asam2024.github.io/.",cs.CV,['cs.CV']
+FreeDrag: Feature Dragging for Reliable Point-based Image Editing,Pengyang Ling · Lin Chen · Pan Zhang · Huaian Chen · Yi Jin · Jinjin Zheng, ,https://arxiv.org/abs/2307.04684,,2307.04684.pdf,FreeDrag: Feature Dragging for Reliable Point-based Image Editing,"To serve the intricate and varied demands of image editing, precise and
+flexible manipulation in image content is indispensable. Recently, Drag-based
+editing methods have gained impressive performance. However, these methods
+predominantly center on point dragging, resulting in two noteworthy drawbacks,
+namely ""miss tracking"", where difficulties arise in accurately tracking the
+predetermined handle points, and ""ambiguous tracking"", where tracked points are
+potentially positioned in wrong regions that closely resemble the handle
+points. To address the above issues, we propose FreeDrag, a feature dragging
+methodology designed to free the burden on point tracking. The FreeDrag
+incorporates two key designs, i.e., template feature via adaptive updating and
+line search with backtracking, the former improves the stability against
+drastic content change by elaborately controls feature updating scale after
+each dragging, while the latter alleviates the misguidance from similar points
+by actively restricting the search area in a line. These two technologies
+together contribute to a more stable semantic dragging with higher efficiency.
+Comprehensive experimental results substantiate that our approach significantly
+outperforms pre-existing methodologies, offering reliable point-based editing
+even in various complex scenarios.",cs.CV,"['cs.CV', 'cs.HC', 'cs.LG']"
+ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization,Weiyao Wang · Pierre Gleize · Hao Tang · Xingyu Chen · Kevin Liang · Matt Feiszli, ,https://arxiv.org/abs/2401.08937,,2401.08937.pdf,ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization,"Neural Radiance Fields (NeRF) exhibit remarkable performance for Novel View
+Synthesis (NVS) given a set of 2D images. However, NeRF training requires
+accurate camera pose for each input view, typically obtained by
+Structure-from-Motion (SfM) pipelines. Recent works have attempted to relax
+this constraint, but they still often rely on decent initial poses which they
+can refine. Here we aim at removing the requirement for pose initialization. We
+present Incremental CONfidence (ICON), an optimization procedure for training
+NeRFs from 2D video frames. ICON only assumes smooth camera motion to estimate
+initial guess for poses. Further, ICON introduces ``confidence"": an adaptive
+measure of model quality used to dynamically reweight gradients. ICON relies on
+high-confidence poses to learn NeRF, and high-confidence 3D structure (as
+encoded by NeRF) to learn poses. We show that ICON, without prior pose
+initialization, achieves superior performance in both CO3D and HO3D versus
+methods which use SfM pose.",cs.CV,['cs.CV']
+StyLitGAN: Image-based Relighting via Latent Control,Anand Bhattad · James Soole · David Forsyth, ,https://ar5iv.labs.arxiv.org/html/2306.00987,,2306.00987.pdf,"StyleGAN knows Normal, Depth, Albedo, and More","Intrinsic images, in the original sense, are image-like maps of scene
+properties like depth, normal, albedo or shading. This paper demonstrates that
+StyleGAN can easily be induced to produce intrinsic images. The procedure is
+straightforward. We show that, if StyleGAN produces $G({w})$ from latents
+${w}$, then for each type of intrinsic image, there is a fixed offset ${d}_c$
+so that $G({w}+{d}_c)$ is that type of intrinsic image for $G({w})$. Here
+${d}_c$ is {\em independent of ${w}$}. The StyleGAN we used was pretrained by
+others, so this property is not some accident of our training regime. We show
+that there are image transformations StyleGAN will {\em not} produce in this
+fashion, so StyleGAN is not a generic image regression engine.
+  It is conceptually exciting that an image generator should ``know'' and
+represent intrinsic images. There may also be practical advantages to using a
+generative model to produce intrinsic images. The intrinsic images obtained
+from StyleGAN compare well both qualitatively and quantitatively with those
+obtained by using SOTA image regression techniques; but StyleGAN's intrinsic
+images are robust to relighting effects, unlike SOTA methods.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Single Mesh Diffusion Models with Field Latents for Texture Generation,Thomas W. Mitchel · Carlos Esteves · Ameesh Makadia,https://single-mesh-diffusion.github.io/,https://arxiv.org/abs/2312.09250,,2312.09250.pdf,Single Mesh Diffusion Models with Field Latents for Texture Generation,"We introduce a framework for intrinsic latent diffusion models operating
+directly on the surfaces of 3D shapes, with the goal of synthesizing
+high-quality textures. Our approach is underpinned by two contributions: field
+latents, a latent representation encoding textures as discrete vector fields on
+the mesh vertices, and field latent diffusion models, which learn to denoise a
+diffusion process in the learned latent space on the surface. We consider a
+single-textured-mesh paradigm, where our models are trained to generate
+variations of a given texture on a mesh. We show the synthesized textures are
+of superior fidelity compared those from existing single-textured-mesh
+generative models. Our models can also be adapted for user-controlled editing
+tasks such as inpainting and label-guided generation. The efficacy of our
+approach is due in part to the equivariance of our proposed framework under
+isometries, allowing our models to seamlessly reproduce details across locally
+similar regions and opening the door to a notion of generative texture
+transfer.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Label-Efficient Group Robustness via Out-of-Distribution Concept Curation,Yiwei Yang · Anthony Liu · Robert Wolfe · Aylin Caliskan · Bill Howe, ,https://arxiv.org/abs/2403.06392,,2403.06392.pdf,Towards Robust Out-of-Distribution Generalization Bounds via Sharpness,"Generalizing to out-of-distribution (OOD) data or unseen domain, termed OOD
+generalization, still lacks appropriate theoretical guarantees. Canonical OOD
+bounds focus on different distance measurements between source and target
+domains but fail to consider the optimization property of the learned model. As
+empirically shown in recent work, the sharpness of learned minima influences
+OOD generalization. To bridge this gap between optimization and OOD
+generalization, we study the effect of sharpness on how a model tolerates data
+change in domain shift which is usually captured by ""robustness"" in
+generalization. In this paper, we give a rigorous connection between sharpness
+and robustness, which gives better OOD guarantees for robust algorithms. It
+also provides a theoretical backing for ""flat minima leads to better OOD
+generalization"". Overall, we propose a sharpness-based OOD generalization bound
+by taking robustness into consideration, resulting in a tighter bound than
+non-robust guarantees. Our findings are supported by the experiments on a ridge
+regression model, as well as the experiments on deep learning classification
+tasks.",cs.LG,['cs.LG']
+EventPS: Real-Time Photometric Stereo Using an Event Camera,Bohan Yu · Jieji Ren · Jin Han · Feishi Wang · Jinxiu Liang · Boxin Shi, ,https://arxiv.org/abs/2312.11911,,2312.11911.pdf,"EVI-SAM: Robust, Real-time, Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping","Event cameras are bio-inspired, motion-activated sensors that demonstrate
+substantial potential in handling challenging situations, such as motion blur
+and high-dynamic range. In this paper, we proposed EVI-SAM to tackle the
+problem of 6 DoF pose tracking and 3D reconstruction using monocular event
+camera. A novel event-based hybrid tracking framework is designed to estimate
+the pose, leveraging the robustness of feature matching and the precision of
+direct alignment. Specifically, we develop an event-based 2D-2D alignment to
+construct the photometric constraint, and tightly integrate it with the
+event-based reprojection constraint. The mapping module recovers the dense and
+colorful depth of the scene through the image-guided event-based mapping
+method. Subsequently, the appearance, texture, and surface mesh of the 3D scene
+can be reconstructed by fusing the dense depth map from multiple viewpoints
+using truncated signed distance function (TSDF) fusion. To the best of our
+knowledge, this is the first non-learning work to realize event-based dense
+mapping. Numerical evaluations are performed on both publicly available and
+self-collected datasets, which qualitatively and quantitatively demonstrate the
+superior performance of our method. Our EVI-SAM effectively balances accuracy
+and robustness while maintaining computational efficiency, showcasing superior
+pose tracking and dense mapping performance in challenging scenarios. Video
+Demo: https://youtu.be/Nn40U4e5Si8.",cs.CV,"['cs.CV', 'cs.RO']"
+Towards Understanding and Improving Adversarial Robustness of Vision Transformers,Samyak Jain · Tanima Dutta, ,https://arxiv.org/html/2208.09602v2,,,Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective,"The Vision Transformer has emerged as a powerful tool for image
+classification tasks, surpassing the performance of convolutional neural
+networks (CNNs). Recently, many researchers have attempted to understand the
+robustness of Transformers against adversarial attacks. However, previous
+researches have focused solely on perturbations in the spatial domain. This
+paper proposes an additional perspective that explores the adversarial
+robustness of Transformers against frequency-selective perturbations in the
+spectral domain. To facilitate comparison between these two domains, an attack
+framework is formulated as a flexible tool for implementing attacks on images
+in the spatial and spectral domains. The experiments reveal that Transformers
+rely more on phase and low frequency information, which can render them more
+vulnerable to frequency-selective attacks than CNNs. This work offers new
+insights into the properties and adversarial robustness of Transformers.",cs.CV,['cs.CV']
+On Train-Test Class Overlap and Detection for Image Retrieval,Chull Hwan Song · Jooyoung Yoon · Taebaek Hwang · Shunghyun Choi · Yeong Hyeon Gu · Yannis Avrithis, ,https://arxiv.org/abs/2404.01524,,2404.01524.pdf,On Train-Test Class Overlap and Detection for Image Retrieval,"How important is it for training and evaluation sets to not have class
+overlap in image retrieval? We revisit Google Landmarks v2 clean, the most
+popular training set, by identifying and removing class overlap with Revisited
+Oxford and Paris [34], the most popular evaluation set. By comparing the
+original and the new RGLDv2-clean on a benchmark of reproduced state-of-the-art
+methods, our findings are striking. Not only is there a dramatic drop in
+performance, but it is inconsistent across methods, changing the ranking.What
+does it take to focus on objects or interest and ignore background clutter when
+indexing? Do we need to train an object detector and the representation
+separately? Do we need location supervision? We introduce Single-stage
+Detect-to-Retrieve (CiDeR), an end-to-end, single-stage pipeline to detect
+objects of interest and extract a global image representation. We outperform
+previous state-of-the-art on both existing training sets and the new
+RGLDv2-clean. Our dataset is available at
+https://github.com/dealicious-inc/RGLDv2-clean.",cs.CV,"['cs.CV', 'cs.AI']"
+Semantic Line Combination Detector,JINWON KO · Dongkwon Jin · Chang-Su Kim, ,https://arxiv.org/abs/2404.18399,,2404.18399.pdf,Semantic Line Combination Detector,"A novel algorithm, called semantic line combination detector (SLCD), to find
+an optimal combination of semantic lines is proposed in this paper. It
+processes all lines in each line combination at once to assess the overall
+harmony of the lines. First, we generate various line combinations from
+reliable lines. Second, we estimate the score of each line combination and
+determine the best one. Experimental results demonstrate that the proposed SLCD
+outperforms existing semantic line detectors on various datasets. Moreover, it
+is shown that SLCD can be applied effectively to three vision tasks of
+vanishing point detection, symmetry axis detection, and composition-based image
+retrieval. Our codes are available at https://github.com/Jinwon-Ko/SLCD.",cs.CV,['cs.CV']
+Robust Noisy Correspondence Learning with Equivariant Similarity Consistency,Yuchen Yang · Erkun Yang · Likai Wang · Cheng Deng, ,,https://dl.acm.org/doi/10.1145/3662732,,,,,nan
+Event-based Structure-from-Orbit,Ethan Elms · Yasir Latif · Tae Ha Park · Tat-Jun Chin, ,https://arxiv.org/abs/2405.06216,,2405.06216.pdf,Event-based Structure-from-Orbit,"Event sensors offer high temporal resolution visual sensing, which makes them
+ideal for perceiving fast visual phenomena without suffering from motion blur.
+Certain applications in robotics and vision-based navigation require 3D
+perception of an object undergoing circular or spinning motion in front of a
+static camera, such as recovering the angular velocity and shape of the object.
+The setting is equivalent to observing a static object with an orbiting camera.
+In this paper, we propose event-based structure-from-orbit (eSfO), where the
+aim is to simultaneously reconstruct the 3D structure of a fast spinning object
+observed from a static event camera, and recover the equivalent orbital motion
+of the camera. Our contributions are threefold: since state-of-the-art event
+feature trackers cannot handle periodic self-occlusion due to the spinning
+motion, we develop a novel event feature tracker based on spatio-temporal
+clustering and data association that can better track the helical trajectories
+of valid features in the event data. The feature tracks are then fed to our
+novel factor graph-based structure-from-orbit back-end that calculates the
+orbital motion parameters (e.g., spin rate, relative rotational axis) that
+minimize the reprojection error. For evaluation, we produce a new event dataset
+of objects under spinning motion. Comparisons against ground truth indicate the
+efficacy of eSfO.",cs.CV,['cs.CV']
+HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions,Hao Xu · Li Haipeng · Yinqiao Wang · Shuaicheng Liu · Chi-Wing Fu, ,https://arxiv.org/abs/2403.18575,,2403.18575.pdf,HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions,"Reconstructing 3D hand mesh robustly from a single image is very challenging,
+due to the lack of diversity in existing real-world datasets. While data
+synthesis helps relieve the issue, the syn-to-real gap still hinders its usage.
+In this work, we present HandBooster, a new approach to uplift the data
+diversity and boost the 3D hand-mesh reconstruction performance by training a
+conditional generative space on hand-object interactions and purposely sampling
+the space to synthesize effective data samples. First, we construct versatile
+content-aware conditions to guide a diffusion model to produce realistic images
+with diverse hand appearances, poses, views, and backgrounds; favorably,
+accurate 3D annotations are obtained for free. Then, we design a novel
+condition creator based on our similarity-aware distribution sampling
+strategies to deliberately find novel and realistic interaction poses that are
+distinctive from the training set. Equipped with our method, several baselines
+can be significantly improved beyond the SOTA on the HO3D and DexYCB
+benchmarks. Our code will be released on
+https://github.com/hxwork/HandBooster_Pytorch.",cs.CV,['cs.CV']
+Customization Assistant for Text-to-image Generation,Yufan Zhou · Ruiyi Zhang · Jiuxiang Gu · Tong Sun, ,https://arxiv.org/abs/2312.03045,,2312.03045.pdf,Customization Assistant for Text-to-image Generation,"Customizing pre-trained text-to-image generation model has attracted massive
+research interest recently, due to its huge potential in real-world
+applications. Although existing methods are able to generate creative content
+for a novel concept contained in single user-input image, their capability are
+still far from perfection. Specifically, most existing methods require
+fine-tuning the generative model on testing images. Some existing methods do
+not require fine-tuning, while their performance are unsatisfactory.
+Furthermore, the interaction between users and models are still limited to
+directive and descriptive prompts such as instructions and captions. In this
+work, we build a customization assistant based on pre-trained large language
+model and diffusion model, which can not only perform customized generation in
+a tuning-free manner, but also enable more user-friendly interactions: users
+can chat with the assistant and input either ambiguous text or clear
+instruction. Specifically, we propose a new framework consists of a new model
+design and a novel training strategy. The resulting assistant can perform
+customized generation in 2-5 seconds without any test time fine-tuning.
+Extensive experiments are conducted, competitive results have been obtained
+across different domains, illustrating the effectiveness of the proposed
+method.",cs.CV,['cs.CV']
+Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering,Jiawei Yao · Qi Qian · Juhua Hu,https://github.com/Alexander-Yao/Multi-MaP,https://arxiv.org/abs/2404.15655,,2404.15655.pdf,Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering,"Multiple clustering has gained significant attention in recent years due to
+its potential to reveal multiple hidden structures of data from different
+perspectives. The advent of deep multiple clustering techniques has notably
+advanced the performance by uncovering complex patterns and relationships
+within large datasets. However, a major challenge arises as users often do not
+need all the clusterings that algorithms generate, and figuring out the one
+needed requires a substantial understanding of each clustering result.
+Traditionally, aligning a user's brief keyword of interest with the
+corresponding vision components was challenging, but the emergence of
+multi-modal and large language models (LLMs) has begun to bridge this gap. In
+response, given unlabeled target visual data, we propose Multi-MaP, a novel
+method employing a multi-modal proxy learning process. It leverages CLIP
+encoders to extract coherent text and image embeddings, with GPT-4 integrating
+users' interests to formulate effective textual contexts. Moreover, reference
+word constraint and concept-level constraint are designed to learn the optimal
+text proxy according to the user's interest. Multi-MaP not only adeptly
+captures a user's interest via a keyword but also facilitates identifying
+relevant clusterings. Our extensive experiments show that Multi-MaP
+consistently outperforms state-of-the-art methods in all benchmark
+multi-clustering vision tasks. Our code is available at
+https://github.com/Alexander-Yao/Multi-MaP.",cs.CV,['cs.CV']
+Anchor-based Robust Finetuning of Vision-Language Models,Jinwei Han · Zhiwen Lin · Zhongyisun Sun · Yingguo Gao · Ke Yan · Shouhong Ding · Yuan Gao · Gui-Song Xia,https://github.com/LixDemon/ARF,https://arxiv.org/abs/2404.06244,,2404.06244.pdf,Anchor-based Robust Finetuning of Vision-Language Models,"We aim at finetuning a vision-language model without hurting its
+out-of-distribution (OOD) generalization. We address two types of OOD
+generalization, i.e., i) domain shift such as natural to sketch images, and ii)
+zero-shot capability to recognize the category that was not contained in the
+finetune data. Arguably, the diminished OOD generalization after finetuning
+stems from the excessively simplified finetuning target, which only provides
+the class information, such as ``a photo of a [CLASS]''. This is distinct from
+the process in that CLIP was pretrained, where there is abundant text
+supervision with rich semantic information. Therefore, we propose to compensate
+for the finetune process using auxiliary supervision with rich semantic
+information, which acts as anchors to preserve the OOD generalization.
+Specifically, two types of anchors are elaborated in our method, including i)
+text-compensated anchor which uses the images from the finetune set but
+enriches the text supervision from a pretrained captioner, ii) image-text-pair
+anchor which is retrieved from the dataset similar to pretraining data of CLIP
+according to the downstream task, associating with the original CLIP text with
+rich semantics. Those anchors are utilized as auxiliary semantic information to
+maintain the original feature space of CLIP, thereby preserving the OOD
+generalization capabilities. Comprehensive experiments demonstrate that our
+method achieves in-distribution performance akin to conventional finetuning
+while attaining new state-of-the-art results on domain shift and zero-shot
+learning benchmarks.",cs.CV,['cs.CV']
+LEAD: Exploring Logit Space Evolution for Model Selection,Zixuan Hu · Xiaotong Li · SHIXIANG TANG · Jun Liu · Yichun Hu · Ling-Yu Duan, ,https://arxiv.org/abs/2308.15074,,2308.15074.pdf,Exploring Model Transferability through the Lens of Potential Energy,"Transfer learning has become crucial in computer vision tasks due to the vast
+availability of pre-trained deep learning models. However, selecting the
+optimal pre-trained model from a diverse pool for a specific downstream task
+remains a challenge. Existing methods for measuring the transferability of
+pre-trained models rely on statistical correlations between encoded static
+features and task labels, but they overlook the impact of underlying
+representation dynamics during fine-tuning, leading to unreliable results,
+especially for self-supervised models. In this paper, we present an insightful
+physics-inspired approach named PED to address these challenges. We reframe the
+challenge of model selection through the lens of potential energy and directly
+model the interaction forces that influence fine-tuning dynamics. By capturing
+the motion of dynamic representations to decline the potential energy within a
+force-driven physical model, we can acquire an enhanced and more stable
+observation for estimating transferability. The experimental results on 10
+downstream tasks and 12 self-supervised models demonstrate that our approach
+can seamlessly integrate into existing ranking techniques and enhance their
+performances, revealing its effectiveness for the model selection task and its
+potential for understanding the mechanism in transfer learning. Code will be
+available at https://github.com/lixiaotong97/PED.",cs.CV,"['cs.CV', 'cs.LG']"
+Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions,Weizhen He · Yiheng Deng · SHIXIANG TANG · Qihao CHEN · Qingsong Xie · Yizhou Wang · Lei Bai · Feng Zhu · Rui Zhao · Wanli Ouyang · Donglian Qi · Yunfeng Yan, ,https://arxiv.org/abs/2306.07520,,2306.07520.pdf,Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions,"Human intelligence can retrieve any person according to both visual and
+language descriptions. However, the current computer vision community studies
+specific person re-identification (ReID) tasks in different scenarios
+separately, which limits the applications in the real world. This paper strives
+to resolve this problem by proposing a new instruct-ReID task that requires the
+model to retrieve images according to the given image or language instructions.
+Our instruct-ReID is a more general ReID setting, where existing 6 ReID tasks
+can be viewed as special cases by designing different instructions. We propose
+a large-scale OmniReID benchmark and an adaptive triplet loss as a baseline
+method to facilitate research in this new setting. Experimental results show
+that the proposed multi-purpose ReID model, trained on our OmniReID benchmark
+without fine-tuning, can improve +0.5%, +0.6%, +7.7% mAP on Market1501, MSMT17,
+CUHK03 for traditional ReID, +6.4%, +7.1%, +11.2% mAP on PRCC, VC-Clothes, LTCC
+for clothes-changing ReID, +11.7% mAP on COCAS+ real2 for clothes template
+based clothes-changing ReID when using only RGB images, +24.9% mAP on COCAS+
+real2 for our newly defined language-instructed ReID, +4.3% on LLCM for
+visible-infrared ReID, +2.6% on CUHK-PEDES for text-to-image ReID. The
+datasets, the model, and code will be available at
+https://github.com/hwz-zju/Instruct-ReID.",cs.CV,['cs.CV']
+Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation,Siteng Huang · Biao Gong · Yutong Feng · Xi Chen · Yuqian Fu · Yu Liu · Donglin Wang, ,https://arxiv.org/abs/2311.15841,,2311.15841.pdf,Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation,"This study focuses on a novel task in text-to-image (T2I) generation, namely
+action customization. The objective of this task is to learn the co-existing
+action from limited data and generalize it to unseen humans or even animals.
+Experimental results show that existing subject-driven customization methods
+fail to learn the representative characteristics of actions and struggle in
+decoupling actions from context features, including appearance. To overcome the
+preference for low-level features and the entanglement of high-level features,
+we propose an inversion-based method Action-Disentangled Identifier (ADI) to
+learn action-specific identifiers from the exemplar images. ADI first expands
+the semantic conditioning space by introducing layer-wise identifier tokens,
+thereby increasing the representational richness while distributing the
+inversion across different features. Then, to block the inversion of
+action-agnostic features, ADI extracts the gradient invariance from the
+constructed sample triples and masks the updates of irrelevant channels. To
+comprehensively evaluate the task, we present an ActionBench that includes a
+variety of actions, each accompanied by meticulously selected samples. Both
+quantitative and qualitative results show that our ADI outperforms existing
+baselines in action-customized T2I generation. Our project page is at
+https://adi-t2i.github.io/ADI.",cs.CV,['cs.CV']
+ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models,Xinyu Tian · Shu Zou · Zhaoyuan Yang · Jing Zhang, ,https://arxiv.org/abs/2311.16494,,2311.16494.pdf,ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models,"Although soft prompt tuning is effective in efficiently adapting
+Vision-Language (V&L) models for downstream tasks, it shows limitations in
+dealing with distribution shifts. We address this issue with Attribute-Guided
+Prompt Tuning (ArGue), making three key contributions. 1) In contrast to the
+conventional approach of directly appending soft prompts preceding class names,
+we align the model with primitive visual attributes generated by Large Language
+Models (LLMs). We posit that a model's ability to express high confidence in
+these attributes signifies its capacity to discern the correct class
+rationales. 2) We introduce attribute sampling to eliminate disadvantageous
+attributes, thus only semantically meaningful attributes are preserved. 3) We
+propose negative prompting, explicitly enumerating class-agnostic attributes to
+activate spurious correlations and encourage the model to generate highly
+orthogonal probability distributions in relation to these negative features. In
+experiments, our method significantly outperforms current state-of-the-art
+prompt tuning methods on both novel class prediction and out-of-distribution
+generalization tasks.",cs.CV,['cs.CV']
+Narrative Action Evaluation with Prompt-Guided Multimodal Interaction,Shiyi Zhang · Sule Bai · Guangyi Chen · Lei Chen · Jiwen Lu · Junle Wang · Yansong Tang,https://github.com/shiyi-zh0408/NAE_CVPR2024,https://arxiv.org/abs/2404.14471,,2404.14471.pdf,Narrative Action Evaluation with Prompt-Guided Multimodal Interaction,"In this paper, we investigate a new problem called narrative action
+evaluation (NAE). NAE aims to generate professional commentary that evaluates
+the execution of an action. Unlike traditional tasks such as score-based action
+quality assessment and video captioning involving superficial sentences, NAE
+focuses on creating detailed narratives in natural language. These narratives
+provide intricate descriptions of actions along with objective evaluations. NAE
+is a more challenging task because it requires both narrative flexibility and
+evaluation rigor. One existing possible solution is to use multi-task learning,
+where narrative language and evaluative information are predicted separately.
+However, this approach results in reduced performance for individual tasks
+because of variations between tasks and differences in modality between
+language information and evaluation information. To address this, we propose a
+prompt-guided multimodal interaction framework. This framework utilizes a pair
+of transformers to facilitate the interaction between different modalities of
+information. It also uses prompts to transform the score regression task into a
+video-text matching task, thus enabling task interactivity. To support further
+research in this field, we re-annotate the MTL-AQA and FineGym datasets with
+high-quality and comprehensive action narration. Additionally, we establish
+benchmarks for NAE. Extensive experiment results prove that our method
+outperforms separate learning methods and naive multi-task learning methods.
+Data and code are released at https://github.com/shiyi-zh0408/NAE_CVPR2024.",cs.CV,['cs.CV']
+Improved Implicit Neural Representation with Fourier Reparameterized Training,Kexuan Shi · Xingyu Zhou · Shuhang Gu, ,https://arxiv.org/abs/2401.07402,,2401.07402.pdf,Improved Implicit Neural Representation with Fourier Bases Reparameterized Training,"Implicit Neural Representation (INR) as a mighty representation paradigm has
+achieved success in various computer vision tasks recently. Due to the
+low-frequency bias issue of vanilla multi-layer perceptron (MLP), existing
+methods have investigated advanced techniques, such as positional encoding and
+periodic activation function, to improve the accuracy of INR. In this paper, we
+connect the network training bias with the reparameterization technique and
+theoretically prove that weight reparameterization could provide us a chance to
+alleviate the spectral bias of MLP. Based on our theoretical analysis, we
+propose a Fourier reparameterization method which learns coefficient matrix of
+fixed Fourier bases to compose the weights of MLP. We evaluate the proposed
+Fourier reparameterization method on different INR tasks with various MLP
+architectures, including vanilla MLP, MLP with positional encoding and MLP with
+advanced activation function, etc. The superiority approximation results on
+different MLP architectures clearly validate the advantage of our proposed
+method. Armed with our Fourier reparameterization method, better INR with more
+textures and less artifacts can be learned from the training data.",cs.CV,['cs.CV']
+"Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications",Karren Yang · Anurag Ranjan · Jen-Hao Rick Chang · Raviteja Vemulapalli · Oncel Tuzel, ,https://arxiv.org/abs/2311.18168,,2311.18168.pdf,"Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications","We consider the task of animating 3D facial geometry from speech signal.
+Existing works are primarily deterministic, focusing on learning a one-to-one
+mapping from speech signal to 3D face meshes on small datasets with limited
+speakers. While these models can achieve high-quality lip articulation for
+speakers in the training set, they are unable to capture the full and diverse
+distribution of 3D facial motions that accompany speech in the real world.
+Importantly, the relationship between speech and facial motion is one-to-many,
+containing both inter-speaker and intra-speaker variations and necessitating a
+probabilistic approach. In this paper, we identify and address key challenges
+that have so far limited the development of probabilistic models: lack of
+datasets and metrics that are suitable for training and evaluating them, as
+well as the difficulty of designing a model that generates diverse results
+while remaining faithful to a strong conditioning signal as speech. We first
+propose large-scale benchmark datasets and metrics suitable for probabilistic
+modeling. Then, we demonstrate a probabilistic model that achieves both
+diversity and fidelity to speech, outperforming other methods across the
+proposed benchmarks. Finally, we showcase useful applications of probabilistic
+models trained on these large-scale datasets: we can generate diverse
+speech-driven 3D facial motion that matches unseen speaker styles extracted
+from reference clips; and our synthetic meshes can be used to improve the
+performance of downstream audio-visual models.",cs.CV,"['cs.CV', 'cs.LG', 'eess.AS']"
+EFormer: Enhanced Transformer towards Semantic-Contour Features of Foreground for Portraits Matting,Zitao Wang · Qiguang Miao · Yue Xi · Peipei Zhao, ,https://arxiv.org/abs/2308.12831,,2308.12831.pdf,EFormer: Enhanced Transformer towards Semantic-Contour Features of Foreground for Portraits Matting,"The portrait matting task aims to extract an alpha matte with complete
+semantics and finely-detailed contours. In comparison to CNN-based approaches,
+transformers with self-attention module have a better capacity to capture
+long-range dependencies and low-frequency semantic information of a portrait.
+However, the recent research shows that self-attention mechanism struggles with
+modeling high-frequency contour information and capturing fine contour details,
+which can lead to bias while predicting the portrait's contours. To deal with
+this issue, we propose EFormer to enhance the model's attention towards both of
+the low-frequency semantic and high-frequency contour features. For the
+high-frequency contours, our research demonstrates that cross-attention module
+between different resolutions can guide our model to allocate attention
+appropriately to these contour regions. Supported on this, we can successfully
+extract the high-frequency detail information around the portrait's contours,
+which are previously ignored by self-attention. Based on cross-attention
+module, we further build a semantic and contour detector (SCD) to accurately
+capture both of the low-frequency semantic and high-frequency contour features.
+And we design contour-edge extraction branch and semantic extraction branch to
+extract refined high-frequency contour features and complete low-frequency
+semantic information, respectively. Finally, we fuse the two kinds of features
+and leverage segmentation head to generate a predicted portrait matte.
+Experiments on VideoMatte240K (JPEG SD Format) and Adobe Image Matting (AIM)
+datasets demonstrate that EFormer outperforms previous portrait matte methods.",cs.CV,['cs.CV']
+Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation,Bingfeng Zhang · Siyue Yu · Yunchao Wei · Yao Zhao · Jimin Xiao, ,https://arxiv.org/html/2405.14294v1,,2405.14294v1.pdf,Tuning-free Universally-Supervised Semantic Segmentation,"This work presents a tuning-free semantic segmentation framework based on
+classifying SAM masks by CLIP, which is universally applicable to various types
+of supervision. Initially, we utilize CLIP's zero-shot classification ability
+to generate pseudo-labels or perform open-vocabulary segmentation. However, the
+misalignment between mask and CLIP text embeddings leads to suboptimal results.
+To address this issue, we propose discrimination-bias aligned CLIP to closely
+align mask and text embedding, offering an overhead-free performance gain. We
+then construct a global-local consistent classifier to classify SAM masks,
+which reveals the intrinsic structure of high-quality embeddings produced by
+DBA-CLIP and demonstrates robustness against noisy pseudo-labels. Extensive
+experiments validate the efficiency and effectiveness of our method, and we
+achieve state-of-the-art (SOTA) or competitive performance across various
+datasets and supervision types.",cs.CV,['cs.CV']
+A Simple Baseline for Efficient Hand Mesh Reconstruction,zhishan zhou · shihao zhou · Zhi Lv · minqiang zou · Yao Tang · Jiajun Liang,https://simplehand.github.io/,https://arxiv.org/abs/2403.01813,,2403.01813.pdf,A Simple Baseline for Efficient Hand Mesh Reconstruction,"3D hand pose estimation has found broad application in areas such as gesture
+recognition and human-machine interaction tasks. As performance improves, the
+complexity of the systems also increases, which can limit the comparative
+analysis and practical implementation of these methods. In this paper, we
+propose a simple yet effective baseline that not only surpasses
+state-of-the-art (SOTA) methods but also demonstrates computational efficiency.
+To establish this baseline, we abstract existing work into two components: a
+token generator and a mesh regressor, and then examine their core structures. A
+core structure, in this context, is one that fulfills intrinsic functions,
+brings about significant improvements, and achieves excellent performance
+without unnecessary complexities. Our proposed approach is decoupled from any
+modifications to the backbone, making it adaptable to any modern models. Our
+method outperforms existing solutions, achieving state-of-the-art (SOTA)
+results across multiple datasets. On the FreiHAND dataset, our approach
+produced a PA-MPJPE of 5.7mm and a PA-MPVPE of 6.0mm. Similarly, on the Dexycb
+dataset, we observed a PA-MPJPE of 5.5mm and a PA-MPVPE of 5.0mm. As for
+performance speed, our method reached up to 33 frames per second (fps) when
+using HRNet and up to 70 fps when employing FastViT-MA36",cs.CV,['cs.CV']
+Score-Guided Diffusion for 3D Human Recovery,Anastasis Stathopoulos · Ligong Han · Dimitris N. Metaxas,https://statho.github.io/ScoreHMR/,http://export.arxiv.org/abs/2403.09623,,2403.09623.pdf,Score-Guided Diffusion for 3D Human Recovery,"We present Score-Guided Human Mesh Recovery (ScoreHMR), an approach for
+solving inverse problems for 3D human pose and shape reconstruction. These
+inverse problems involve fitting a human body model to image observations,
+traditionally solved through optimization techniques. ScoreHMR mimics model
+fitting approaches, but alignment with the image observation is achieved
+through score guidance in the latent space of a diffusion model. The diffusion
+model is trained to capture the conditional distribution of the human model
+parameters given an input image. By guiding its denoising process with a
+task-specific score, ScoreHMR effectively solves inverse problems for various
+applications without the need for retraining the task-agnostic diffusion model.
+We evaluate our approach on three settings/applications. These are: (i)
+single-frame model fitting; (ii) reconstruction from multiple uncalibrated
+views; (iii) reconstructing humans in video sequences. ScoreHMR consistently
+outperforms all optimization baselines on popular benchmarks across all
+settings. We make our code and models available at the
+https://statho.github.io/ScoreHMR.",cs.CV,['cs.CV']
+Diversified and Personalized Multi-rater Medical Image Segmentation,Yicheng Wu · Xiangde Luo · Zhe Xu · Xiaoqing Guo · Lie Ju · Zongyuan Ge · Wenjun Liao · Jianfei Cai,https://github.com/ycwu1997/D-Persona,https://arxiv.org/abs/2403.13417,,2403.13417.pdf,Diversified and Personalized Multi-rater Medical Image Segmentation,"Annotation ambiguity due to inherent data uncertainties such as blurred
+boundaries in medical scans and different observer expertise and preferences
+has become a major obstacle for training deep-learning based medical image
+segmentation models. To address it, the common practice is to gather multiple
+annotations from different experts, leading to the setting of multi-rater
+medical image segmentation. Existing works aim to either merge different
+annotations into the ""groundtruth"" that is often unattainable in numerous
+medical contexts, or generate diverse results, or produce personalized results
+corresponding to individual expert raters. Here, we bring up a more ambitious
+goal for multi-rater medical image segmentation, i.e., obtaining both
+diversified and personalized results. Specifically, we propose a two-stage
+framework named D-Persona (first Diversification and then Personalization). In
+Stage I, we exploit multiple given annotations to train a Probabilistic U-Net
+model, with a bound-constrained loss to improve the prediction diversity. In
+this way, a common latent space is constructed in Stage I, where different
+latent codes denote diversified expert opinions. Then, in Stage II, we design
+multiple attention-based projection heads to adaptively query the corresponding
+expert prompts from the shared latent space, and then perform the personalized
+medical image segmentation. We evaluated the proposed model on our in-house
+Nasopharyngeal Carcinoma dataset and the public lung nodule dataset (i.e.,
+LIDC-IDRI). Extensive experiments demonstrated our D-Persona can provide
+diversified and personalized results at the same time, achieving new SOTA
+performance for multi-rater medical image segmentation. Our code will be
+released at https://github.com/ycwu1997/D-Persona.",cs.CV,['cs.CV']
+AnyDoor: Zero-shot Object-level Image Customization,Xi Chen · Lianghua Huang · Yu Liu · Yujun Shen · Deli Zhao · Hengshuang Zhao, ,https://arxiv.org/abs/2307.09481,,2307.09481.pdf,AnyDoor: Zero-shot Object-level Image Customization,"This work presents AnyDoor, a diffusion-based image generator with the power
+to teleport target objects to new scenes at user-specified locations in a
+harmonious way. Instead of tuning parameters for each object, our model is
+trained only once and effortlessly generalizes to diverse object-scene
+combinations at the inference stage. Such a challenging zero-shot setting
+requires an adequate characterization of a certain object. To this end, we
+complement the commonly used identity feature with detail features, which are
+carefully designed to maintain texture details yet allow versatile local
+variations (e.g., lighting, orientation, posture, etc.), supporting the object
+in favorably blending with different surroundings. We further propose to borrow
+knowledge from video datasets, where we can observe various forms (i.e., along
+the time axis) of a single object, leading to stronger model generalizability
+and robustness. Extensive experiments demonstrate the superiority of our
+approach over existing alternatives as well as its great potential in
+real-world applications, such as virtual try-on and object moving. Project page
+is https://damo-vilab.github.io/AnyDoor-Page/.",cs.CV,['cs.CV']
+Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis?,Hanxin Zhu · Tianyu He · Xin Li · Bingchen Li · Zhibo Chen, ,https://arxiv.org/abs/2403.06092,,2403.06092.pdf,Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis?,"Neural Radiance Field (NeRF) has achieved superior performance for novel view
+synthesis by modeling the scene with a Multi-Layer Perception (MLP) and a
+volume rendering procedure, however, when fewer known views are given (i.e.,
+few-shot view synthesis), the model is prone to overfit the given views. To
+handle this issue, previous efforts have been made towards leveraging learned
+priors or introducing additional regularizations. In contrast, in this paper,
+we for the first time provide an orthogonal method from the perspective of
+network structure. Given the observation that trivially reducing the number of
+model parameters alleviates the overfitting issue, but at the cost of missing
+details, we propose the multi-input MLP (mi-MLP) that incorporates the inputs
+(i.e., location and viewing direction) of the vanilla MLP into each layer to
+prevent the overfitting issue without harming detailed synthesis. To further
+reduce the artifacts, we propose to model colors and volume density separately
+and present two regularization terms. Extensive experiments on multiple
+datasets demonstrate that: 1) although the proposed mi-MLP is easy to
+implement, it is surprisingly effective as it boosts the PSNR of the baseline
+from $14.73$ to $24.23$. 2) the overall framework achieves state-of-the-art
+results on a wide range of benchmarks. We will release the code upon
+publication.",cs.CV,['cs.CV']
+PIGEON: Predicting Image Geolocations,Lukas Haas · Michal Skreta · Silas Alberti · Chelsea Finn,https://lukashaas.github.io/PIGEON-CVPR24/,,https://huggingface.co/papers/2307.05845,,,,,nan
+Nearest Is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks,Boheng Li · Yishuo Cai · Haowei Li · Feng Xue · Zhifeng Li · Yiming Li, ,https://arxiv.org/abs/2405.12725,,2405.12725.pdf,Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks,"Model quantization is widely used to compress and accelerate deep neural
+networks. However, recent studies have revealed the feasibility of weaponizing
+model quantization via implanting quantization-conditioned backdoors (QCBs).
+These special backdoors stay dormant on released full-precision models but will
+come into effect after standard quantization. Due to the peculiarity of QCBs,
+existing defenses have minor effects on reducing their threats or are even
+infeasible. In this paper, we conduct the first in-depth analysis of QCBs. We
+reveal that the activation of existing QCBs primarily stems from the nearest
+rounding operation and is closely related to the norms of neuron-wise
+truncation errors (i.e., the difference between the continuous full-precision
+weights and its quantized version). Motivated by these insights, we propose
+Error-guided Flipped Rounding with Activation Preservation (EFRAP), an
+effective and practical defense against QCBs. Specifically, EFRAP learns a
+non-nearest rounding strategy with neuron-wise error norm and layer-wise
+activation preservation guidance, flipping the rounding strategies of neurons
+crucial for backdoor effects but with minimal impact on clean accuracy.
+Extensive evaluations on benchmark datasets demonstrate that our EFRAP can
+defeat state-of-the-art QCB attacks under various settings. Code is available
+at https://github.com/AntigoneRandy/QuantBackdoor_EFRAP.",cs.CR,"['cs.CR', 'cs.CV']"
+Interactive3D: Create What You Want by Interactive 3D Generation,Shaocong Dong · Lihe Ding · Zhanpeng Huang · Zibin Wang · Tianfan Xue · Dan Xu, ,https://arxiv.org/abs/2404.16510,,2404.16510.pdf,Interactive3D: Create What You Want by Interactive 3D Generation,"3D object generation has undergone significant advancements, yielding
+high-quality results. However, fall short of achieving precise user control,
+often yielding results that do not align with user expectations, thus limiting
+their applicability. User-envisioning 3D object generation faces significant
+challenges in realizing its concepts using current generative models due to
+limited interaction capabilities. Existing methods mainly offer two approaches:
+(i) interpreting textual instructions with constrained controllability, or (ii)
+reconstructing 3D objects from 2D images. Both of them limit customization to
+the confines of the 2D reference and potentially introduce undesirable
+artifacts during the 3D lifting process, restricting the scope for direct and
+versatile 3D modifications. In this work, we introduce Interactive3D, an
+innovative framework for interactive 3D generation that grants users precise
+control over the generative process through extensive 3D interaction
+capabilities. Interactive3D is constructed in two cascading stages, utilizing
+distinct 3D representations. The first stage employs Gaussian Splatting for
+direct user interaction, allowing modifications and guidance of the generative
+direction at any intermediate step through (i) Adding and Removing components,
+(ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv)
+Semantic Editing. Subsequently, the Gaussian splats are transformed into
+InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to
+further add details and extract the geometry in the second stage. Our
+experiments demonstrate that Interactive3D markedly improves the
+controllability and quality of 3D generation. Our project webpage is available
+at \url{https://interactive-3d.github.io/}.",cs.GR,"['cs.GR', 'cs.CV']"
+Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models,Matthew Kowal · Richard P. Wildes · Kosta Derpanis,https://yorkucvil.github.io/VCC/,https://arxiv.org/abs/2404.02233,,2404.02233.pdf,Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models,"Understanding what deep network models capture in their learned
+representations is a fundamental challenge in computer vision. We present a new
+methodology to understanding such vision models, the Visual Concept Connectome
+(VCC), which discovers human interpretable concepts and their interlayer
+connections in a fully unsupervised manner. Our approach simultaneously reveals
+fine-grained concepts at a layer, connection weightings across all layers and
+is amendable to global analysis of network structure (e.g., branching pattern
+of hierarchical concept assemblies). Previous work yielded ways to extract
+interpretable concepts from single layers and examine their impact on
+classification, but did not afford multilayer concept analysis across an entire
+network architecture. Quantitative and qualitative empirical results show the
+effectiveness of VCCs in the domain of image classification. Also, we leverage
+VCCs for the application of failure mode debugging to reveal where mistakes
+arise in deep networks.",cs.CV,['cs.CV']
+GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding,Hao Li · Dingwen Zhang · Yalun Dai · Nian Liu · Lechao Cheng · Li Jingfeng · Jingdong Wang · Junwei Han, ,https://arxiv.org/abs/2311.11863,,2311.11863.pdf,GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding,"Applying NeRF to downstream perception tasks for scene understanding and
+representation is becoming increasingly popular. Most existing methods treat
+semantic prediction as an additional rendering task, \textit{i.e.}, the ""label
+rendering"" task, to build semantic NeRFs. However, by rendering
+semantic/instance labels per pixel without considering the contextual
+information of the rendered image, these methods usually suffer from unclear
+boundary segmentation and abnormal segmentation of pixels within an object. To
+solve this problem, we propose Generalized Perception NeRF (GP-NeRF), a novel
+pipeline that makes the widely used segmentation model and NeRF work compatibly
+under a unified framework, for facilitating context-aware 3D scene perception.
+To accomplish this goal, we introduce transformers to aggregate radiance as
+well as semantic embedding fields jointly for novel views and facilitate the
+joint volumetric rendering of both fields. In addition, we propose two
+self-distillation mechanisms, i.e., the Semantic Distill Loss and the
+Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality
+of the semantic field and the maintenance of geometric consistency. In
+evaluation, we conduct experimental comparisons under two perception tasks
+(\textit{i.e.} semantic and instance segmentation) using both synthetic and
+real-world datasets. Notably, our method outperforms SOTA approaches by 6.94\%,
+11.76\%, and 8.47\% on generalized semantic segmentation, finetuning semantic
+segmentation, and instance segmentation, respectively.",cs.CV,['cs.CV']
+Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception,Lei Fan · Mingfu Liang · Yunxuan Li · Gang Hua · Ying Wu, ,https://arxiv.org/abs/2311.13793,,2311.13793.pdf,Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception,"Active recognition enables robots to intelligently explore novel
+observations, thereby acquiring more information while circumventing undesired
+viewing conditions. Recent approaches favor learning policies from simulated or
+collected data, wherein appropriate actions are more frequently selected when
+the recognition is accurate. However, most recognition modules are developed
+under the closed-world assumption, which makes them ill-equipped to handle
+unexpected inputs, such as the absence of the target object in the current
+observation. To address this issue, we propose treating active recognition as a
+sequential evidence-gathering process, providing by-step uncertainty
+quantification and reliable prediction under the evidence combination theory.
+Additionally, the reward function developed in this paper effectively
+characterizes the merit of actions when operating in open-world environments.
+To evaluate the performance, we collect a dataset from an indoor simulator,
+encompassing various recognition challenges such as distance, occlusion levels,
+and visibility. Through a series of experiments on recognition and robustness
+analysis, we demonstrate the necessity of introducing uncertainties to active
+recognition and the superior performance of the proposed method.",cs.CV,"['cs.CV', 'cs.RO']"
+CURSOR: Scalable Mixed-Order Hypergraph Matching with CUR Decomposition,Qixuan Zheng · Ming Zhang · Hong Yan, ,https://arxiv.org/abs/2402.16594,,2402.16594.pdf,CURSOR: Scalable Mixed-Order Hypergraph Matching with CUR Decomposition,"To achieve greater accuracy, hypergraph matching algorithms require
+exponential increases in computational resources. Recent kd-tree-based
+approximate nearest neighbor (ANN) methods, despite the sparsity of their
+compatibility tensor, still require exhaustive calculations for large-scale
+graph matching. This work utilizes CUR tensor decomposition and introduces a
+novel cascaded second and third-order hypergraph matching framework (CURSOR)
+for efficient hypergraph matching. A CUR-based second-order graph matching
+algorithm is used to provide a rough match, and then the core of CURSOR, a
+fiber-CUR-based tensor generation method, directly calculates entries of the
+compatibility tensor by leveraging the initial second-order match result. This
+significantly decreases the time complexity and tensor density. A probability
+relaxation labeling (PRL)-based matching algorithm, especially suitable for
+sparse tensors, is developed. Experiment results on large-scale synthetic
+datasets and widely-adopted benchmark sets demonstrate the superiority of
+CURSOR over existing methods. The tensor generation method in CURSOR can be
+integrated seamlessly into existing hypergraph matching methods to improve
+their performance and lower their computational costs.",cs.CV,['cs.CV']
+Total Selfie: Generating Full-Body Selfies,Bowei Chen · Brian Curless · Ira Kemelmacher-Shlizerman · Steve Seitz, ,https://arxiv.org/abs/2308.14740,,2308.14740.pdf,Total Selfie: Generating Full-Body Selfies,"We present a method to generate full-body selfies from photographs originally
+taken at arms length. Because self-captured photos are typically taken close
+up, they have limited field of view and exaggerated perspective that distorts
+facial shapes. We instead seek to generate the photo some one else would take
+of you from a few feet away. Our approach takes as input four selfies of your
+face and body, a background image, and generates a full-body selfie in a
+desired target pose. We introduce a novel diffusion-based approach to combine
+all of this information into high-quality, well-composed photos of you with the
+desired pose and background.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Novel View Synthesis with View-Dependent Effects from a Single Image,Juan Luis Gonzalez Bello · Munchurl Kim,https://kaist-viclab.github.io/monovde-site/,https://arxiv.org/abs/2312.08071v1,,2312.08071v1.pdf,Novel View Synthesis with View-Dependent Effects from a Single Image,"In this paper, we firstly consider view-dependent effects into single
+image-based novel view synthesis (NVS) problems. For this, we propose to
+exploit the camera motion priors in NVS to model view-dependent appearance or
+effects (VDE) as the negative disparity in the scene. By recognizing
+specularities ""follow"" the camera motion, we infuse VDEs into the input images
+by aggregating input pixel colors along the negative depth region of the
+epipolar lines. Also, we propose a `relaxed volumetric rendering' approximation
+that allows computing the densities in a single pass, improving efficiency for
+NVS from single images. Our method can learn single-image NVS from image
+sequences only, which is a completely self-supervised learning method, for the
+first time requiring neither depth nor camera pose annotations. We present
+extensive experiment results and show that our proposed method can learn NVS
+with VDEs, outperforming the SOTA single-view NVS methods on the RealEstate10k
+and MannequinChallenge datasets.",cs.CV,"['cs.CV', 'eess.IV']"
+An Asymmetric Augmented Self-Supervised Learning Method for Unsupervised Fine-Grained Image Hashing,Feiran Hu · Chenlin Zhang · Jiangliang GUO · Xiu-Shen Wei · Lin Zhao · Anqi Xu · Lingyan Gao, ,,https://link.springer.com/article/10.1007/s11263-024-02009-7,,,,,nan
+TRINS: Towards Multimodal Language Models That Can Read,Ruiyi Zhang · Yanzhe Zhang · Jian Chen · Yufan Zhou · Jiuxiang Gu · Changyou Chen · Tong Sun, ,https://arxiv.org/html/2401.10005v1,,2401.10005v1.pdf,Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation,"The increasing demand for intelligent systems capable of interpreting and
+reasoning about visual content requires the development of Large Multi-Modal
+Models (LMMs) that are not only accurate but also have explicit reasoning
+capabilities. This paper presents a novel approach to imbue an LMM with the
+ability to conduct explicit reasoning based on visual content and textual
+instructions. We introduce a system that can ask a question to acquire
+necessary knowledge, thereby enhancing the robustness and explicability of the
+reasoning process. Our method comprises the development of a novel dataset
+generated by a Large Language Model (LLM), designed to promote chain-of-thought
+reasoning combined with a question-asking mechanism. We designed an LMM, which
+has high capabilities on region awareness to address the intricate requirements
+of image-text alignment. The model undergoes a three-stage training phase,
+starting with large-scale image-text alignment using a large-scale datasets,
+followed by instruction tuning, and fine-tuning with a focus on
+chain-of-thought reasoning. The results demonstrate a stride toward a more
+robust, accurate, and interpretable LMM, capable of reasoning explicitly and
+seeking information proactively when confronted with ambiguous visual input.",cs.CV,"['cs.CV', 'cs.CL']"
+DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors,Biwen Lei · Kai Yu · Mengyang Feng · Miaomiao Cui · Xuansong Xie, ,https://arxiv.org/abs/2312.16837,,2312.16837.pdf,DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors,"Text-guided domain adaptation and generation of 3D-aware portraits find many
+applications in various fields. However, due to the lack of training data and
+the challenges in handling the high variety of geometry and appearance, the
+existing methods for these tasks suffer from issues like inflexibility,
+instability, and low fidelity. In this paper, we propose a novel framework
+DiffusionGAN3D, which boosts text-guided 3D domain adaptation and generation by
+combining 3D GANs and diffusion priors. Specifically, we integrate the
+pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
+models. The former provides a strong foundation for stable and high-quality
+avatar generation from text. And the diffusion models in turn offer powerful
+priors and guide the 3D generator finetuning with informative direction to
+achieve flexible and efficient text-guided domain adaptation. To enhance the
+diversity in domain adaptation and the generation capability in text-to-avatar,
+we introduce the relative distance loss and case-specific learnable triplane
+respectively. Besides, we design a progressive texture refinement module to
+improve the texture quality for both tasks above. Extensive experiments
+demonstrate that the proposed framework achieves excellent results in both
+domain adaptation and text-to-avatar tasks, outperforming existing methods in
+terms of generation quality and efficiency. The project homepage is at
+https://younglbw.github.io/DiffusionGAN3D-homepage/.",cs.CV,['cs.CV']
+Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions,Saeed Khorram · Mingqi Jiang · Mohamad Shahbazi · Mohamad Hosein Danesh · Li Fuxin, ,https://arxiv.org/abs/2402.17065,,2402.17065.pdf,Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions,"Despite the extensive research on training generative adversarial networks
+(GANs) with limited training data, learning to generate images from long-tailed
+training distributions remains fairly unexplored. In the presence of imbalanced
+multi-class training data, GANs tend to favor classes with more samples,
+leading to the generation of low-quality and less diverse samples in tail
+classes. In this study, we aim to improve the training of class-conditional
+GANs with long-tailed data. We propose a straightforward yet effective method
+for knowledge sharing, allowing tail classes to borrow from the rich
+information from classes with more abundant training data. More concretely, we
+propose modifications to existing class-conditional GAN architectures to ensure
+that the lower-resolution layers of the generator are trained entirely
+unconditionally while reserving class-conditional generation for the
+higher-resolution layers. Experiments on several long-tail benchmarks and GAN
+architectures demonstrate a significant improvement over existing methods in
+both the diversity and fidelity of the generated images. The code is available
+at https://github.com/khorrams/utlo.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection,Trevine Oorloff · Surya Koppisetti · Nicolo Bonettini · Divyaraj Solanki · Ben Colman · Yaser Yacoob · Ali Shahriyari · Gaurav Bharaj, ,https://arxiv.org/abs/2310.03827,,2310.03827.pdf,Integrating Audio-Visual Features for Multimodal Deepfake Detection,"Deepfakes are AI-generated media in which an image or video has been
+digitally modified. The advancements made in deepfake technology have led to
+privacy and security issues. Most deepfake detection techniques rely on the
+detection of a single modality. Existing methods for audio-visual detection do
+not always surpass that of the analysis based on single modalities. Therefore,
+this paper proposes an audio-visual-based method for deepfake detection, which
+integrates fine-grained deepfake identification with binary classification. We
+categorize the samples into four types by combining labels specific to each
+single modality. This method enhances the detection under intra-domain and
+cross-domain testing.",cs.CV,['cs.CV']
+Masked AutoDecoder is Effective Multi-Task Vision Generalist,Han Qiu · Jiaxing Huang · Peng Gao · Lewei Lu · Xiaoqin Zhang · Shijian Lu, ,https://arxiv.org/abs/2403.07692,,2403.07692.pdf,Masked AutoDecoder is Effective Multi-Task Vision Generalist,"Inspired by the success of general-purpose models in NLP, recent studies
+attempt to unify different vision tasks in the same sequence format and employ
+autoregressive Transformers for sequence prediction. They apply uni-directional
+attention to capture sequential dependencies and generate task sequences
+recursively. However, such autoregressive Transformers may not fit vision tasks
+well, as vision task sequences usually lack the sequential dependencies
+typically observed in natural languages. In this work, we design Masked
+AutoDecoder~(MAD), an effective multi-task vision generalist. MAD consists of
+two core designs. First, we develop a parallel decoding framework that
+introduces bi-directional attention to capture contextual dependencies
+comprehensively and decode vision task sequences in parallel. Second, we design
+a masked sequence modeling approach that learns rich task contexts by masking
+and reconstructing task sequences. In this way, MAD handles all the tasks by a
+single network branch and a simple cross-entropy loss with minimal
+task-specific designs. Extensive experiments demonstrate the great potential of
+MAD as a new paradigm for unifying various vision tasks. MAD achieves superior
+performance and inference efficiency compared to autoregressive counterparts
+while obtaining competitive accuracy with task-specific models. Code will be
+released.",cs.CV,['cs.CV']
+HiPose: Hierarchical Binary Surface Encoding and Correspondence Pruning for RGB-D 6DoF Object Pose Estimation,Yongliang Lin · Yongzhi Su · Praveen Nathan · Sandeep Inuganti · Yan Di · Martin Sundermeyer · Fabian Manhardt · Didier Stricker · Jason Rambach · Yu Zhang, ,https://arxiv.org/abs/2311.12588,,2311.12588.pdf,HiPose: Hierarchical Binary Surface Encoding and Correspondence Pruning for RGB-D 6DoF Object Pose Estimation,"In this work, we present a novel dense-correspondence method for 6DoF object
+pose estimation from a single RGB-D image. While many existing data-driven
+methods achieve impressive performance, they tend to be time-consuming due to
+their reliance on rendering-based refinement approaches. To circumvent this
+limitation, we present HiPose, which establishes 3D-3D correspondences in a
+coarse-to-fine manner with a hierarchical binary surface encoding. Unlike
+previous dense-correspondence methods, we estimate the correspondence surface
+by employing point-to-surface matching and iteratively constricting the surface
+until it becomes a correspondence point while gradually removing outliers.
+Extensive experiments on public benchmarks LM-O, YCB-V, and T-Less demonstrate
+that our method surpasses all refinement-free methods and is even on par with
+expensive refinement-based approaches. Crucially, our approach is
+computationally efficient and enables real-time critical applications with high
+accuracy requirements.",cs.CV,['cs.CV']
+Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D,Karran Pandey · Paul Guerrero · Matheus Gadelha · Yannick Hold-Geoffroy · Karan Singh · Niloy J. Mitra,https://diffusionhandles.github.io/,https://arxiv.org/abs/2312.02190,,2312.02190.pdf,Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D,"Diffusion Handles is a novel approach to enabling 3D object edits on
+diffusion images. We accomplish these edits using existing pre-trained
+diffusion models, and 2D image depth estimation, without any fine-tuning or 3D
+object retrieval. The edited results remain plausible, photo-real, and preserve
+object identity. Diffusion Handles address a critically missing facet of
+generative image based creative design, and significantly advance the
+state-of-the-art in generative image editing. Our key insight is to lift
+diffusion activations for an object to 3D using a proxy depth, 3D-transform the
+depth and associated activations, and project them back to image space. The
+diffusion process applied to the manipulated activations with identity control,
+produces plausible edited images showing complex 3D occlusion and lighting
+effects. We evaluate Diffusion Handles: quantitatively, on a large synthetic
+data benchmark; and qualitatively by a user study, showing our output to be
+more plausible, and better than prior art at both, 3D editing and identity
+control. Project Webpage: https://diffusionhandles.github.io/",cs.CV,"['cs.CV', 'cs.GR']"
+CAM Back Again: Large Kernel CNNs from a Weakly Supervised Object Localization Perspective,Shunsuke Yasuki · Masato Taki, ,,https://github.com/snskysk/CAM-Back-Again,,,,,nan
+A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models,Julio Silva-Rodríguez · Sina Hajimiri · Ismail Ben Ayed · Jose Dolz,https://jusiro.github.io/projects/clap,https://arxiv.org/abs/2312.12730,,2312.12730.pdf,A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models,"Efficient transfer learning (ETL) is receiving increasing attention to adapt
+large pre-trained language-vision models on downstream tasks with a few labeled
+samples. While significant progress has been made, we reveal that
+state-of-the-art ETL approaches exhibit strong performance only in
+narrowly-defined experimental setups, and with a careful adjustment of
+hyperparameters based on a large corpus of labeled samples. In particular, we
+make two interesting, and surprising empirical observations. First, to
+outperform a simple Linear Probing baseline, these methods require to optimize
+their hyper-parameters on each target task. And second, they typically
+underperform -- sometimes dramatically -- standard zero-shot predictions in the
+presence of distributional drifts. Motivated by the unrealistic assumptions
+made in the existing literature, i.e., access to a large validation set and
+case-specific grid-search for optimal hyperparameters, we propose a novel
+approach that meets the requirements of real-world scenarios. More concretely,
+we introduce a CLass-Adaptive linear Probe (CLAP) objective, whose balancing
+term is optimized via an adaptation of the general Augmented Lagrangian method
+tailored to this context. We comprehensively evaluate CLAP on a broad span of
+datasets and scenarios, demonstrating that it consistently outperforms SoTA
+approaches, while yet being a much more efficient alternative.",cs.CV,['cs.CV']
+DiaLoc: An Iterative Approach to Embodied Dialog Localization,Chao Zhang · Mohan Li · Ignas Budvytis · Stephan Liwicki, ,https://arxiv.org/abs/2403.06846,,2403.06846.pdf,DiaLoc: An Iterative Approach to Embodied Dialog Localization,"Multimodal learning has advanced the performance for many vision-language
+tasks. However, most existing works in embodied dialog research focus on
+navigation and leave the localization task understudied. The few existing
+dialog-based localization approaches assume the availability of entire dialog
+prior to localizaiton, which is impractical for deployed dialog-based
+localization. In this paper, we propose DiaLoc, a new dialog-based localization
+framework which aligns with a real human operator behavior. Specifically, we
+produce an iterative refinement of location predictions which can visualize
+current pose believes after each dialog turn. DiaLoc effectively utilizes the
+multimodal data for multi-shot localization, where a fusion encoder fuses
+vision and dialog information iteratively. We achieve state-of-the-art results
+on embodied dialog-based localization task, in single-shot (+7.08% in
+Acc5@valUnseen) and multi-shot settings (+10.85% in Acc5@valUnseen). DiaLoc
+narrows the gap between simulation and real-world applications, opening doors
+for future research on collaborative localization and navigation.",cs.CV,['cs.CV']
+De-Diffusion Makes Text a Strong Cross-Modal Interface,Chen Wei · Chenxi Liu · Siyuan Qiao · Zhishuai Zhang · Alan L. Yuille · Jiahui Yu, ,https://arxiv.org/abs/2311.00618,,2311.00618.pdf,De-Diffusion Makes Text a Strong Cross-Modal Interface,"We demonstrate text as a strong cross-modal interface. Rather than relying on
+deep embeddings to connect image and language as the interface representation,
+our approach represents an image as text, from which we enjoy the
+interpretability and flexibility inherent to natural language. We employ an
+autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
+The encoder is trained to transform an input image into text, which is then fed
+into the fixed text-to-image diffusion decoder to reconstruct the original
+input -- a process we term De-Diffusion. Experiments validate both the
+precision and comprehensiveness of De-Diffusion text representing images, such
+that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
+for diverse multi-modal tasks. For example, a single De-Diffusion model can
+generalize to provide transferable prompts for different text-to-image tools,
+and also achieves a new state of the art on open-ended vision-language tasks by
+simply prompting large language models with few-shot examples.",cs.CV,['cs.CV']
+MMM: Generative Masked Motion Model,Ekkasit Pinyoanuntapong · Pu Wang · Minwoo Lee · Chen Chen, ,https://arxiv.org/abs/2312.03596,,2312.03596.pdf,MMM: Generative Masked Motion Model,"Recent advances in text-to-motion generation using diffusion and
+autoregressive models have shown promising results. However, these models often
+suffer from a trade-off between real-time performance, high fidelity, and
+motion editability. To address this gap, we introduce MMM, a novel yet simple
+motion generation paradigm based on Masked Motion Model. MMM consists of two
+key components: (1) a motion tokenizer that transforms 3D human motion into a
+sequence of discrete tokens in latent space, and (2) a conditional masked
+motion transformer that learns to predict randomly masked motion tokens,
+conditioned on the pre-computed text tokens. By attending to motion and text
+tokens in all directions, MMM explicitly captures inherent dependency among
+motion tokens and semantic mapping between motion and text tokens. During
+inference, this allows parallel and iterative decoding of multiple motion
+tokens that are highly consistent with fine-grained text descriptions,
+therefore simultaneously achieving high-fidelity and high-speed motion
+generation. In addition, MMM has innate motion editability. By simply placing
+mask tokens in the place that needs editing, MMM automatically fills the gaps
+while guaranteeing smooth transitions between editing and non-editing parts.
+Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that MMM
+surpasses current leading methods in generating high-quality motion (evidenced
+by superior FID scores of 0.08 and 0.429), while offering advanced editing
+features such as body-part modification, motion in-betweening, and the
+synthesis of long motion sequences. In addition, MMM is two orders of magnitude
+faster on a single mid-range GPU than editable motion diffusion models. Our
+project page is available at \url{https://exitudio.github.io/MMM-page}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+RMem: Restricted Memory Banks Improve Video Object Segmentation,Junbao Zhou · Ziqi Pang · Yu-Xiong Wang, ,https://arxiv.org/abs/2403.11529,,2403.11529.pdf,Video Object Segmentation with Dynamic Query Modulation,"Storing intermediate frame segmentations as memory for long-range context
+modeling, spatial-temporal memory-based methods have recently showcased
+impressive results in semi-supervised video object segmentation (SVOS).
+However, these methods face two key limitations: 1) relying on non-local
+pixel-level matching to read memory, resulting in noisy retrieved features for
+segmentation; 2) segmenting each object independently without interaction.
+These shortcomings make the memory-based methods struggle in similar object and
+multi-object segmentation. To address these issues, we propose a query
+modulation method, termed QMVOS. This method summarizes object features into
+dynamic queries and then treats them as dynamic filters for mask prediction,
+thereby providing high-level descriptions and object-level perception for the
+model. Efficient and effective multi-object interactions are realized through
+inter-query attention. Extensive experiments demonstrate that our method can
+bring significant improvements to the memory-based SVOS method and achieve
+competitive performance on standard SVOS benchmarks. The code is available at
+https://github.com/zht8506/QMVOS.",cs.CV,['cs.CV']
+Neural Implicit Morphing of Face Images,Guilherme Schardong · Tiago Novello · Hallison Paz · Iurii Medvedev · Vinícius Silva · Luiz Velho · Nuno Gonçalves,https://schardong.github.io/ifmorph/,https://arxiv.org/abs/2308.13888,,2308.13888.pdf,Neural Implicit Morphing of Face Images,"Face morphing is a problem in computer graphics with numerous artistic and
+forensic applications. It is challenging due to variations in pose, lighting,
+gender, and ethnicity. This task consists of a warping for feature alignment
+and a blending for a seamless transition between the warped images. We propose
+to leverage coord-based neural networks to represent such warpings and
+blendings of face images. During training, we exploit the smoothness and
+flexibility of such networks by combining energy functionals employed in
+classical approaches without discretizations. Additionally, our method is
+time-dependent, allowing a continuous warping/blending of the images. During
+morphing inference, we need both direct and inverse transformations of the
+time-dependent warping. The first (second) is responsible for warping the
+target (source) image into the source (target) image. Our neural warping stores
+those maps in a single network dismissing the need for inverting them. The
+results of our experiments indicate that our method is competitive with both
+classical and generative models under the lens of image quality and
+face-morphing detectors. Aesthetically, the resulting images present a seamless
+blending of diverse faces not yet usual in the literature.",cs.CV,"['cs.CV', 'cs.LG', 'I.4.8; I.4.10']"
+"Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action",Jiasen Lu · Christopher Clark · Sangho Lee · Zichen Zhang · Savya Khosla · Ryan Marten · Derek Hoiem · Aniruddha Kembhavi,https://unified-io-2.allenai.org/,https://arxiv.org/abs/2312.17172,,2312.17172.pdf,"Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action","We present Unified-IO 2, the first autoregressive multimodal model that is
+capable of understanding and generating image, text, audio, and action. To
+unify different modalities, we tokenize inputs and outputs -- images, text,
+audio, action, bounding boxes, etc., into a shared semantic space and then
+process them with a single encoder-decoder transformer model. Since training
+with such diverse modalities is challenging, we propose various architectural
+improvements to stabilize model training. We train our model from scratch on a
+large multimodal pre-training corpus from diverse sources with a multimodal
+mixture of denoisers objective. To learn an expansive set of skills, such as
+following multimodal instructions, we construct and finetune on an ensemble of
+120 datasets with prompts and augmentations. With a single unified model,
+Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
+strong results in more than 35 benchmarks, including image generation and
+understanding, natural language understanding, video and audio understanding,
+and robotic manipulation. We release all our models to the research community.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+SUGAR: Pre-training 3D Visual Representation for Robotics,Shizhe Chen · Ricardo Garcia Pinel · Ivan Laptev · Cordelia Schmid, ,https://arxiv.org/abs/2404.01491,,2404.01491.pdf,SUGAR: Pre-training 3D Visual Representations for Robotics,"Learning generalizable visual representations from Internet data has yielded
+promising results for robotics. Yet, prevailing approaches focus on
+pre-training 2D representations, being sub-optimal to deal with occlusions and
+accurately localize objects in complex 3D scenes. Meanwhile, 3D representation
+learning has been limited to single-object understanding. To address these
+limitations, we introduce a novel 3D pre-training framework for robotics named
+SUGAR that captures semantic, geometric and affordance properties of objects
+through 3D point clouds. We underscore the importance of cluttered scenes in 3D
+representation learning, and automatically construct a multi-object dataset
+benefiting from cost-free supervision in simulation. SUGAR employs a versatile
+transformer-based model to jointly address five pre-training tasks, namely
+cross-modal knowledge distillation for semantic learning, masked point modeling
+to understand geometry structures, grasping pose synthesis for object
+affordance, 3D instance segmentation and referring expression grounding to
+analyze cluttered scenes. We evaluate our learned representation on three
+robotic-related tasks, namely, zero-shot 3D object recognition, referring
+expression grounding, and language-driven robotic manipulation. Experimental
+results show that SUGAR's 3D representation outperforms state-of-the-art 2D and
+3D representations.",cs.CV,['cs.CV']
+GenN2N: Generative NeRF2NeRF Translation,Xiangyue Liu · Han Xue · Kunming Luo · Ping Tan · Li Yi, ,https://arxiv.org/abs/2404.02788,,2404.02788.pdf,GenN2N: Generative NeRF2NeRF Translation,"We present GenN2N, a unified NeRF-to-NeRF translation framework for various
+NeRF translation tasks such as text-driven NeRF editing, colorization,
+super-resolution, inpainting, etc. Unlike previous methods designed for
+individual translation tasks with task-specific schemes, GenN2N achieves all
+these NeRF editing tasks by employing a plug-and-play image-to-image translator
+to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF
+space. Since the 3D consistency of 2D edits may not be assured, we propose to
+model the distribution of the underlying 3D edits through a generative model
+that can cover all possible edited NeRFs. To model the distribution of 3D
+edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes
+images while decoding NeRFs. The latent space is trained to align with a
+Gaussian distribution and the NeRFs are supervised through an adversarial loss
+on its renderings. To ensure the latent code does not depend on 2D viewpoints
+but truly reflects the 3D edits, we also regularize the latent code through a
+contrastive learning scheme. Extensive experiments on various editing tasks
+show GenN2N, as a universal framework, performs as well or better than
+task-specific specialists while possessing flexible generative power. More
+results on our project page: https://xiangyueliu.github.io/GenN2N/",cs.CV,['cs.CV']
+UniHuman: A Unified Model For Editing Human Images in the Wild,Nannan Li · Qing Liu · Krishna Kumar Singh · Yilin Wang · Jianming Zhang · Bryan A. Plummer · Zhe Lin, ,https://arxiv.org/abs/2312.14985,,2312.14985.pdf,UniHuman: A Unified Model for Editing Human Images in the Wild,"Human image editing includes tasks like changing a person's pose, their
+clothing, or editing the image according to a text prompt. However, prior work
+often tackles these tasks separately, overlooking the benefit of mutual
+reinforcement from learning them jointly. In this paper, we propose UniHuman, a
+unified model that addresses multiple facets of human image editing in
+real-world settings. To enhance the model's generation quality and
+generalization capacity, we leverage guidance from human visual encoders and
+introduce a lightweight pose-warping module that can exploit different pose
+representations, accommodating unseen textures and patterns. Furthermore, to
+bridge the disparity between existing human editing benchmarks with real-world
+data, we curated 400K high-quality human image-text pairs for training and
+collected 2K human images for out-of-domain testing, both encompassing diverse
+clothing styles, backgrounds, and age groups. Experiments on both in-domain and
+out-of-domain test sets demonstrate that UniHuman outperforms task-specific
+models by a significant margin. In user studies, UniHuman is preferred by the
+users in an average of 77% of cases. Our project is available at
+https://github.com/NannanLi999/UniHuman.",cs.CV,['cs.CV']
+Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,Tsai-Shien Chen · Aliaksandr Siarohin · Willi Menapace · Ekaterina Deyneka · Hsiang-wei Chao · Byung Jeon · Yuwei Fang · Hsin-Ying Lee · Jian Ren · Ming-Hsuan Yang · Sergey Tulyakov,https://snap-research.github.io/Panda-70M/,https://arxiv.org/abs/2402.19479,,2402.19479.pdf,Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,"The quality of the data and annotation upper-bounds the quality of a
+downstream model. While there exist large text corpora and image-text pairs,
+high-quality video-text data is much harder to collect. First of all, manual
+labeling is more time-consuming, as it requires an annotator to watch an entire
+video. Second, videos have a temporal dimension, consisting of several scenes
+stacked together, and showing multiple actions. Accordingly, to establish a
+video dataset with high-quality captions, we propose an automatic approach
+leveraging multimodal inputs, such as textual video description, subtitles, and
+individual video frames. Specifically, we curate 3.8M high-resolution videos
+from the publicly available HD-VILA-100M dataset. We then split them into
+semantically consistent video clips, and apply multiple cross-modality teacher
+models to obtain captions for each video. Next, we finetune a retrieval model
+on a small subset where the best caption of each video is manually selected and
+then employ the model in the whole dataset to select the best caption as the
+annotation. In this way, we get 70M videos paired with high-quality text
+captions. We dub the dataset as Panda-70M. We show the value of the proposed
+dataset on three downstream tasks: video captioning, video and text retrieval,
+and text-driven video generation. The models trained on the proposed data score
+substantially better on the majority of metrics across all the tasks.",cs.CV,['cs.CV']
+Personalized Residuals for Concept-Driven Text-to-Image Generation,Cusuh Ham · Matthew Fisher · James Hays · Nicholas Kolkin · Yuchen Liu · Richard Zhang · Tobias Hinz, ,https://arxiv.org/abs/2405.12978,,2405.12978.pdf,Personalized Residuals for Concept-Driven Text-to-Image Generation,"We present personalized residuals and localized attention-guided sampling for
+efficient concept-driven generation using text-to-image diffusion models. Our
+method first represents concepts by freezing the weights of a pretrained
+text-conditioned diffusion model and learning low-rank residuals for a small
+subset of the model's layers. The residual-based approach then directly enables
+application of our proposed sampling technique, which applies the learned
+residuals only in areas where the concept is localized via cross-attention and
+applies the original diffusion weights in all other regions. Localized sampling
+therefore combines the learned identity of the concept with the existing
+generative prior of the underlying diffusion model. We show that personalized
+residuals effectively capture the identity of a concept in ~3 minutes on a
+single GPU without the use of regularization images and with fewer parameters
+than previous models, and localized sampling allows using the original model as
+strong prior for large parts of the image.",cs.CV,['cs.CV']
+SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection,Mingxuan Liu · Tyler Hayes · Elisa Ricci · Gabriela Csurka · Riccardo Volpi,https://github.com/naver/shine,https://arxiv.org/abs/2405.10053,,2405.10053.pdf,SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection,"Open-vocabulary object detection (OvOD) has transformed detection into a
+language-guided task, empowering users to freely define their class
+vocabularies of interest during inference. However, our initial investigation
+indicates that existing OvOD detectors exhibit significant variability when
+dealing with vocabularies across various semantic granularities, posing a
+concern for real-world deployment. To this end, we introduce Semantic Hierarchy
+Nexus (SHiNe), a novel classifier that uses semantic knowledge from class
+hierarchies. It runs offline in three steps: i) it retrieves relevant
+super-/sub-categories from a hierarchy for each target class; ii) it integrates
+these categories into hierarchy-aware sentences; iii) it fuses these sentence
+embeddings to generate the nexus classifier vector. Our evaluation on various
+detection benchmarks demonstrates that SHiNe enhances robustness across diverse
+vocabulary granularities, achieving up to +31.9% mAP50 with ground truth
+hierarchies, while retaining improvements using hierarchies generated by large
+language models. Moreover, when applied to open-vocabulary classification on
+ImageNet-1k, SHiNe improves the CLIP zero-shot baseline by +2.8% accuracy.
+SHiNe is training-free and can be seamlessly integrated with any off-the-shelf
+OvOD detector, without incurring additional computational overhead during
+inference. The code is open source.",cs.CV,['cs.CV']
+Diff-BGM: A Diffusion Model for Video Background Music Generation,Sizhe Li · Yiming Qin · Minghang Zheng · Xin Jin · Yang Liu, ,http://export.arxiv.org/abs/2405.11913,,2405.11913.pdf,Diff-BGM: A Diffusion Model for Video Background Music Generation,"When editing a video, a piece of attractive background music is
+indispensable. However, video background music generation tasks face several
+challenges, for example, the lack of suitable training datasets, and the
+difficulties in flexibly controlling the music generation process and
+sequentially aligning the video and music. In this work, we first propose a
+high-quality music-video dataset BGM909 with detailed annotation and shot
+detection to provide multi-modal information about the video and music. We then
+present evaluation metrics to assess music quality, including music diversity
+and alignment between music and video with retrieval precision metrics.
+Finally, we propose the Diff-BGM framework to automatically generate the
+background music for a given video, which uses different signals to control
+different aspects of the music during the generation process, i.e., uses
+dynamic video features to control music rhythm and semantic features to control
+the melody and atmosphere. We propose to align the video and music sequentially
+by introducing a segment-aware cross-attention layer. Experiments verify the
+effectiveness of our proposed method. The code and models are available at
+https://github.com/sizhelee/Diff-BGM.",cs.CV,['cs.CV']
+Efficient Detection of Long Consistent Cycles and its Application to Distributed Synchronization,Shaohan Li · Yunpeng Shi · Gilad Lerman, ,,https://www.semanticscholar.org/paper/Fully-distributed-synchronization-on-directed-via-Xia-Li/23d2c7b0150d90992f60c1d8a94d263beacb2bb0,,,,,nan
+Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection,Suyeon Kim · Dongha Lee · SeongKu Kang · Sukang Chae · Sanghwan Jang · Hwanjo Yu, ,https://arxiv.org/abs/2405.19902,,2405.19902.pdf,Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection,"Label noise, commonly found in real-world datasets, has a detrimental impact
+on a model's generalization. To effectively detect incorrectly labeled
+instances, previous works have mostly relied on distinguishable training
+signals, such as training loss, as indicators to differentiate between clean
+and noisy labels. However, they have limitations in that the training signals
+incompletely reveal the model's behavior and are not effectively generalized to
+various noise types, resulting in limited detection accuracy. In this paper, we
+propose DynaCor framework that distinguishes incorrectly labeled instances from
+correctly labeled ones based on the dynamics of the training signals. To cope
+with the absence of supervision for clean and noisy labels, DynaCor first
+introduces a label corruption strategy that augments the original dataset with
+intentionally corrupted labels, enabling indirect simulation of the model's
+behavior on noisy labels. Then, DynaCor learns to identify clean and noisy
+instances by inducing two clearly distinguishable clusters from the latent
+representations of training dynamics. Our comprehensive experiments show that
+DynaCor outperforms the state-of-the-art competitors and shows strong
+robustness to various noise types and noise rates.",cs.LG,"['cs.LG', 'stat.ML']"
+Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation,Qi Yang · Xing Nie · Tong Li · Gaopengfei · Ying Guo · Cheng Zhen · Pengfei Yan · Shiming Xiang, ,https://arxiv.org/abs/2312.06462,,2312.06462.pdf,Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation,"Recently, an audio-visual segmentation (AVS) task has been introduced, aiming
+to group pixels with sounding objects within a given video. This task
+necessitates a first-ever audio-driven pixel-level understanding of the scene,
+posing significant challenges. In this paper, we propose an innovative
+audio-visual transformer framework, termed COMBO, an acronym for COoperation of
+Multi-order Bilateral relatiOns. For the first time, our framework explores
+three types of bilateral entanglements within AVS: pixel entanglement, modality
+entanglement, and temporal entanglement. Regarding pixel entanglement, we
+employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate
+more precise visual features from the foundational model. For modality
+entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to
+align corresponding visual and auditory signals bi-directionally. As for
+temporal entanglement, we introduce an innovative adaptive inter-frame
+consistency loss according to the inherent rules of temporal. Comprehensive
+experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou
+on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that
+COMBO surpasses previous state-of-the-art methods. Code and more results will
+be publicly available at https://yannqi.github.io/AVS-COMBO/.",cs.CV,"['cs.CV', 'cs.AI', 'cs.SD', 'eess.AS']"
+EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars,Nikita Drobyshev · Antoni Bigata Casademunt · Konstantinos Vougioukas · Zoe Landgraf · Stavros Petridis · Maja Pantic, ,https://arxiv.org/abs/2404.19110,,2404.19110.pdf,EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars,"Head avatars animated by visual signals have gained popularity, particularly
+in cross-driving synthesis where the driver differs from the animated
+character, a challenging but highly practical approach. The recently presented
+MegaPortraits model has demonstrated state-of-the-art results in this domain.
+We conduct a deep examination and evaluation of this model, with a particular
+focus on its latent space for facial expression descriptors, and uncover
+several limitations with its ability to express intense face motions. To
+address these limitations, we propose substantial changes in both training
+pipeline and model architecture, to introduce our EMOPortraits model, where we:
+  Enhance the model's capability to faithfully support intense, asymmetric face
+expressions, setting a new state-of-the-art result in the emotion transfer
+task, surpassing previous methods in both metrics and quality.
+  Incorporate speech-driven mode to our model, achieving top-tier performance
+in audio-driven facial animation, making it possible to drive source identity
+through diverse modalities, including visual signal, audio, or a blend of both.
+  We propose a novel multi-view video dataset featuring a wide range of intense
+and asymmetric facial expressions, filling the gap with absence of such data in
+existing datasets.",cs.CV,['cs.CV']
+NC-TTT: A Noise Constrastive Approach for Test-Time Training,David OSOWIECHI · Gustavo Vargas Hakim · Mehrdad Noori · Milad Cheraghalikhani · Ali Bahri · Moslem Yazdanpanah · Ismail Ben Ayed · Christian Desrosiers, ,https://arxiv.org/abs/2404.08392,,2404.08392.pdf,NC-TTT: A Noise Contrastive Approach for Test-Time Training,"Despite their exceptional performance in vision tasks, deep learning models
+often struggle when faced with domain shifts during testing. Test-Time Training
+(TTT) methods have recently gained popularity by their ability to enhance the
+robustness of models through the addition of an auxiliary objective that is
+jointly optimized with the main task. Being strictly unsupervised, this
+auxiliary objective is used at test time to adapt the model without any access
+to labels. In this work, we propose Noise-Contrastive Test-Time Training
+(NC-TTT), a novel unsupervised TTT technique based on the discrimination of
+noisy feature maps. By learning to classify noisy views of projected feature
+maps, and then adapting the model accordingly on new domains, classification
+performance can be recovered by an important margin. Experiments on several
+popular test-time adaptation baselines demonstrate the advantages of our method
+compared to recent approaches for this task. The code can be found
+at:https://github.com/GustavoVargasHakim/NCTTT.git",cs.CV,"['cs.CV', 'cs.LG']"
+Forecasting of 3D Whole-body Human Poses with Grasping Objects,yan haitao · Qiongjie Cui · Jiexin Xie · Shijie Guo, ,https://arxiv.org/abs/2312.11972,,2312.11972.pdf,Expressive Forecasting of 3D Whole-body Human Motions,"Human motion forecasting, with the goal of estimating future human behavior
+over a period of time, is a fundamental task in many real-world applications.
+However, existing works typically concentrate on predicting the major joints of
+the human body without considering the delicate movements of the human hands.
+In practical applications, hand gesture plays an important role in human
+communication with the real world, and expresses the primary intention of human
+beings. In this work, we are the first to formulate a whole-body human pose
+forecasting task, which jointly predicts the future body and hand activities.
+Correspondingly, we propose a novel Encoding-Alignment-Interaction (EAI)
+framework that aims to predict both coarse (body joints) and fine-grained
+(gestures) activities collaboratively, enabling expressive and
+cross-facilitated forecasting of 3D whole-body human motions. Specifically, our
+model involves two key constituents: cross-context alignment (XCA) and
+cross-context interaction (XCI). Considering the heterogeneous information
+within the whole-body, XCA aims to align the latent features of various human
+components, while XCI focuses on effectively capturing the context interaction
+among the human components. We conduct extensive experiments on a
+newly-introduced large-scale benchmark and achieve state-of-the-art
+performance. The code is public for research purposes at
+https://github.com/Dingpx/EAI.",cs.CV,['cs.CV']
+QN-Mixer: A Quasi-Newton MLP-Mixer Model for Sparse-View CT Reconstruction,Ishak Ayad · Nicolas Larue · Mai K. Nguyen, ,https://arxiv.org/abs/2402.17951,,2402.17951.pdf,QN-Mixer: A Quasi-Newton MLP-Mixer Model for Sparse-View CT Reconstruction,"Inverse problems span across diverse fields. In medical contexts, computed
+tomography (CT) plays a crucial role in reconstructing a patient's internal
+structure, presenting challenges due to artifacts caused by inherently
+ill-posed inverse problems. Previous research advanced image quality via
+post-processing and deep unrolling algorithms but faces challenges, such as
+extended convergence times with ultra-sparse data. Despite enhancements,
+resulting images often show significant artifacts, limiting their effectiveness
+for real-world diagnostic applications. We aim to explore deep second-order
+unrolling algorithms for solving imaging inverse problems, emphasizing their
+faster convergence and lower time complexity compared to common first-order
+methods like gradient descent. In this paper, we introduce QN-Mixer, an
+algorithm based on the quasi-Newton approach. We use learned parameters through
+the BFGS algorithm and introduce Incept-Mixer, an efficient neural architecture
+that serves as a non-local regularization term, capturing long-range
+dependencies within images. To address the computational demands typically
+associated with quasi-Newton algorithms that require full Hessian matrix
+computations, we present a memory-efficient alternative. Our approach
+intelligently downsamples gradient information, significantly reducing
+computational requirements while maintaining performance. The approach is
+validated through experiments on the sparse-view CT problem, involving various
+datasets and scanning protocols, and is compared with post-processing and deep
+unrolling state-of-the-art approaches. Our method outperforms existing
+approaches and achieves state-of-the-art performance in terms of SSIM and PSNR,
+all while reducing the number of unrolling iterations required.",eess.IV,"['eess.IV', 'cs.CV']"
+Class Tokens Infusion for Weakly Supervised Semantic Segmentation,Sung-Hoon Yoon · Hoyong Kwon · Hyeonseong Kim · Kuk-Jin Yoon, ,http://export.arxiv.org/abs/2308.03005,,2308.03005.pdf,MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation,"This paper proposes a novel transformer-based framework that aims to enhance
+weakly supervised semantic segmentation (WSSS) by generating accurate
+class-specific object localization maps as pseudo labels. Building upon the
+observation that the attended regions of the one-class token in the standard
+vision transformer can contribute to a class-agnostic localization map, we
+explore the potential of the transformer model to capture class-specific
+attention for class-discriminative object localization by learning multiple
+class tokens. We introduce a Multi-Class Token transformer, which incorporates
+multiple class tokens to enable class-aware interactions with the patch tokens.
+To achieve this, we devise a class-aware training strategy that establishes a
+one-to-one correspondence between the output class tokens and the ground-truth
+class labels. Moreover, a Contrastive-Class-Token (CCT) module is proposed to
+enhance the learning of discriminative class tokens, enabling the model to
+better capture the unique characteristics and properties of each class. As a
+result, class-discriminative object localization maps can be effectively
+generated by leveraging the class-to-patch attentions associated with different
+class tokens. To further refine these localization maps, we propose the
+utilization of patch-level pairwise affinity derived from the patch-to-patch
+transformer attention. Furthermore, the proposed framework seamlessly
+complements the Class Activation Mapping (CAM) method, resulting in
+significantly improved WSSS performance on the PASCAL VOC 2012 and MS COCO 2014
+datasets. These results underline the importance of the class token for WSSS.",cs.CV,['cs.CV']
+Dual-consistency Model Inversion for Non-exemplar Class Incremental Learning,Zihuan Qiu · Yi Xu · Fanman Meng · Hongliang Li · Linfeng Xu · Qingbo Wu, ,https://ar5iv.labs.arxiv.org/html/2303.10891,,2303.10891.pdf,Non-Exemplar Online Class-incremental Continual Learning via Dual-prototype Self-augment and Refinement,"This paper investigates a new, practical, but challenging problem named
+Non-exemplar Online Class-incremental continual Learning (NO-CL), which aims to
+preserve the discernibility of base classes without buffering data examples and
+efficiently learn novel classes continuously in a single-pass (i.e., online)
+data stream. The challenges of this task are mainly two-fold: (1) Both base and
+novel classes suffer from severe catastrophic forgetting as no previous samples
+are available for replay. (2) As the online data can only be observed once,
+there is no way to fully re-train the whole model, e.g., re-calibrate the
+decision boundaries via prototype alignment or feature distillation. In this
+paper, we propose a novel Dual-prototype Self-augment and Refinement method
+(DSR) for NO-CL problem, which consists of two strategies: 1) Dual class
+prototypes: vanilla and high-dimensional prototypes are exploited to utilize
+the pre-trained information and obtain robust quasi-orthogonal representations
+rather than example buffers for both privacy preservation and memory reduction.
+2) Self-augment and refinement: Instead of updating the whole network, we
+optimize high-dimensional prototypes alternatively with the extra projection
+module based on self-augment vanilla prototypes, through a bi-level
+optimization problem. Extensive experiments demonstrate the effectiveness and
+superiority of the proposed DSR in NO-CL.",cs.CV,['cs.CV']
+MoDE: CLIP Data Experts via Clustering,Jiawei Ma · Po-Yao Huang · Saining Xie · Shang-Wen Li · Luke Zettlemoyer · Shih-Fu Chang · Wen-tau Yih · Hu Xu,https://github.com/facebookresearch/MetaCLIP/tree/main/mode,https://arxiv.org/abs/2404.16030,,2404.16030.pdf,MoDE: CLIP Data Experts via Clustering,"The success of contrastive language-image pretraining (CLIP) relies on the
+supervision from the pairing between images and captions, which tends to be
+noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
+a system of CLIP data experts via clustering. Each data expert is trained on
+one data cluster, being less sensitive to false negative noises in other
+clusters. At inference time, we ensemble their outputs by applying weights
+determined through the correlation between task metadata and cluster
+conditions. To estimate the correlation precisely, the samples in one cluster
+should be semantically similar, but the number of data experts should still be
+reasonable for training and inference. As such, we consider the ontology in
+human language and propose to use fine-grained cluster centers to represent
+each data expert at a coarse-grained level. Experimental studies show that four
+CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
+OpenCLIP on zero-shot image classification but with less ($<$35\%) training
+cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
+include new data experts. The code is available at
+https://github.com/facebookresearch/MetaCLIP/tree/main/mode.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+FSC: Few-point Shape Completion,Xianzu Wu · Xianfeng Wu · Tianyu Luan · Yajing Bai · Zhongyuan Lai · Junsong Yuan, ,https://arxiv.org/abs/2403.07359v4,,2403.07359v4.pdf,FSC: Few-point Shape Completion,"While previous studies have demonstrated successful 3D object shape
+completion with a sufficient number of points, they often fail in scenarios
+when a few points, e.g. tens of points, are observed. Surprisingly, via entropy
+analysis, we find that even a few points, e.g. 64 points, could retain
+substantial information to help recover the 3D shape of the object. To address
+the challenge of shape completion with very sparse point clouds, we then
+propose Few-point Shape Completion (FSC) model, which contains a novel
+dual-branch feature extractor for handling extremely sparse inputs, coupled
+with an extensive branch for maximal point utilization with a saliency branch
+for dynamic importance assignment. This model is further bolstered by a
+two-stage revision network that refines both the extracted features and the
+decoder output, enhancing the detail and authenticity of the completed point
+cloud. Our experiments demonstrate the feasibility of recovering 3D shapes from
+a few points. The proposed Few-point Shape Completion (FSC) model outperforms
+previous methods on both few-point inputs and many-point inputs, and shows good
+generalizability to different object categories.",cs.CV,['cs.CV']
+Equivariant Multi-Modality Image Fusion,Zixiang Zhao · Haowen Bai · Jiangshe Zhang · Yulun Zhang · Kai Zhang · Shuang Xu · Dongdong Chen · Radu Timofte · Luc Van Gool, ,https://arxiv.org/abs/2402.02235,,2402.02235.pdf,Image Fusion via Vision-Language Model,"Image fusion integrates essential information from multiple source images
+into a single composite, emphasizing the highlighting structure and textures,
+and refining imperfect areas. Existing methods predominantly focus on
+pixel-level and semantic visual features for recognition. However, they
+insufficiently explore the deeper semantic information at a text-level beyond
+vision. Therefore, we introduce a novel fusion paradigm named image Fusion via
+vIsion-Language Model (FILM), for the first time, utilizing explicit textual
+information in different source images to guide image fusion. In FILM, input
+images are firstly processed to generate semantic prompts, which are then fed
+into ChatGPT to obtain rich textual descriptions. These descriptions are fused
+in the textual domain and guide the extraction of crucial visual features from
+the source images through cross-attention, resulting in a deeper level of
+contextual understanding directed by textual semantic information. The final
+fused image is created by vision feature decoder. This paradigm achieves
+satisfactory results in four image fusion tasks: infrared-visible, medical,
+multi-exposure, and multi-focus image fusion. We also propose a vision-language
+dataset containing ChatGPT-based paragraph descriptions for the ten image
+fusion datasets in four fusion tasks, facilitating future research in
+vision-language model-based image fusion. Code and dataset will be released.",cs.CV,['cs.CV']
+High-Quality Facial Geometry and Appearance Capture at Home,Yuxuan Han · Junfeng Lyu · Feng Xu,https://yxuhan.github.io/CoRA/index.html,https://arxiv.org/abs/2312.03442,,2312.03442.pdf,High-Quality Facial Geometry and Appearance Capture at Home,"Facial geometry and appearance capture have demonstrated tremendous success
+in 3D scanning real humans in studios. Recent works propose to democratize this
+technique while keeping the results high quality. However, they are still
+inconvenient for daily usage. In addition, they focus on an easier problem of
+only capturing facial skin. This paper proposes a novel method for high-quality
+face capture, featuring an easy-to-use system and the capability to model the
+complete face with skin, mouth interior, hair, and eyes. We reconstruct facial
+geometry and appearance from a single co-located smartphone flashlight sequence
+captured in a dim room where the flashlight is the dominant light source (e.g.
+rooms with curtains or at night). To model the complete face, we propose a
+novel hybrid representation to effectively model both eyes and other facial
+regions, along with novel techniques to learn it from images. We apply a
+combined lighting model to compactly represent real illuminations and exploit a
+morphable face albedo model as a reflectance prior to disentangle diffuse and
+specular. Experiments show that our method can capture high-quality 3D
+relightable scans.",cs.CV,['cs.CV']
+Multi-Object Tracking in the Dark,Xinzhe Wang · Kang Ma · Qiankun Liu · Yunhao Zou · Ying Fu, ,https://arxiv.org/abs/2405.06600,,2405.06600.pdf,Multi-Object Tracking in the Dark,"Low-light scenes are prevalent in real-world applications (e.g. autonomous
+driving and surveillance at night). Recently, multi-object tracking in various
+practical use cases have received much attention, but multi-object tracking in
+dark scenes is rarely considered. In this paper, we focus on multi-object
+tracking in dark scenes. To address the lack of datasets, we first build a
+Low-light Multi-Object Tracking (LMOT) dataset. LMOT provides well-aligned
+low-light video pairs captured by our dual-camera system, and high-quality
+multi-object tracking annotations for all videos. Then, we propose a low-light
+multi-object tracking method, termed as LTrack. We introduce the adaptive
+low-pass downsample module to enhance low-frequency components of images
+outside the sensor noises. The degradation suppression learning strategy
+enables the model to learn invariant information under noise disturbance and
+image quality degradation. These components improve the robustness of
+multi-object tracking in dark scenes. We conducted a comprehensive analysis of
+our LMOT dataset and proposed LTrack. Experimental results demonstrate the
+superiority of the proposed method and its competitiveness in real night
+low-light scenes. Dataset and Code: https: //github.com/ying-fu/LMOT",cs.CV,['cs.CV']
+VideoCon: Robust Video-Language Alignment via Contrast Captions,Hritik Bansal · Yonatan Bitton · Idan Szpektor · Kai-Wei Chang · Aditya Grover, ,https://arxiv.org/abs/2311.10111,,2311.10111.pdf,VideoCon: Robust Video-Language Alignment via Contrast Captions,"Despite being (pre)trained on a massive amount of data, state-of-the-art
+video-language alignment models are not robust to semantically-plausible
+contrastive changes in the video captions. Our work addresses this by
+identifying a broad spectrum of contrast misalignments, such as replacing
+entities, actions, and flipping event order, which alignment models should be
+robust against. To this end, we introduce the VideoCon, a video-language
+alignment dataset constructed by a large language model that generates
+plausible contrast video captions and explanations for differences between
+original and contrast video captions. Then, a generative video-language model
+is finetuned with VideoCon to assess video-language entailment and generate
+explanations. Our VideoCon-based alignment model significantly outperforms
+current models. It exhibits a 12-point increase in AUC for the video-language
+alignment task on human-generated contrast captions. Finally, our model sets
+new state of the art zero-shot performance in temporally-extensive
+video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video
+question answering (ATP-Hard). Moreover, our model shows superior performance
+on novel videos and human-crafted captions and explanations. Our code and data
+are available at https://github.com/Hritikbansal/videocon.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding,Sicong Leng · Hang Zhang · Guanzheng Chen · Xin Li · Shijian Lu · Chunyan Miao · Lidong Bing, ,https://arxiv.org/abs/2311.16922,,2311.16922.pdf,Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding,"Large Vision-Language Models (LVLMs) have advanced considerably, intertwining
+visual recognition and language understanding to generate content that is not
+only coherent but also contextually attuned. Despite their success, LVLMs still
+suffer from the issue of object hallucinations, where models generate plausible
+yet incorrect outputs that include objects that do not exist in the images. To
+mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple
+and training-free method that contrasts output distributions derived from
+original and distorted visual inputs. The proposed VCD effectively reduces the
+over-reliance on statistical bias and unimodal priors, two essential causes of
+object hallucinations. This adjustment ensures the generated content is closely
+grounded to visual inputs, resulting in contextually accurate outputs. Our
+experiments show that VCD, without either additional training or the usage of
+external tools, significantly mitigates the object hallucination issue across
+different LVLM families. Beyond mitigating object hallucinations, VCD also
+excels in general LVLM benchmarks, highlighting its wide-ranging applicability.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+SHINOBI: SHape and Illumination using Neural Object decomposition via BRDF optimization and Inverse rendering from unconstrained Image collections,Andreas Engelhardt · Amit Raj · Mark Boss · Yunzhi Zhang · Abhishek Kar · Yuanzhen Li · Ricardo Martin-Brualla · Jonathan T. Barron · Deqing Sun · Hendrik Lensch · Varun Jampani, ,https://arxiv.org/abs/2401.10171,,2401.10171.pdf,SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild,"We present SHINOBI, an end-to-end framework for the reconstruction of shape,
+material, and illumination from object images captured with varying lighting,
+pose, and background. Inverse rendering of an object based on unconstrained
+image collections is a long-standing challenge in computer vision and graphics
+and requires a joint optimization over shape, radiance, and pose. We show that
+an implicit shape representation based on a multi-resolution hash encoding
+enables faster and robust shape reconstruction with joint camera alignment
+optimization that outperforms prior work. Further, to enable the editing of
+illumination and object reflectance (i.e. material) we jointly optimize BRDF
+and illumination together with the object's shape. Our method is class-agnostic
+and works on in-the-wild image collections of objects to produce relightable 3D
+assets for several use cases such as AR/VR, movies, games, etc. Project page:
+https://shinobi.aengelhardt.com Video:
+https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be",cs.CV,"['cs.CV', 'cs.GR']"
+HEAL-SWIN: A Vision Transformer On The Sphere,Oscar Carlsson · Jan E. Gerken · Hampus Linander · Heiner Spiess · Fredrik Ohlsson · Christoffer Petersson · Daniel Persson, ,https://arxiv.org/abs/2307.07313,,2307.07313.pdf,HEAL-SWIN: A Vision Transformer On The Sphere,"High-resolution wide-angle fisheye images are becoming more and more
+important for robotics applications such as autonomous driving. However, using
+ordinary convolutional neural networks or vision transformers on this data is
+problematic due to projection and distortion losses introduced when projecting
+to a rectangular grid on the plane. We introduce the HEAL-SWIN transformer,
+which combines the highly uniform Hierarchical Equal Area iso-Latitude
+Pixelation (HEALPix) grid used in astrophysics and cosmology with the
+Hierarchical Shifted-Window (SWIN) transformer to yield an efficient and
+flexible model capable of training on high-resolution, distortion-free
+spherical data. In HEAL-SWIN, the nested structure of the HEALPix grid is used
+to perform the patching and windowing operations of the SWIN transformer,
+enabling the network to process spherical representations with minimal
+computational overhead. We demonstrate the superior performance of our model on
+both synthetic and real automotive datasets, as well as a selection of other
+image datasets, for semantic segmentation, depth regression and classification
+tasks. Our code is publicly available at
+https://github.com/JanEGerken/HEAL-SWIN.",cs.CV,"['cs.CV', 'cs.LG']"
+BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning,Siyuan Liang · Mingli Zhu · Aishan Liu · Baoyuan Wu · Xiaochun Cao · Ee-Chien Chang, ,https://arxiv.org/abs/2311.12075,,2311.12075.pdf,BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning,"Studying backdoor attacks is valuable for model copyright protection and
+enhancing defenses. While existing backdoor attacks have successfully infected
+multimodal contrastive learning models such as CLIP, they can be easily
+countered by specialized backdoor defenses for MCL models. This paper reveals
+the threats in this practical scenario that backdoor attacks can remain
+effective even after defenses and introduces the \emph{\toolns} attack, which
+is resistant to backdoor detection and model fine-tuning defenses. To achieve
+this, we draw motivations from the perspective of the Bayesian rule and propose
+a dual-embedding guided framework for backdoor attacks. Specifically, we ensure
+that visual trigger patterns approximate the textual target semantics in the
+embedding space, making it challenging to detect the subtle parameter
+variations induced by backdoor learning on such natural trigger patterns.
+Additionally, we optimize the visual trigger patterns to align the poisoned
+samples with target vision features in order to hinder the backdoor unlearning
+through clean fine-tuning. Extensive experiments demonstrate that our attack
+significantly outperforms state-of-the-art baselines (+45.3% ASR) in the
+presence of SoTA backdoor defenses, rendering these mitigation and detection
+strategies virtually ineffective. Furthermore, our approach effectively attacks
+some more rigorous scenarios like downstream tasks. We believe that this paper
+raises awareness regarding the potential threats associated with the practical
+application of multimodal contrastive learning and encourages the development
+of more robust defense mechanisms.",cs.CV,['cs.CV']
+Flexible Depth Completion for Sparse and Varying Point Densities,Jinhyung Park · Yu-Jhe Li · Kris Kitani, ,https://arxiv.org/abs/2405.09342,,2405.09342.pdf,Progressive Depth Decoupling and Modulating for Flexible Depth Completion,"Image-guided depth completion aims at generating a dense depth map from
+sparse LiDAR data and RGB image. Recent methods have shown promising
+performance by reformulating it as a classification problem with two sub-tasks:
+depth discretization and probability prediction. They divide the depth range
+into several discrete depth values as depth categories, serving as priors for
+scene depth distributions. However, previous depth discretization methods are
+easy to be impacted by depth distribution variations across different scenes,
+resulting in suboptimal scene depth distribution priors. To address the above
+problem, we propose a progressive depth decoupling and modulating network,
+which incrementally decouples the depth range into bins and adaptively
+generates multi-scale dense depth maps in multiple stages. Specifically, we
+first design a Bins Initializing Module (BIM) to construct the seed bins by
+exploring the depth distribution information within a sparse depth map,
+adapting variations of depth distribution. Then, we devise an incremental depth
+decoupling branch to progressively refine the depth distribution information
+from global to local. Meanwhile, an adaptive depth modulating branch is
+developed to progressively improve the probability representation from
+coarse-grained to fine-grained. And the bi-directional information interactions
+are proposed to strengthen the information interaction between those two
+branches (sub-tasks) for promoting information complementation in each branch.
+Further, we introduce a multi-scale supervision mechanism to learn the depth
+distribution information in latent features and enhance the adaptation
+capability across different scenes. Experimental results on public datasets
+demonstrate that our method outperforms the state-of-the-art methods. The code
+will be open-sourced at [this https URL](https://github.com/Cisse-away/PDDM).",cs.CV,['cs.CV']
+Neural Fields as Distributions: Signal Processing Beyond Euclidean Space,Daniel Rebain · Soroosh Yazdani · Kwang Moo Yi · Andrea Tagliasacchi, ,https://arxiv.org/abs/2404.13024,,,BANF: Band-limited Neural Fields for Levels of Detail Reconstruction,"Largely due to their implicit nature, neural fields lack a direct mechanism
+for filtering, as Fourier analysis from discrete signal processing is not
+directly applicable to these representations. Effective filtering of neural
+fields is critical to enable level-of-detail processing in downstream
+applications, and support operations that involve sampling the field on regular
+grids (e.g. marching cubes). Existing methods that attempt to decompose neural
+fields in the frequency domain either resort to heuristics or require extensive
+modifications to the neural field architecture. We show that via a simple
+modification, one can obtain neural fields that are low-pass filtered, and in
+turn show how this can be exploited to obtain a frequency decomposition of the
+entire signal. We demonstrate the validity of our technique by investigating
+level-of-detail reconstruction, and showing how coarser representations can be
+computed effectively.",cs.CV,"['cs.CV', 'eess.IV']"
+Accelerating Neural Field Training via Soft Mining,Shakiba Kheradmand · Daniel Rebain · Gopal Sharma · Hossam Isack · Abhishek Kar · Andrea Tagliasacchi · Kwang Moo Yi, ,https://arxiv.org/abs/2312.00075,,2312.00075.pdf,Accelerating Neural Field Training via Soft Mining,"We present an approach to accelerate Neural Field training by efficiently
+selecting sampling locations. While Neural Fields have recently become popular,
+it is often trained by uniformly sampling the training domain, or through
+handcrafted heuristics. We show that improved convergence and final training
+quality can be achieved by a soft mining technique based on importance
+sampling: rather than either considering or ignoring a pixel completely, we
+weigh the corresponding loss by a scalar. To implement our idea we use Langevin
+Monte-Carlo sampling. We show that by doing so, regions with higher error are
+being selected more frequently, leading to more than 2x improvement in
+convergence speed. The code and related resources for this study are publicly
+available at https://ubc-vision.github.io/nf-soft-mining/.",cs.CV,['cs.CV']
+Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences,Minyoung Hwang · Luca Weihs · Chanwoo Park · Kimin Lee · Aniruddha Kembhavi · Kiana Ehsani, ,https://arxiv.org/abs/2312.09337,,2312.09337.pdf,Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences,"Customizing robotic behaviors to be aligned with diverse human preferences is
+an underexplored challenge in the field of embodied AI. In this paper, we
+present Promptable Behaviors, a novel framework that facilitates efficient
+personalization of robotic agents to diverse human preferences in complex
+environments. We use multi-objective reinforcement learning to train a single
+policy adaptable to a broad spectrum of preferences. We introduce three
+distinct methods to infer human preferences by leveraging different types of
+interactions: (1) human demonstrations, (2) preference feedback on trajectory
+comparisons, and (3) language instructions. We evaluate the proposed method in
+personalized object-goal navigation and flee navigation tasks in ProcTHOR and
+RoboTHOR, demonstrating the ability to prompt agent behaviors to satisfy human
+preferences in various scenarios. Project page:
+https://promptable-behaviors.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Step differences in instructional video,Tushar Nagarajan · Lorenzo Torresani, ,https://arxiv.org/abs/2404.16222,,2404.16222.pdf,Step Differences in Instructional Video,"Comparing a user video to a reference how-to video is a key requirement for
+AR/VR technology delivering personalized assistance tailored to the user's
+progress. However, current approaches for language-based assistance can only
+answer questions about a single video. We propose an approach that first
+automatically generates large amounts of visual instruction tuning data
+involving pairs of videos from HowTo100M by leveraging existing step
+annotations and accompanying narrations, and then trains a video-conditioned
+language model to jointly reason across multiple raw videos. Our model achieves
+state-of-the-art performance at identifying differences between video pairs and
+ranking videos based on the severity of these differences, and shows promising
+ability to perform general reasoning over multiple videos.",cs.CV,['cs.CV']
+LEMON: Learning 3D Human-Object Interaction Relation from 2D Images,Yuhang Yang · Wei Zhai · Hongchen Luo · Yang Cao · Zheng-Jun Zha,https://yyvhang.github.io/LEMON/,https://arxiv.org/abs/2312.08963,,2312.08963.pdf,LEMON: Learning 3D Human-Object Interaction Relation from 2D Images,"Learning 3D human-object interaction relation is pivotal to embodied AI and
+interaction modeling. Most existing methods approach the goal by learning to
+predict isolated interaction elements, e.g., human contact, object affordance,
+and human-object spatial relation, primarily from the perspective of either the
+human or the object. Which underexploit certain correlations between the
+interaction counterparts (human and object), and struggle to address the
+uncertainty in interactions. Actually, objects' functionalities potentially
+affect humans' interaction intentions, which reveals what the interaction is.
+Meanwhile, the interacting humans and objects exhibit matching geometric
+structures, which presents how to interact. In light of this, we propose
+harnessing these inherent correlations between interaction counterparts to
+mitigate the uncertainty and jointly anticipate the above interaction elements
+in 3D space. To achieve this, we present LEMON (LEarning 3D huMan-Object
+iNteraction relation), a unified model that mines interaction intentions of the
+counterparts and employs curvatures to guide the extraction of geometric
+correlations, combining them to anticipate the interaction elements. Besides,
+the 3D Interaction Relation dataset (3DIR) is collected to serve as the test
+bed for training and evaluation. Extensive experiments demonstrate the
+superiority of LEMON over methods estimating each element in isolation.",cs.CV,['cs.CV']
+Physical Property Understanding from Language-Embedded Feature Fields,Albert J. Zhai · Yuan Shen · Emily Y. Chen · Gloria Wang · Xinlei Wang · Sheng Wang · Kaiyu Guan · Shenlong Wang, ,https://arxiv.org/abs/2404.04242,,2404.04242.pdf,Physical Property Understanding from Language-Embedded Feature Fields,"Can computers perceive the physical properties of objects solely through
+vision? Research in cognitive science and vision science has shown that humans
+excel at identifying materials and estimating their physical properties based
+purely on visual appearance. In this paper, we present a novel approach for
+dense prediction of the physical properties of objects using a collection of
+images. Inspired by how humans reason about physics through vision, we leverage
+large language models to propose candidate materials for each object. We then
+construct a language-embedded point cloud and estimate the physical properties
+of each 3D point using a zero-shot kernel regression approach. Our method is
+accurate, annotation-free, and applicable to any object in the open world.
+Experiments demonstrate the effectiveness of the proposed approach in various
+physical property reasoning tasks, such as estimating the mass of common
+objects, as well as other properties like friction and hardness.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval,Fang Kaipeng · Jingkuan Song · Lianli Gao · Pengpeng Zeng · Zhi-Qi Cheng · Xiyao LI · Heng Tao Shen,https://github.com/fangkaipeng/ProS,https://arxiv.org/abs/2312.12478,,2312.12478.pdf,ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval,"The goal of Universal Cross-Domain Retrieval (UCDR) is to achieve robust
+performance in generalized test scenarios, wherein data may belong to strictly
+unknown domains and categories during training. Recently, pre-trained models
+with prompt tuning have shown strong generalization capabilities and attained
+noteworthy achievements in various downstream tasks, such as few-shot learning
+and video-text retrieval. However, applying them directly to UCDR may not
+sufficiently to handle both domain shift (i.e., adapting to unfamiliar domains)
+and semantic shift (i.e., transferring to unknown categories). To this end, we
+propose \textbf{Pro}mpting-to-\textbf{S}imulate (ProS), the first method to
+apply prompt tuning for UCDR. ProS employs a two-step process to simulate
+Content-aware Dynamic Prompts (CaDP) which can impact models to produce
+generalized features for UCDR. Concretely, in Prompt Units Learning stage, we
+introduce two Prompt Units to individually capture domain and semantic
+knowledge in a mask-and-align way. Then, in Context-aware Simulator Learning
+stage, we train a Content-aware Prompt Simulator under a simulated test
+scenarios to produce the corresponding CaDP. Extensive experiments conducted on
+three benchmark datasets show that our method achieves new state-of-the-art
+performance without bringing excessive parameters. Our method is publicly
+available at https://github.com/fangkaipeng/ProS.",cs.CV,['cs.CV']
+CoDi-2: Interleaved and In-Context Any-to-Any Generation,Zineng Tang · Ziyi Yang · MAHMOUD KHADEMI · Yang Liu · Chenguang Zhu · Mohit Bansal, ,https://arxiv.org/abs/2311.18775,,2311.18775.pdf,"CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation","We present CoDi-2, a versatile and interactive Multimodal Large Language
+Model (MLLM) that can follow complex multimodal interleaved instructions,
+conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any
+input-output modality paradigm. By aligning modalities with language for both
+encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not
+only understand complex modality-interleaved instructions and in-context
+examples, but also autoregressively generate grounded and coherent multimodal
+outputs in the continuous feature space. To train CoDi-2, we build a
+large-scale generation dataset encompassing in-context multimodal instructions
+across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot
+capabilities for multimodal generation, such as in-context learning, reasoning,
+and compositionality of any-to-any modality generation through multi-round
+interactive conversation. CoDi-2 surpasses previous domain-specific models on
+tasks such as subject-driven image generation, vision transformation, and audio
+editing. CoDi-2 signifies a substantial breakthrough in developing a
+comprehensive multimodal foundation model adept at interpreting in-context
+language-vision-audio interleaved instructions and producing multimodal
+outputs.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG', 'cs.SD', 'eess.AS']"
+EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models,Jingyuan Yang · Jiawei Feng · Hui Huang, ,https://arxiv.org/abs/2401.04608,,2401.04608.pdf,EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models,"Recent years have witnessed remarkable progress in image generation task,
+where users can create visually astonishing images with high-quality. However,
+existing text-to-image diffusion models are proficient in generating concrete
+concepts (dogs) but encounter challenges with more abstract ones (emotions).
+Several efforts have been made to modify image emotions with color and style
+adjustments, facing limitations in effectively conveying emotions with fixed
+image contents. In this work, we introduce Emotional Image Content Generation
+(EICG), a new task to generate semantic-clear and emotion-faithful images given
+emotion categories. Specifically, we propose an emotion space and construct a
+mapping network to align it with the powerful Contrastive Language-Image
+Pre-training (CLIP) space, providing a concrete interpretation of abstract
+emotions. Attribute loss and emotion confidence are further proposed to ensure
+the semantic diversity and emotion fidelity of the generated images. Our method
+outperforms the state-of-the-art text-to-image approaches both quantitatively
+and qualitatively, where we derive three custom metrics, i.e., emotion
+accuracy, semantic clarity and semantic diversity. In addition to generation,
+our method can help emotion understanding and inspire emotional art design.",cs.CV,['cs.CV']
+Rapid 3D Model Generation with Intuitive 3D Input,Tianrun Chen · Chaotao Ding · Shangzhan Zhang · Chunan Yu · Ying Zang · Zejian Li · Sida Peng · Lingyun Sun, ,https://ar5iv.labs.arxiv.org/html/2309.13006,,2309.13006.pdf,Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches,"The rapid development of AR/VR brings tremendous demands for 3D content.
+While the widely-used Computer-Aided Design (CAD) method requires a
+time-consuming and labor-intensive modeling process, sketch-based 3D modeling
+offers a potential solution as a natural form of computer-human interaction.
+However, the sparsity and ambiguity of sketches make it challenging to generate
+high-fidelity content reflecting creators' ideas. Precise drawing from multiple
+views or strategic step-by-step drawings is often required to tackle the
+challenge but is not friendly to novice users. In this work, we introduce a
+novel end-to-end approach, Deep3DSketch+, which performs 3D modeling using only
+a single free-hand sketch without inputting multiple sketches or view
+information. Specifically, we introduce a lightweight generation network for
+efficient inference in real-time and a structural-aware adversarial training
+approach with a Stroke Enhancement Module (SEM) to capture the structural
+information to facilitate learning of the realistic and fine-detailed shape
+structures for high-fidelity performance. Extensive experiments demonstrated
+the effectiveness of our approach with the state-of-the-art (SOTA) performance
+on both synthetic and real datasets.",cs.CV,['cs.CV']
+L-MAGIC: Language Model Assisted Generation of Images with Consistency,zhipeng cai · Matthias Mueller · Reiner Birkl · Diana Wofk · Shao-Yen Tseng · JunDa Cheng · Gabriela Ben Melech Stan · Vasudev Lal · Michael Paulitsch, ,https://arxiv.org/abs/2311.16500,,2311.16500.pdf,LLMGA: Multimodal Large Language Model based Generation Assistant,"In this paper, we introduce a Multimodal Large Language Model-based
+Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and
+proficiency in reasoning, comprehension, and response inherent in Large
+Language Models (LLMs) to assist users in image generation and editing.
+Diverging from existing approaches where Multimodal Large Language Models
+(MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our
+LLMGA provides a detailed language generation prompt for precise control over
+SD. This not only augments LLM context understanding but also reduces noise in
+generation prompts, yields images with more intricate and precise content, and
+elevates the interpretability of the network. To this end, we curate a
+comprehensive dataset comprising prompt refinement, similar image generation,
+inpainting \& outpainting, and instruction-based editing. Moreover, we propose
+a two-stage training scheme. In the first stage, we train the MLLM to grasp the
+properties of image generation and editing, enabling it to generate detailed
+prompts. In the second stage, we optimize SD to align with the MLLM's
+generation prompts. Additionally, we propose a reference-based restoration
+network to alleviate texture, brightness, and contrast disparities between
+generated and preserved regions during inpainting and outpainting. Extensive
+results show that LLMGA has promising generation and editing capabilities and
+can enable more flexible and expansive applications in an interactive manner.",cs.CV,['cs.CV']
+Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering,Kim Youwang · Tae-Hyun Oh · Gerard Pons-Moll, ,https://arxiv.org/abs/2312.11360v1,,2312.11360v1.pdf,Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering,"We present Paint-it, a text-driven high-fidelity texture map synthesis method
+for 3D meshes via neural re-parameterized texture optimization. Paint-it
+synthesizes texture maps from a text description by
+synthesis-through-optimization, exploiting the Score-Distillation Sampling
+(SDS). We observe that directly applying SDS yields undesirable texture quality
+due to its noisy gradients. We reveal the importance of texture
+parameterization when using SDS. Specifically, we propose Deep Convolutional
+Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes
+the physically-based rendering (PBR) texture maps with randomly initialized
+convolution-based neural kernels, instead of a standard pixel-based
+parameterization. We show that DC-PBR inherently schedules the optimization
+curriculum according to texture frequency and naturally filters out the noisy
+signals from SDS. In experiments, Paint-it obtains remarkable quality PBR
+texture maps within 15 min., given only a text description. We demonstrate the
+generalizability and practicality of Paint-it by synthesizing high-quality
+texture maps for large-scale mesh datasets and showing test-time applications
+such as relighting and material control using a popular graphics engine.
+Project page: https://kim-youwang.github.io/paint-it",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation,Peng Lu · Tao Jiang · Yining Li · Xiangtai Li · Kai Chen · Wenming Yang,https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo,https://arxiv.org/abs/2312.07526,,2312.07526.pdf,RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation,"Real-time multi-person pose estimation presents significant challenges in
+balancing speed and precision. While two-stage top-down methods slow down as
+the number of people in the image increases, existing one-stage methods often
+fail to simultaneously deliver high accuracy and real-time performance. This
+paper introduces RTMO, a one-stage pose estimation framework that seamlessly
+integrates coordinate classification by representing keypoints using dual 1-D
+heatmaps within the YOLO architecture, achieving accuracy comparable to
+top-down methods while maintaining high speed. We propose a dynamic coordinate
+classifier and a tailored loss function for heatmap learning, specifically
+designed to address the incompatibilities between coordinate classification and
+dense prediction models. RTMO outperforms state-of-the-art one-stage pose
+estimators, achieving 1.1% higher AP on COCO while operating about 9 times
+faster with the same backbone. Our largest model, RTMO-l, attains 74.8% AP on
+COCO val2017 and 141 FPS on a single V100 GPU, demonstrating its efficiency and
+accuracy. The code and models are available at
+https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo.",cs.CV,['cs.CV']
+Multi-Session SLAM using Wide-Baseline Optical Flow,Lahav Lipson · Jia Deng, ,https://arxiv.org/abs/2404.15263,,2404.15263.pdf,Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization,"We introduce a new system for Multi-Session SLAM, which tracks camera motion
+across multiple disjoint videos under a single global reference. Our approach
+couples the prediction of optical flow with solver layers to estimate camera
+pose. The backbone is trained end-to-end using a novel differentiable solver
+for wide-baseline two-view pose. The full system can connect disjoint
+sequences, perform visual odometry, and global optimization. Compared to
+existing approaches, our design is accurate and robust to catastrophic
+failures. Code is available at github.com/princeton-vl/MultiSlam_DiffPose",cs.CV,['cs.CV']
+Action Scene Graphs for Long-Form Understanding of Egocentric Videos,Ivan Rodin · Antonino Furnari · Kyle Min · Subarna Tripathi · Giovanni Maria Farinella,https://github.com/fpv-iplab/easg,https://arxiv.org/abs/2312.03391,,2312.03391.pdf,Action Scene Graphs for Long-Form Understanding of Egocentric Videos,"We present Egocentric Action Scene Graphs (EASGs), a new representation for
+long-form understanding of egocentric videos. EASGs extend standard
+manually-annotated representations of egocentric videos, such as verb-noun
+action labels, by providing a temporally evolving graph-based description of
+the actions performed by the camera wearer, including interacted objects, their
+relationships, and how actions unfold in time. Through a novel annotation
+procedure, we extend the Ego4D dataset by adding manually labeled Egocentric
+Action Scene Graphs offering a rich set of annotations designed for long-from
+egocentric video understanding. We hence define the EASG generation task and
+provide a baseline approach, establishing preliminary benchmarks. Experiments
+on two downstream tasks, egocentric action anticipation and egocentric activity
+summarization, highlight the effectiveness of EASGs for long-form egocentric
+video understanding. We will release the dataset and the code to replicate
+experiments and annotations.",cs.CV,['cs.CV']
+Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations,Sangmin Lee · Bolin Lai · Fiona Ryan · Bikram Boote · James Rehg,https://sangmin-git.github.io/projects/MMSI,https://arxiv.org/abs/2403.02090,,2403.02090.pdf,Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations,"Understanding social interactions involving both verbal and non-verbal cues
+is essential for effectively interpreting social situations. However, most
+prior works on multimodal social cues focus predominantly on single-person
+behaviors or rely on holistic visual representations that are not aligned to
+utterances in multi-party environments. Consequently, they are limited in
+modeling the intricate dynamics of multi-party interactions. In this paper, we
+introduce three new challenging tasks to model the fine-grained dynamics
+between multiple people: speaking target identification, pronoun coreference
+resolution, and mentioned player prediction. We contribute extensive data
+annotations to curate these new challenges in social deduction game settings.
+Furthermore, we propose a novel multimodal baseline that leverages densely
+aligned language-visual representations by synchronizing visual features with
+their corresponding utterances. This facilitates concurrently capturing verbal
+and non-verbal cues pertinent to social reasoning. Experiments demonstrate the
+effectiveness of the proposed approach with densely aligned multimodal
+representations in modeling fine-grained social interactions. Project website:
+https://sangmin-git.github.io/projects/MMSI.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Splatter Image: Ultra-Fast Single-View 3D Reconstruction,Stanislaw Szymanowicz · Christian Rupprecht · Andrea Vedaldi, ,https://arxiv.org/abs/2312.13150,,2312.13150.pdf,Splatter Image: Ultra-Fast Single-View 3D Reconstruction,"We introduce the \method, an ultra-efficient approach for monocular 3D object
+reconstruction. Splatter Image is based on Gaussian Splatting, which allows
+fast and high-quality reconstruction of 3D scenes from multiple images. We
+apply Gaussian Splatting to monocular reconstruction by learning a neural
+network that, at test time, performs reconstruction in a feed-forward manner,
+at 38 FPS. Our main innovation is the surprisingly straightforward design of
+this network, which, using 2D operators, maps the input image to one 3D
+Gaussian per pixel. The resulting set of Gaussians thus has the form an image,
+the Splatter Image. We further extend the method take several images as input
+via cross-view attention. Owning to the speed of the renderer (588 FPS), we use
+a single GPU for training while generating entire images at each iteration to
+optimize perceptual metrics like LPIPS. On several synthetic, real,
+multi-category and large-scale benchmark datasets, we achieve better results in
+terms of PSNR, LPIPS, and other metrics while training and evaluating much
+faster than prior works. Code, models, demo and more results are available at
+https://szymanowiczs.github.io/splatter-image.",cs.CV,['cs.CV']
+MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception,Thien-Minh Nguyen · Shenghai Yuan · Thien Nguyen · Pengyu Yin · Haozhi Cao · Lihua Xie · Maciej Wozniak · Patric Jensfelt · Marko Thiel · Justin Ziegenbein · Noel Blunder, ,https://arxiv.org/abs/2403.11496,,2403.11496.pdf,MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception,"Perception plays a crucial role in various robot applications. However,
+existing well-annotated datasets are biased towards autonomous driving
+scenarios, while unlabelled SLAM datasets are quickly over-fitted, and often
+lack environment and domain variations. To expand the frontier of these fields,
+we introduce a comprehensive dataset named MCD (Multi-Campus Dataset),
+featuring a wide range of sensing modalities, high-accuracy ground truth, and
+diverse challenging environments across three Eurasian university campuses. MCD
+comprises both CCS (Classical Cylindrical Spinning) and NRE (Non-Repetitive
+Epicyclic) lidars, high-quality IMUs (Inertial Measurement Units), cameras, and
+UWB (Ultra-WideBand) sensors. Furthermore, in a pioneering effort, we introduce
+semantic annotations of 29 classes over 59k sparse NRE lidar scans across three
+domains, thus providing a novel challenge to existing semantic segmentation
+research upon this largely unexplored lidar modality. Finally, we propose, for
+the first time to the best of our knowledge, continuous-time ground truth based
+on optimization-based registration of lidar-inertial data on large survey-grade
+prior maps, which are also publicly released, each several times the size of
+existing ones. We conduct a rigorous evaluation of numerous state-of-the-art
+algorithms on MCD, report their performance, and highlight the challenges
+awaiting solutions from the research community.",cs.RO,"['cs.RO', 'cs.AI']"
+FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio,Chao Xu · Yang Liu · Jiazheng Xing · Weida Wang · Mingze Sun · Jun Dan · Tianxin Huang · Siyuan Li · Zhi-Qi Cheng · Ying Tai · Baigui Sun, ,https://arxiv.org/abs/2403.01901,,2403.01901.pdf,FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio,"In this paper, we abstract the process of people hearing speech, extracting
+meaningful cues, and creating various dynamically audio-consistent talking
+faces, termed Listening and Imagining, into the task of high-fidelity diverse
+talking faces generation from a single audio. Specifically, it involves two
+critical challenges: one is to effectively decouple identity, content, and
+emotion from entangled audio, and the other is to maintain intra-video
+diversity and inter-video consistency. To tackle the issues, we first dig out
+the intricate relationships among facial factors and simplify the decoupling
+process, tailoring a Progressive Audio Disentanglement for accurate facial
+geometry and semantics learning, where each stage incorporates a customized
+training module responsible for a specific factor. Secondly, to achieve
+visually diverse and audio-synchronized animation solely from input audio
+within a single model, we introduce the Controllable Coherent Frame generation,
+which involves the flexible integration of three trainable adapters with frozen
+Latent Diffusion Models (LDMs) to focus on maintaining facial geometry and
+semantics, as well as texture and temporal coherence between frames. In this
+way, we inherit high-quality diverse generation from LDMs while significantly
+improving their controllability at a low training cost. Extensive experiments
+demonstrate the flexibility and effectiveness of our method in handling this
+paradigm. The codes will be released at
+https://github.com/modelscope/facechain.",cs.CV,['cs.CV']
+NeISF: Neural Incident Stokes Field for Geometry and Material Estimation,Chenhao Li · Taishi Ono · Takeshi Uemori · Hajime Mihara · Alexander Gatto · Hajime Nagahara · Yusuke Moriuchi, ,https://arxiv.org/abs/2311.13187v1,,2311.13187v1.pdf,NeISF: Neural Incident Stokes Field for Geometry and Material Estimation,"Multi-view inverse rendering is the problem of estimating the scene
+parameters such as shapes, materials, or illuminations from a sequence of
+images captured under different viewpoints. Many approaches, however, assume
+single light bounce and thus fail to recover challenging scenarios like
+inter-reflections. On the other hand, simply extending those methods to
+consider multi-bounced light requires more assumptions to alleviate the
+ambiguity. To address this problem, we propose Neural Incident Stokes Fields
+(NeISF), a multi-view inverse rendering framework that reduces ambiguities
+using polarization cues. The primary motivation for using polarization cues is
+that it is the accumulation of multi-bounced light, providing rich information
+about geometry and material. Based on this knowledge, the proposed incident
+Stokes field efficiently models the accumulated polarization effect with the
+aid of an original physically-based differentiable polarimetric renderer.
+Lastly, experimental results show that our method outperforms the existing
+works in synthetic and real scenarios.",cs.CV,['cs.CV']
+PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection,Kuan-Chih Huang · Weijie Lyu · Ming-Hsuan Yang · Yi-Hsuan Tsai, ,https://arxiv.org/abs/2312.08371,,2312.08371.pdf,PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection,"Recent temporal LiDAR-based 3D object detectors achieve promising performance
+based on the two-stage proposal-based approach. They generate 3D box candidates
+from the first-stage dense detector, followed by different temporal aggregation
+methods. However, these approaches require per-frame objects or whole point
+clouds, posing challenges related to memory bank utilization. Moreover, point
+clouds and trajectory features are combined solely based on concatenation,
+which may neglect effective interactions between them. In this paper, we
+propose a point-trajectory transformer with long short-term memory for
+efficient temporal 3D object detection. To this end, we only utilize point
+clouds of current-frame objects and their historical trajectories as input to
+minimize the memory bank storage requirement. Furthermore, we introduce modules
+to encode trajectory features, focusing on long short-term and future-aware
+perspectives, and then effectively aggregate them with point cloud features. We
+conduct extensive experiments on the large-scale Waymo dataset to demonstrate
+that our approach performs well against state-of-the-art methods. Code and
+models will be made publicly available at https://github.com/kuanchihhuang/PTT.",cs.CV,['cs.CV']
+SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation,Chen Sichen · Yingyi Zhang · Siming Huang · Ran Yi · Ke Fan · Ruixin Zhang · Peixian Chen · Jun Wang · Shouhong Ding · Lizhuang Ma, ,https://arxiv.org/abs/2404.03518,,2404.03518.pdf,SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation,"Recently, transformer-based methods have achieved state-of-the-art prediction
+quality on human pose estimation(HPE). Nonetheless, most of these
+top-performing transformer-based models are too computation-consuming and
+storage-demanding to deploy on edge computing platforms. Those
+transformer-based models that require fewer resources are prone to
+under-fitting due to their smaller scale and thus perform notably worse than
+their larger counterparts. Given this conundrum, we introduce SDPose, a new
+self-distillation method for improving the performance of small
+transformer-based models. To mitigate the problem of under-fitting, we design a
+transformer module named Multi-Cycled Transformer(MCT) based on multiple-cycled
+forwards to more fully exploit the potential of small model parameters.
+Further, in order to prevent the additional inference compute-consuming brought
+by MCT, we introduce a self-distillation scheme, extracting the knowledge from
+the MCT module to a naive forward model. Specifically, on the MSCOCO validation
+dataset, SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs.
+Furthermore, SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset
+with 6.2M parameters and 4.7 GFLOPs, achieving a new state-of-the-art among
+predominant tiny neural network methods. Our code is available at
+https://github.com/MartyrPenink/SDPose.",cs.CV,['cs.CV']
+Uncertainty-aware Action Decoupling Transformer for Action Anticipation,Hongji Guo · Nakul Agarwal · Shao-Yuan Lo · Kwonjoon Lee · Qiang Ji, ,https://arxiv.org/abs/2309.16397,,2309.16397.pdf,Uncertainty-Aware Decision Transformer for Stochastic Driving Environments,"Offline Reinforcement Learning (RL) has emerged as a promising framework for
+learning policies without active interactions, making it especially appealing
+for autonomous driving tasks. Recent successes of Transformers inspire casting
+offline RL as sequence modeling, which performs well in long-horizon tasks.
+However, they are overly optimistic in stochastic environments with incorrect
+assumptions that the same goal can be consistently achieved by identical
+actions. In this paper, we introduce an UNcertainty-awaRE deciSion Transformer
+(UNREST) for planning in stochastic driving environments without introducing
+additional transition or complex generative models. Specifically, UNREST
+estimates state uncertainties by the conditional mutual information between
+transitions and returns, and segments sequences accordingly. Discovering the
+`uncertainty accumulation' and `temporal locality' properties of driving
+environments, UNREST replaces the global returns in decision transformers with
+less uncertain truncated returns, to learn from true outcomes of agent actions
+rather than environment transitions. We also dynamically evaluate environmental
+uncertainty during inference for cautious planning. Extensive experimental
+results demonstrate UNREST's superior performance in various driving scenarios
+and the power of our uncertainty estimation strategy.",cs.LG,"['cs.LG', 'cs.AI']"
+Robust Overfitting Does Matter: Test-Time Adversarial Purification With FGSM,Linyu Tang · Lei Zhang, ,https://arxiv.org/abs/2403.11448,,2403.11448.pdf,Robust Overfitting Does Matter: Test-Time Adversarial Purification With FGSM,"Numerous studies have demonstrated the susceptibility of deep neural networks
+(DNNs) to subtle adversarial perturbations, prompting the development of many
+advanced adversarial defense methods aimed at mitigating adversarial attacks.
+Current defense strategies usually train DNNs for a specific adversarial attack
+method and can achieve good robustness in defense against this type of
+adversarial attack. Nevertheless, when subjected to evaluations involving
+unfamiliar attack modalities, empirical evidence reveals a pronounced
+deterioration in the robustness of DNNs. Meanwhile, there is a trade-off
+between the classification accuracy of clean examples and adversarial examples.
+Most defense methods often sacrifice the accuracy of clean examples in order to
+improve the adversarial robustness of DNNs. To alleviate these problems and
+enhance the overall robust generalization of DNNs, we propose the Test-Time
+Pixel-Level Adversarial Purification (TPAP) method. This approach is based on
+the robust overfitting characteristic of DNNs to the fast gradient sign method
+(FGSM) on training and test datasets. It utilizes FGSM for adversarial
+purification, to process images for purifying unknown adversarial perturbations
+from pixels at testing time in a ""counter changes with changelessness"" manner,
+thereby enhancing the defense capability of DNNs against various unknown
+adversarial attacks. Extensive experimental results show that our method can
+effectively improve both overall robust generalization of DNNs, notably over
+previous methods.",cs.CV,['cs.CV']
+Compositional Video Understanding with Spatiotemporal Structure-based Transformers,Hoyeoung Yun · Jinwoo Ahn · Minseo Kim · Eun-Sol Kim, ,https://arxiv.org/abs/2401.10831,,2401.10831.pdf,Understanding Video Transformers via Universal Concept Discovery,"This paper studies the problem of concept-based interpretability of
+transformer representations for videos. Concretely, we seek to explain the
+decision-making process of video transformers based on high-level,
+spatiotemporal concepts that are automatically discovered. Prior research on
+concept-based interpretability has concentrated solely on image-level tasks.
+Comparatively, video models deal with the added temporal dimension, increasing
+complexity and posing challenges in identifying dynamic concepts over time. In
+this work, we systematically address these challenges by introducing the first
+Video Transformer Concept Discovery (VTCD) algorithm. To this end, we propose
+an efficient approach for unsupervised identification of units of video
+transformer representations - concepts, and ranking their importance to the
+output of a model. The resulting concepts are highly interpretable, revealing
+spatio-temporal reasoning mechanisms and object-centric representations in
+unstructured video models. Performing this analysis jointly over a diverse set
+of supervised and self-supervised representations, we discover that some of
+these mechanism are universal in video transformers. Finally, we show that VTCD
+can be used for fine-grained action recognition and video object segmentation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration,Mingyuan Meng · Dagan Feng · Lei Bi · Jinman Kim, ,https://arxiv.org/abs/2311.16707,,2311.16707.pdf,Full-resolution MLPs Empower Medical Dense Prediction,"Dense prediction is a fundamental requirement for many medical vision tasks
+such as medical image restoration, registration, and segmentation. The most
+popular vision model, Convolutional Neural Networks (CNNs), has reached
+bottlenecks due to the intrinsic locality of convolution operations. Recently,
+transformers have been widely adopted for dense prediction for their capability
+to capture long-range visual dependence. However, due to the high computational
+complexity and large memory consumption of self-attention operations,
+transformers are usually used at downsampled feature resolutions. Such usage
+cannot effectively leverage the tissue-level textural information available
+only at the full image resolution. This textural information is crucial for
+medical dense prediction as it can differentiate the subtle human anatomy in
+medical images. In this study, we hypothesize that Multi-layer Perceptrons
+(MLPs) are superior alternatives to transformers in medical dense prediction
+where tissue-level details dominate the performance, as MLPs enable long-range
+dependence at the full image resolution. To validate our hypothesis, we develop
+a full-resolution hierarchical MLP framework that uses MLPs beginning from the
+full image resolution. We evaluate this framework with various MLP blocks on a
+wide range of medical dense prediction tasks including restoration,
+registration, and segmentation. Extensive experiments on six public
+well-benchmarked datasets show that, by simply using MLPs at full resolution,
+our framework outperforms its CNN and transformer counterparts and achieves
+state-of-the-art performance on various medical dense prediction tasks.",eess.IV,"['eess.IV', 'cs.CV']"
+Multimodal Representation Learning by Alternating Unimodal Adaptation,Xiaohui Zhang · Xiaohui Zhang · Jaehong Yoon · Mohit Bansal · Huaxiu Yao, ,https://arxiv.org/abs/2311.10707,,2311.10707.pdf,Multimodal Representation Learning by Alternating Unimodal Adaptation,"Multimodal learning, which integrates data from diverse sensory modes, plays
+a pivotal role in artificial intelligence. However, existing multimodal
+learning methods often struggle with challenges where some modalities appear
+more dominant than others during multimodal learning, resulting in suboptimal
+performance. To address this challenge, we propose MLA (Multimodal Learning
+with Alternating Unimodal Adaptation). MLA reframes the conventional joint
+multimodal learning process by transforming it into an alternating unimodal
+learning process, thereby minimizing interference between modalities.
+Simultaneously, it captures cross-modal interactions through a shared head,
+which undergoes continuous optimization across different modalities. This
+optimization process is controlled by a gradient modification mechanism to
+prevent the shared head from losing previously acquired information. During the
+inference phase, MLA utilizes a test-time uncertainty-based model fusion
+mechanism to integrate multimodal information. Extensive experiments are
+conducted on five diverse datasets, encompassing scenarios with complete
+modalities and scenarios with missing modalities. These experiments demonstrate
+the superiority of MLA over competing prior approaches. Our code is available
+at
+https://github.com/Cecile-hi/Multimodal-Learning-with-Alternating-Unimodal-Adaptation.",cs.LG,"['cs.LG', 'cs.CV']"
+Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition,Xiang Li · Jinglu Wang · Xiaohao Xu · Xiulian Peng · Rita Singh · Yan Lu · Bhiksha Raj, ,https://arxiv.org/abs/2310.00132,,2310.00132.pdf,QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition,"Audiovisual segmentation (AVS) is a challenging task that aims to segment
+visual objects in videos according to their associated acoustic cues. With
+multiple sound sources and background disturbances involved, establishing
+robust correspondences between audio and visual contents poses unique
+challenges due to (1) complex entanglement across sound sources and (2)
+frequent changes in the occurrence of distinct sound events. Assuming sound
+events occur independently, the multi-source semantic space can be represented
+as the Cartesian product of single-source sub-spaces. We are motivated to
+decompose the multi-source audio semantics into single-source semantics for
+more effective interactions with visual content. We propose a semantic
+decomposition method based on product quantization, where the multi-source
+semantics can be decomposed and represented by several disentangled and
+noise-suppressed single-source semantics. Furthermore, we introduce a
+global-to-local quantization mechanism, which distills knowledge from stable
+global (clip-level) features into local (frame-level) ones, to handle frequent
+changes in audio semantics. Extensive experiments demonstrate that our
+semantically decomposed audio representation significantly improves AVS
+performance, e.g., +21.2% mIoU on the challenging AVS-Semantic benchmark with
+ResNet50 backbone. https://github.com/lxa9867/QSD.",cs.CV,['cs.CV']
+MuRF: Multi-Baseline Radiance Fields,Haofei Xu · Anpei Chen · Yuedong Chen · Christos Sakaridis · Yulun Zhang · Marc Pollefeys · Andreas Geiger · Fisher Yu,https://haofeixu.github.io/murf/,https://arxiv.org/abs/2312.04565v1,,2312.04565v1.pdf,MuRF: Multi-Baseline Radiance Fields,"We present Multi-Baseline Radiance Fields (MuRF), a general feed-forward
+approach to solving sparse view synthesis under multiple different baseline
+settings (small and large baselines, and different number of input views). To
+render a target novel view, we discretize the 3D space into planes parallel to
+the target image plane, and accordingly construct a target view frustum volume.
+Such a target volume representation is spatially aligned with the target view,
+which effectively aggregates relevant information from the input views for
+high-quality rendering. It also facilitates subsequent radiance field
+regression with a convolutional network thanks to its axis-aligned nature. The
+3D context modeled by the convolutional network enables our method to synthesis
+sharper scene structures than prior works. Our MuRF achieves state-of-the-art
+performance across multiple different baseline settings and diverse scenarios
+ranging from simple objects (DTU) to complex indoor and outdoor scenes
+(RealEstate10K and LLFF). We also show promising zero-shot generalization
+abilities on the Mip-NeRF 360 dataset, demonstrating the general applicability
+of MuRF.",cs.CV,['cs.CV']
+Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection,Taeheon Kim · Sebin Shin · Youngjoon Yu · Hak Gu Kim · Yong Man Ro, ,https://arxiv.org/abs/2403.01300,,2403.01300.pdf,Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection,"RGBT multispectral pedestrian detection has emerged as a promising solution
+for safety-critical applications that require day/night operations. However,
+the modality bias problem remains unsolved as multispectral pedestrian
+detectors learn the statistical bias in datasets. Specifically, datasets in
+multispectral pedestrian detection mainly distribute between ROTO (day) and
+RXTO (night) data; the majority of the pedestrian labels statistically co-occur
+with their thermal features. As a result, multispectral pedestrian detectors
+show poor generalization ability on examples beyond this statistical
+correlation, such as ROTX data. To address this problem, we propose a novel
+Causal Mode Multiplexer (CMM) framework that effectively learns the causalities
+between multispectral inputs and predictions. Moreover, we construct a new
+dataset (ROTX-MP) to evaluate modality bias in multispectral pedestrian
+detection. ROTX-MP mainly includes ROTX examples not presented in previous
+datasets. Extensive experiments demonstrate that our proposed CMM framework
+generalizes well on existing datasets (KAIST, CVC-14, FLIR) and the new
+ROTX-MP. We will release our new dataset to the public for future research.",cs.CV,['cs.CV']
+GoMVS: Geometrically Consistent Cost Aggregation for Multi-View Stereo,Jiang Wu · Rui Li · Haofei Xu · Wenxun Zhao · Yu Zhu · Jinqiu Sun · Yanning Zhang, ,https://arxiv.org/abs/2404.07992v1,,2404.07992v1.pdf,GoMVS: Geometrically Consistent Cost Aggregation for Multi-View Stereo,"Matching cost aggregation plays a fundamental role in learning-based
+multi-view stereo networks. However, directly aggregating adjacent costs can
+lead to suboptimal results due to local geometric inconsistency. Related
+methods either seek selective aggregation or improve aggregated depth in the 2D
+space, both are unable to handle geometric inconsistency in the cost volume
+effectively. In this paper, we propose GoMVS to aggregate geometrically
+consistent costs, yielding better utilization of adjacent geometries. More
+specifically, we correspond and propagate adjacent costs to the reference pixel
+by leveraging the local geometric smoothness in conjunction with surface
+normals. We achieve this by the geometric consistent propagation (GCP) module.
+It computes the correspondence from the adjacent depth hypothesis space to the
+reference depth space using surface normals, then uses the correspondence to
+propagate adjacent costs to the reference geometry, followed by a convolution
+for aggregation. Our method achieves new state-of-the-art performance on DTU,
+Tanks & Temple, and ETH3D datasets. Notably, our method ranks 1st on the Tanks
+& Temple Advanced benchmark.",cs.CV,['cs.CV']
+Test-Time Adaptation for Depth Completion,Hyoungseob Park · Anjali W Gupta · Alex Wong, ,https://arxiv.org/abs/2402.03312,,2402.03312.pdf,Test-Time Adaptation for Depth Completion,"It is common to observe performance degradation when transferring models
+trained on some (source) datasets to target testing data due to a domain gap
+between them. Existing methods for bridging this gap, such as domain adaptation
+(DA), may require the source data on which the model was trained (often not
+available), while others, i.e., source-free DA, require many passes through the
+testing data. We propose an online test-time adaptation method for depth
+completion, the task of inferring a dense depth map from a single image and
+associated sparse depth map, that closes the performance gap in a single pass.
+We first present a study on how the domain shift in each data modality affects
+model performance. Based on our observations that the sparse depth modality
+exhibits a much smaller covariate shift than the image, we design an embedding
+module trained in the source domain that preserves a mapping from features
+encoding only sparse depth to those encoding image and sparse depth. During
+test time, sparse depth features are projected using this map as a proxy for
+source domain features and are used as guidance to train a set of auxiliary
+parameters (i.e., adaptation layer) to align image and sparse depth features
+from the target test domain to that of the source domain. We evaluate our
+method on indoor and outdoor scenarios and show that it improves over baselines
+by an average of 21.1%.",cs.CV,"['cs.CV', 'cs.LG']"
+FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning,Rishub Tamirisa · Chulin Xie · Wenxuan Bao · Andy Zhou · Ron Arel · Aviv Shamsian, ,https://arxiv.org/abs/2404.02478,,2404.02478.pdf,FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning,"Standard federated learning approaches suffer when client data distributions
+have sufficient heterogeneity. Recent methods addressed the client data
+heterogeneity issue via personalized federated learning (PFL) - a class of FL
+algorithms aiming to personalize learned global knowledge to better suit the
+clients' local data distributions. Existing PFL methods usually decouple global
+updates in deep neural networks by performing personalization on particular
+layers (i.e. classifier heads) and global aggregation for the rest of the
+network. However, preselecting network layers for personalization may result in
+suboptimal storage of global knowledge. In this work, we propose FedSelect, a
+novel PFL algorithm inspired by the iterative subnetwork discovery procedure
+used for the Lottery Ticket Hypothesis. FedSelect incrementally expands
+subnetworks to personalize client parameters, concurrently conducting global
+aggregations on the remaining parameters. This approach enables the
+personalization of both client parameters and subnetwork structure during the
+training process. Finally, we show that FedSelect outperforms recent
+state-of-the-art PFL algorithms under challenging client data heterogeneity
+settings and demonstrates robustness to various real-world distributional
+shifts. Our code is available at https://github.com/lapisrocks/fedselect.",cs.LG,"['cs.LG', 'cs.AI']"
+Mitigating Motion Blur in Neural Radiance Fields with Events and Frames,Marco Cannici · Davide Scaramuzza,https://github.com/uzh-rpg/EvDeblurNeRF,https://arxiv.org/abs/2403.19780,,2403.19780.pdf,Mitigating Motion Blur in Neural Radiance Fields with Events and Frames,"Neural Radiance Fields (NeRFs) have shown great potential in novel view
+synthesis. However, they struggle to render sharp images when the data used for
+training is affected by motion blur. On the other hand, event cameras excel in
+dynamic scenes as they measure brightness changes with microsecond resolution
+and are thus only marginally affected by blur. Recent methods attempt to
+enhance NeRF reconstructions under camera motion by fusing frames and events.
+However, they face challenges in recovering accurate color content or constrain
+the NeRF to a set of predefined camera poses, harming reconstruction quality in
+challenging conditions. This paper proposes a novel formulation addressing
+these issues by leveraging both model- and learning-based modules. We
+explicitly model the blur formation process, exploiting the event double
+integral as an additional model-based prior. Additionally, we model the
+event-pixel response using an end-to-end learnable response function, allowing
+our method to adapt to non-idealities in the real event-camera sensor. We show,
+on synthetic and real data, that the proposed approach outperforms existing
+deblur NeRFs that use only frames as well as those that combine frames and
+events by +6.13dB and +2.48dB, respectively.",cs.CV,['cs.CV']
+Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection,Jongha Kim · Jihwan Park · Jinyoung Park · Jinyoung Kim · Sehyung Kim · Hyunwoo J. Kim,https://github.com/mlvlab/speaq,https://arxiv.org/abs/2403.17709,,2403.17709.pdf,Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection,"Visual Relationship Detection (VRD) has seen significant advancements with
+Transformer-based architectures recently. However, we identify two key
+limitations in a conventional label assignment for training Transformer-based
+VRD models, which is a process of mapping a ground-truth (GT) to a prediction.
+Under the conventional assignment, an unspecialized query is trained since a
+query is expected to detect every relation, which makes it difficult for a
+query to specialize in specific relations. Furthermore, a query is also
+insufficiently trained since a GT is assigned only to a single prediction,
+therefore near-correct or even correct predictions are suppressed by being
+assigned no relation as a GT. To address these issues, we propose Groupwise
+Query Specialization and Quality-Aware Multi-Assignment (SpeaQ). Groupwise
+Query Specialization trains a specialized query by dividing queries and
+relations into disjoint groups and directing a query in a specific query group
+solely toward relations in the corresponding relation group. Quality-Aware
+Multi-Assignment further facilitates the training by assigning a GT to multiple
+predictions that are significantly close to a GT in terms of a subject, an
+object, and the relation in between. Experimental results and analyses show
+that SpeaQ effectively trains specialized queries, which better utilize the
+capacity of a model, resulting in consistent performance gains with zero
+additional inference cost across multiple VRD models and benchmarks. Code is
+available at https://github.com/mlvlab/SpeaQ.",cs.CV,['cs.CV']
+LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images,Jing Zhang · Irving Fang · Hao Wu · Akshat Kaushik · Alice Rodriguez · Hanwen Zhao · Juexiao Zhang · Zhuo Zheng · Radu Iovita · Chen Feng, ,https://arxiv.org/abs/2403.13171,,2403.13171.pdf,LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images,"Lithic Use-Wear Analysis (LUWA) using microscopic images is an underexplored
+vision-for-science research area. It seeks to distinguish the worked material,
+which is critical for understanding archaeological artifacts, material
+interactions, tool functionalities, and dental records. However, this
+challenging task goes beyond the well-studied image classification problem for
+common objects. It is affected by many confounders owing to the complex wear
+mechanism and microscopic imaging, which makes it difficult even for human
+experts to identify the worked material successfully. In this paper, we
+investigate the following three questions on this unique vision task for the
+first time:(i) How well can state-of-the-art pre-trained models (like DINOv2)
+generalize to the rarely seen domain? (ii) How can few-shot learning be
+exploited for scarce microscopic images? (iii) How do the ambiguous
+magnification and sensing modality influence the classification accuracy? To
+study these, we collaborated with archaeologists and built the first
+open-source and the largest LUWA dataset containing 23,130 microscopic images
+with different magnifications and sensing modalities. Extensive experiments
+show that existing pre-trained models notably outperform human experts but
+still leave a large gap for improvements. Most importantly, the LUWA dataset
+provides an underexplored opportunity for vision and learning communities and
+complements existing image classification problems on common objects.",cs.CV,['cs.CV']
+Flow-Guided Online Stereo Rectification for Wide Baseline Stereo,Anush Kumar · Fahim Mannan · Omid Hosseini Jafari · Shile Li · Felix Heide,https://light.princeton.edu/online-stereo-recification/,https://arxiv.org/abs/2309.10314,,2309.10314.pdf,Dive Deeper into Rectifying Homography for Stereo Camera Online Self-Calibration,"Accurate estimation of stereo camera extrinsic parameters is the key to
+guarantee the performance of stereo matching algorithms. In prior arts, the
+online self-calibration of stereo cameras has commonly been formulated as a
+specialized visual odometry problem, without taking into account the principles
+of stereo rectification. In this paper, we first delve deeply into the concept
+of rectifying homography, which serves as the cornerstone for the development
+of our novel stereo camera online self-calibration algorithm, for cases where
+only a single pair of images is available. Furthermore, we introduce a simple
+yet effective solution for global optimum extrinsic parameter estimation in the
+presence of stereo video sequences. Additionally, we emphasize the
+impracticality of using three Euler angles and three components in the
+translation vectors for performance quantification. Instead, we introduce four
+new evaluation metrics to quantify the robustness and accuracy of extrinsic
+parameter estimation, applicable to both single-pair and multi-pair cases.
+Extensive experiments conducted across indoor and outdoor environments using
+various experimental setups validate the effectiveness of our proposed
+algorithm. The comprehensive evaluation results demonstrate its superior
+performance in comparison to the baseline algorithm. Our source code, demo
+video, and supplement are publicly available at mias.group/StereoCalibrator.",cs.RO,"['cs.RO', 'cs.CV']"
+Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous and Instruction-guided Driving,Brian Yang · Huangyuan Su · Nikolaos Gkanatsios · Tsung-Wei Ke · Ayush Jain · Jeff Schneider · Katerina Fragkiadaki, ,https://arxiv.org/abs/2402.06559,,2402.06559.pdf,Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following,"Diffusion models excel at modeling complex and multimodal trajectory
+distributions for decision-making and control. Reward-gradient guided denoising
+has been recently proposed to generate trajectories that maximize both a
+differentiable reward function and the likelihood under the data distribution
+captured by a diffusion model. Reward-gradient guided denoising requires a
+differentiable reward function fitted to both clean and noised samples,
+limiting its applicability as a general trajectory optimizer. In this paper, we
+propose DiffusionES, a method that combines gradient-free optimization with
+trajectory denoising to optimize black-box non-differentiable objectives while
+staying in the data manifold. Diffusion-ES samples trajectories during
+evolutionary search from a diffusion model and scores them using a black-box
+reward function. It mutates high-scoring trajectories using a truncated
+diffusion process that applies a small number of noising and denoising steps,
+allowing for much more efficient exploration of the solution space. We show
+that DiffusionES achieves state-of-the-art performance on nuPlan, an
+established closed-loop planning benchmark for autonomous driving. Diffusion-ES
+outperforms existing sampling-based planners, reactive deterministic or
+diffusion-based policies, and reward-gradient guidance. Additionally, we show
+that unlike prior guidance methods, our method can optimize non-differentiable
+language-shaped reward functions generated by few-shot LLM prompting. When
+guided by a human teacher that issues instructions to follow, our method can
+generate novel, highly complex behaviors, such as aggressive lane weaving,
+which are not present in the training data. This allows us to solve the hardest
+nuPlan scenarios which are beyond the capabilities of existing trajectory
+optimization methods and driving policies.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CL', 'cs.RO']"
+Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training,Yipeng Gao · Zeyu Wang · Wei-Shi Zheng · Cihang Xie · Yuyin Zhou,https://github.com/UCSC-VLAA/MixCon3D,https://arxiv.org/abs/2311.01734,,2311.01734.pdf,Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training,"Contrastive learning has emerged as a promising paradigm for 3D open-world
+understanding, i.e., aligning point cloud representation to image and text
+embedding space individually. In this paper, we introduce MixCon3D, a simple
+yet effective method aiming to sculpt holistic 3D representation in contrastive
+language-image-3D pre-training. In contrast to point cloud only, we develop the
+3D object-level representation from complementary perspectives, e.g.,
+multi-view rendered images with the point cloud. Then, MixCon3D performs
+language-3D contrastive learning, comprehensively depicting real-world 3D
+objects and bolstering text alignment. Additionally, we pioneer the first
+thorough investigation of various training recipes for the 3D contrastive
+learning paradigm, building a solid baseline with improved performance.
+Extensive experiments conducted on three representative benchmarks reveal that
+our method significantly improves over the baseline, surpassing the previous
+state-of-the-art performance on the challenging 1,156-category Objaverse-LVIS
+dataset by 5.7%. The versatility of MixCon3D is showcased in applications such
+as text-to-3D retrieval and point cloud captioning, further evidencing its
+efficacy in diverse scenarios. The code is available at
+https://github.com/UCSC-VLAA/MixCon3D.",cs.CV,['cs.CV']
+Cross-spectral Gated-RGB Stereo Depth Estimation,Samuel Brucker · Stefanie Walz · Mario Bijelic · Felix Heide,https://light.princeton.edu/publication/gatedrccbstereo/,https://arxiv.org/abs/2405.12759,,2405.12759.pdf,Cross-spectral Gated-RGB Stereo Depth Estimation,"Gated cameras flood-illuminate a scene and capture the time-gated impulse
+response of a scene. By employing nanosecond-scale gates, existing sensors are
+capable of capturing mega-pixel gated images, delivering dense depth improving
+on today's LiDAR sensors in spatial resolution and depth precision. Although
+gated depth estimation methods deliver a million of depth estimates per frame,
+their resolution is still an order below existing RGB imaging methods. In this
+work, we combine high-resolution stereo HDR RCCB cameras with gated imaging,
+allowing us to exploit depth cues from active gating, multi-view RGB and
+multi-view NIR sensing -- multi-view and gated cues across the entire spectrum.
+The resulting capture system consists only of low-cost CMOS sensors and
+flood-illumination. We propose a novel stereo-depth estimation method that is
+capable of exploiting these multi-modal multi-view depth cues, including the
+active illumination that is measured by the RCCB camera when removing the
+IR-cut filter. The proposed method achieves accurate depth at long ranges,
+outperforming the next best existing method by 39% for ranges of 100 to 220m in
+MAE on accumulated LiDAR ground-truth. Our code, models and datasets are
+available at https://light.princeton.edu/gatedrccbstereo/ .",cs.CV,['cs.CV']
+$\mathcal{Z}^*$: Zero-shot $\underline{S}$tyle $\underline{T}$ransfer via $\underline{A}$ttention $\underline{R}$eweighting,Yingying Deng · Xiangyu He · Fan Tang · Weiming Dong, ,https://arxiv.org/abs/2311.16491,,2311.16491.pdf,$Z^*$: Zero-shot Style Transfer via Attention Rearrangement,"Despite the remarkable progress in image style transfer, formulating style in
+the context of art is inherently subjective and challenging. In contrast to
+existing learning/tuning methods, this study shows that vanilla diffusion
+models can directly extract style information and seamlessly integrate the
+generative prior into the content image without retraining. Specifically, we
+adopt dual denoising paths to represent content/style references in latent
+space and then guide the content image denoising process with style latent
+codes. We further reveal that the cross-attention mechanism in latent diffusion
+models tends to blend the content and style images, resulting in stylized
+outputs that deviate from the original content image. To overcome this
+limitation, we introduce a cross-attention rearrangement strategy. Through
+theoretical analysis and experiments, we demonstrate the effectiveness and
+superiority of the diffusion-based $\underline{Z}$ero-shot $\underline{S}$tyle
+$\underline{T}$ransfer via $\underline{A}$ttention $\underline{R}$earrangement,
+Z-STAR.",cs.CV,['cs.CV']
+CommonCanvas: Open Diffusion Models Trained on Creative-Commons Images,Aaron Gokaslan · A. Feder Cooper · Jasmine Collins · Landan Seguin · Austin Jacobson · Mihir Patel · Jonathan Frankle · Cory Stephenson · Volodymyr Kuleshov, ,https://arxiv.org/abs/2310.16825,,2310.16825.pdf,CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images,"We assemble a dataset of Creative-Commons-licensed (CC) images, which we use
+to train a set of open diffusion models that are qualitatively competitive with
+Stable Diffusion 2 (SD2). This task presents two challenges: (1)
+high-resolution CC images lack the captions necessary to train text-to-image
+generative models; (2) CC images are relatively scarce. In turn, to address
+these challenges, we use an intuitive transfer learning technique to produce a
+set of high-quality synthetic captions paired with curated CC images. We then
+develop a data- and compute-efficient training recipe that requires as little
+as 3% of the LAION-2B data needed to train existing SD2 models, but obtains
+comparable quality. These results indicate that we have a sufficient number of
+CC images (~70 million) for training high-quality models. Our training recipe
+also implements a variety of optimizations that achieve ~3X training speed-ups,
+enabling rapid model iteration. We leverage this recipe to train several
+high-quality text-to-image models, which we dub the CommonCanvas family. Our
+largest model achieves comparable performance to SD2 on a human evaluation,
+despite being trained on our CC dataset that is significantly smaller than
+LAION and using synthetic captions for training. We release our models, data,
+and code at
+https://github.com/mosaicml/diffusion/blob/main/assets/common-canvas.md",cs.CV,"['cs.CV', 'cs.CY']"
+"HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild",Supreeth Narasimhaswamy · Huy Anh Nguyen · Lihan Huang · Minh Hoai, ,https://arxiv.org/abs/2404.13819,,2404.13819.pdf,"HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild","We address the challenging task of identifying, segmenting, and tracking
+hand-held objects, which is crucial for applications such as human action
+segmentation and performance evaluation. This task is particularly challenging
+due to heavy occlusion, rapid motion, and the transitory nature of objects
+being hand-held, where an object may be held, released, and subsequently picked
+up again. To tackle these challenges, we have developed a novel
+transformer-based architecture called HOIST-Former. HOIST-Former is adept at
+spatially and temporally segmenting hands and objects by iteratively pooling
+features from each other, ensuring that the processes of identification,
+segmentation, and tracking of hand-held objects depend on the hands' positions
+and their contextual appearance. We further refine HOIST-Former with a contact
+loss that focuses on areas where hands are in contact with objects. Moreover,
+we also contribute an in-the-wild video dataset called HOIST, which comprises
+4,125 videos complete with bounding boxes, segmentation masks, and tracking IDs
+for hand-held objects. Through experiments on the HOIST dataset and two
+additional public datasets, we demonstrate the efficacy of HOIST-Former in
+segmenting and tracking hand-held objects.",cs.CV,['cs.CV']
+SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples,Phillip Howard · Avinash Madasu · Tiep Le · Gustavo Lujan-Moreno · Anahita Bhiwandiwalla · Vasudev Lal, ,https://arxiv.org/abs/2312.00825,,2312.00825.pdf,SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples,"While vision-language models (VLMs) have achieved remarkable performance
+improvements recently, there is growing evidence that these models also posses
+harmful biases with respect to social attributes such as gender and race. Prior
+studies have primarily focused on probing such bias attributes individually
+while ignoring biases associated with intersections between social attributes.
+This could be due to the difficulty of collecting an exhaustive set of
+image-text pairs for various combinations of social attributes. To address this
+challenge, we employ text-to-image diffusion models to produce counterfactual
+examples for probing intersectional social biases at scale. Our approach
+utilizes Stable Diffusion with cross attention control to produce sets of
+counterfactual image-text pairs that are highly similar in their depiction of a
+subject (e.g., a given occupation) while differing only in their depiction of
+intersectional social attributes (e.g., race & gender). Through our
+over-generate-then-filter methodology, we produce SocialCounterfactuals, a
+high-quality dataset containing 171k image-text pairs for probing
+intersectional biases related to gender, race, and physical characteristics. We
+conduct extensive experiments to demonstrate the usefulness of our generated
+dataset for probing and mitigating intersectional social biases in
+state-of-the-art VLMs.",cs.CV,"['cs.CV', 'cs.AI']"
+Accurate Training Data for Occupancy Map Prediction in Automated Driving using Evidence Theory,Jonas Kälble · Sascha Wirges · Maxim Tatarchenko · Eddy Ilg, ,https://arxiv.org/abs/2405.10575,,2405.10575.pdf,Accurate Training Data for Occupancy Map Prediction in Automated Driving Using Evidence Theory,"Automated driving fundamentally requires knowledge about the surrounding
+geometry of the scene. Modern approaches use only captured images to predict
+occupancy maps that represent the geometry. Training these approaches requires
+accurate data that may be acquired with the help of LiDAR scanners. We show
+that the techniques used for current benchmarks and training datasets to
+convert LiDAR scans into occupancy grid maps yield very low quality, and
+subsequently present a novel approach using evidence theory that yields more
+accurate reconstructions. We demonstrate that these are superior by a large
+margin, both qualitatively and quantitatively, and that we additionally obtain
+meaningful uncertainty estimates. When converting the occupancy maps back to
+depth estimates and comparing them with the raw LiDAR measurements, our method
+yields a MAE improvement of 30% to 52% on nuScenes and 53% on Waymo over other
+occupancy ground-truth data. Finally, we use the improved occupancy maps to
+train a state-of-the-art occupancy prediction method and demonstrate that it
+improves the MAE by 25% on nuScenes.",cs.CV,['cs.CV']
+ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models,Fei Kong · Jinhao Duan · Lichao Sun · Hao Cheng · Renjing Xu · Heng Tao Shen · Xiaofeng Zhu · Xiaoshuang Shi · Kaidi Xu, ,https://arxiv.org/abs/2311.14097,,2311.14097.pdf,ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models,"Though diffusion models excel in image generation, their step-by-step
+denoising leads to slow generation speeds. Consistency training addresses this
+issue with single-step sampling but often produces lower-quality generations
+and requires high training costs. In this paper, we show that optimizing
+consistency training loss minimizes the Wasserstein distance between target and
+generated distributions. As timestep increases, the upper bound accumulates
+previous consistency training losses. Therefore, larger batch sizes are needed
+to reduce both current and accumulated losses. We propose Adversarial
+Consistency Training (ACT), which directly minimizes the Jensen-Shannon (JS)
+divergence between distributions at each timestep using a discriminator.
+Theoretically, ACT enhances generation quality, and convergence. By
+incorporating a discriminator into the consistency training framework, our
+method achieves improved FID scores on CIFAR10 and ImageNet 64$\times$64 and
+LSUN Cat 256$\times$256 datasets, retains zero-shot image inpainting
+capabilities, and uses less than $1/6$ of the original batch size and fewer
+than $1/2$ of the model parameters and training steps compared to the baseline
+method, this leads to a substantial reduction in resource consumption. Our code
+is available:https://github.com/kong13661/ACT",cs.CV,['cs.CV']
+StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image Translation,Sidi Wu · Yizi Chen · Loic Landrieu · Nicolas Gonthier · Samuel Mermet · Lorenz Hurni · Konrad Schindler, ,https://arxiv.org/abs/2403.20142,,2403.20142.pdf,StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image Translation,"Most image-to-image translation models postulate that a unique correspondence
+exists between the semantic classes of the source and target domains. However,
+this assumption does not always hold in real-world scenarios due to divergent
+distributions, different class sets, and asymmetrical information
+representation. As conventional GANs attempt to generate images that match the
+distribution of the target domain, they may hallucinate spurious instances of
+classes absent from the source domain, thereby diminishing the usefulness and
+reliability of translated images. CycleGAN-based methods are also known to hide
+the mismatched information in the generated images to bypass cycle consistency
+objectives, a process known as steganography. In response to the challenge of
+non-bijective image translation, we introduce StegoGAN, a novel model that
+leverages steganography to prevent spurious features in generated images. Our
+approach enhances the semantic consistency of the translated images without
+requiring additional postprocessing or supervision. Our experimental
+evaluations demonstrate that StegoGAN outperforms existing GAN-based models
+across various non-bijective image-to-image translation tasks, both
+qualitatively and quantitatively. Our code and pretrained models are accessible
+at https://github.com/sian-wusidi/StegoGAN.",cs.CV,"['cs.CV', 'eess.IV']"
+LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset,Haolin Liu · Chongjie Ye · Yinyu Nie · Yingfan He · Xiaoguang Han, ,https://arxiv.org/html/2312.12418v1,,2312.12418v1.pdf,LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset,"Instance shape reconstruction from a 3D scene involves recovering the full
+geometries of multiple objects at the semantic instance level. Many methods
+leverage data-driven learning due to the intricacies of scene complexity and
+significant indoor occlusions. Training these methods often requires a
+large-scale, high-quality dataset with aligned and paired shape annotations
+with real-world scans. Existing datasets are either synthetic or misaligned,
+restricting the performance of data-driven methods on real data. To this end,
+we introduce LASA, a Large-scale Aligned Shape Annotation Dataset comprising
+10,412 high-quality CAD annotations aligned with 920 real-world scene scans
+from ArkitScenes, created manually by professional artists. On this top, we
+propose a novel Diffusion-based Cross-Modal Shape Reconstruction (DisCo)
+method. It is empowered by a hybrid feature aggregation design to fuse
+multi-modal inputs and recover high-fidelity object geometries. Besides, we
+present an Occupancy-Guided 3D Object Detection (OccGOD) method and demonstrate
+that our shape annotations provide scene occupancy clues that can further
+improve 3D object detection. Supported by LASA, extensive experiments show that
+our methods achieve state-of-the-art performance in both instance-level scene
+reconstruction and 3D object detection tasks.",cs.CV,['cs.CV']
+Unsupervised Keypoints from Pretrained Diffusion Models,Eric Hedlin · Gopal Sharma · Shweta Mahajan · Xingzhe He · Hossam Isack · Abhishek Kar · Helge Rhodin · Andrea Tagliasacchi · Kwang Moo Yi, ,https://arxiv.org/abs/2312.00065,,2312.00065.pdf,Unsupervised Keypoints from Pretrained Diffusion Models,"Unsupervised learning of keypoints and landmarks has seen significant
+progress with the help of modern neural network architectures, but performance
+is yet to match the supervised counterpart, making their practicability
+questionable. We leverage the emergent knowledge within text-to-image diffusion
+models, towards more robust unsupervised keypoints. Our core idea is to find
+text embeddings that would cause the generative model to consistently attend to
+compact regions in images (i.e. keypoints). To do so, we simply optimize the
+text embedding such that the cross-attention maps within the denoising network
+are localized as Gaussians with small standard deviations. We validate our
+performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD,
+DeepFashion, and Human3.6m datasets. We achieve significantly improved
+accuracy, sometimes even outperforming supervised ones, particularly for data
+that is non-aligned and less curated. Our code is publicly available and can be
+found through our project page: https://ubc-vision.github.io/StableKeypoints/",cs.CV,['cs.CV']
+READ: Retrieval-Enhanced Asymmetric Diffusion for Motion Planning,Takeru Oba · Matthew Walter · Norimichi Ukita,https://obat2343.github.io/READ.github.io/,http://export.arxiv.org/abs/2308.01557,,2308.01557.pdf,Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models,"Learning priors on trajectory distributions can help accelerate robot motion
+planning optimization. Given previously successful plans, learning trajectory
+generative models as priors for a new planning problem is highly desirable.
+Prior works propose several ways on utilizing this prior to bootstrapping the
+motion planning problem. Either sampling the prior for initializations or using
+the prior distribution in a maximum-a-posterior formulation for trajectory
+optimization. In this work, we propose learning diffusion models as priors. We
+then can sample directly from the posterior trajectory distribution conditioned
+on task goals, by leveraging the inverse denoising process of diffusion models.
+Furthermore, diffusion has been recently shown to effectively encode data
+multimodality in high-dimensional settings, which is particularly well-suited
+for large trajectory dataset. To demonstrate our method efficacy, we compare
+our proposed method - Motion Planning Diffusion - against several baselines in
+simulated planar robot and 7-dof robot arm manipulator environments. To assess
+the generalization capabilities of our method, we test it in environments with
+previously unseen obstacles. Our experiments show that diffusion models are
+strong priors to encode high-dimensional trajectory distributions of robot
+motions.",cs.RO,"['cs.RO', 'cs.AI', 'cs.LG']"
+On the Estimation of Image-matching Uncertainty in Visual Place Recognition,Mubariz Zaffar · Liangliang Nan · Julian F. P. Kooij, ,https://arxiv.org/abs/2404.00546,,2404.00546.pdf,On the Estimation of Image-matching Uncertainty in Visual Place Recognition,"In Visual Place Recognition (VPR) the pose of a query image is estimated by
+comparing the image to a map of reference images with known reference poses. As
+is typical for image retrieval problems, a feature extractor maps the query and
+reference images to a feature space, where a nearest neighbor search is then
+performed. However, till recently little attention has been given to
+quantifying the confidence that a retrieved reference image is a correct match.
+Highly certain but incorrect retrieval can lead to catastrophic failure of
+VPR-based localization pipelines. This work compares for the first time the
+main approaches for estimating the image-matching uncertainty, including the
+traditional retrieval-based uncertainty estimation, more recent data-driven
+aleatoric uncertainty estimation, and the compute-intensive geometric
+verification. We further formulate a simple baseline method, ``SUE'', which
+unlike the other methods considers the freely-available poses of the reference
+images in the map. Our experiments reveal that a simple L2-distance between the
+query and reference descriptors is already a better estimate of image-matching
+uncertainty than current data-driven approaches. SUE outperforms the other
+efficient uncertainty estimation methods, and its uncertainty estimates
+complement the computationally expensive geometric verification approach.
+Future works for uncertainty estimation in VPR should consider the baselines
+discussed in this work.",cs.CV,['cs.CV']
+GROUNDHOG: Grounding Large Language Models to Holistic Segmentation,Yichi Zhang · Ziqiao Ma · Xiaofeng Gao · Suhaila Shakiah · Qiaozi Gao · Joyce Chai,https://groundhog-mllm.github.io/,https://arxiv.org/abs/2402.16846,,2402.16846.pdf,GROUNDHOG: Grounding Large Language Models to Holistic Segmentation,"Most multimodal large language models (MLLMs) learn language-to-object
+grounding through causal language modeling where grounded objects are captured
+by bounding boxes as sequences of location tokens. This paradigm lacks
+pixel-level representations that are important for fine-grained visual
+understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM
+developed by grounding Large Language Models to holistic segmentation.
+GROUNDHOG incorporates a masked feature extractor and converts extracted
+features into visual entity tokens for the MLLM backbone, which then connects
+groundable phrases to unified grounding masks by retrieving and merging the
+entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual
+instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by
+harvesting a collection of segmentation-grounded datasets with rich
+annotations. Our experimental results show that GROUNDHOG achieves superior
+performance on various language grounding tasks without task-specific
+fine-tuning, and significantly reduces object hallucination. GROUNDHOG also
+demonstrates better grounding towards complex forms of visual input and
+provides easy-to-understand diagnosis in failure cases.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+"Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model",Shraman Pramanick · Guangxing Han · Rui Hou · Sayan Nag · Ser-Nam Lim · Nicolas Ballas · Qifan Wang · Rama Chellappa · Amjad Almahairi, ,https://arxiv.org/abs/2312.12423,,2312.12423.pdf,"Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model","The ability of large language models (LLMs) to process visual inputs has
+given rise to general-purpose vision systems, unifying various vision-language
+(VL) tasks by instruction tuning. However, due to the enormous diversity in
+input-output formats in the vision domain, existing general-purpose models fail
+to successfully integrate segmentation and multi-image inputs with coarse-level
+tasks into a single framework. In this work, we introduce VistaLLM, a powerful
+visual system that addresses coarse- and fine-grained VL tasks over single and
+multiple input images using a unified framework. VistaLLM utilizes an
+instruction-guided image tokenizer that filters global embeddings using task
+descriptions to extract compressed and refined features from numerous images.
+Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
+represent binary segmentation masks as sequences, significantly improving over
+previously used uniform sampling. To bolster the desired capability of
+VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
+dataset with 6.8M samples. We also address the lack of multi-image grounding
+datasets by introducing a novel task, AttCoSeg (Attribute-level
+Co-Segmentation), which boosts the model's reasoning and grounding capability
+over multiple input images. Extensive experiments on a wide range of V- and VL
+tasks demonstrate the effectiveness of VistaLLM by achieving consistent
+state-of-the-art performance over strong baselines across all downstream tasks.
+Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.",cs.CV,"['cs.CV', 'cs.AI']"
+Spectrum AUC Difference (SAUCD): Human Aligned 3D Shape Evaluation,Tianyu Luan · Zhong Li · Lele Chen · Xuan Gong · Lichang Chen · Yi Xu · Junsong Yuan, ,https://arxiv.org/abs/2403.01619,,2403.01619.pdf,Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluation,"Existing 3D mesh shape evaluation metrics mainly focus on the overall shape
+but are usually less sensitive to local details. This makes them inconsistent
+with human evaluation, as human perception cares about both overall and
+detailed shape. In this paper, we propose an analytic metric named Spectrum
+Area Under the Curve Difference (SAUCD) that demonstrates better consistency
+with human evaluation. To compare the difference between two shapes, we first
+transform the 3D mesh to the spectrum domain using the discrete
+Laplace-Beltrami operator and Fourier transform. Then, we calculate the Area
+Under the Curve (AUC) difference between the two spectrums, so that each
+frequency band that captures either the overall or detailed shape is equitably
+considered. Taking human sensitivity across frequency bands into account, we
+further extend our metric by learning suitable weights for each frequency band
+which better aligns with human perception. To measure the performance of SAUCD,
+we build a 3D mesh evaluation dataset called Shape Grading, along with manual
+annotations from more than 800 subjects. By measuring the correlation between
+our metric and human evaluation, we demonstrate that SAUCD is well aligned with
+human evaluation, and outperforms previous 3D mesh metrics.",cs.CV,"['cs.CV', 'cs.GR']"
+AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation,Haonan Wang · Qixiang ZHANG · Yi Li · Xiaomeng Li, ,https://arxiv.org/abs/2403.01818,,2403.01818.pdf,AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation,"Semi-supervised semantic segmentation (SSSS) has been proposed to alleviate
+the burden of time-consuming pixel-level manual labeling, which leverages
+limited labeled data along with larger amounts of unlabeled data. Current
+state-of-the-art methods train the labeled data with ground truths and
+unlabeled data with pseudo labels. However, the two training flows are
+separate, which allows labeled data to dominate the training process, resulting
+in low-quality pseudo labels and, consequently, sub-optimal results. To
+alleviate this issue, we present AllSpark, which reborns the labeled features
+from unlabeled ones with the channel-wise cross-attention mechanism. We further
+introduce a Semantic Memory along with a Channel Semantic Grouping strategy to
+ensure that unlabeled features adequately represent labeled features. The
+AllSpark shed new light on the architecture level designs of SSSS rather than
+framework level, which avoids increasingly complicated training pipeline
+designs. It can also be regarded as a flexible bottleneck module that can be
+seamlessly integrated into a general transformer-based segmentation model. The
+proposed AllSpark outperforms existing methods across all evaluation protocols
+on Pascal, Cityscapes and COCO benchmarks without bells-and-whistles. Code and
+model weights are available at: https://github.com/xmed-lab/AllSpark.",cs.CV,"['cs.CV', 'cs.AI']"
+Real-Time Simulated Avatar from Head-Mounted Sensors,Zhengyi Luo · Jinkun Cao · Rawal Khirodkar · Alexander Winkler · Jing Huang · Kris Kitani · Weipeng Xu, ,https://arxiv.org/abs/2403.06862,,2403.06862.pdf,Real-Time Simulated Avatar from Head-Mounted Sensors,"We present SimXR, a method for controlling a simulated avatar from
+information (headset pose and cameras) obtained from AR / VR headsets. Due to
+the challenging viewpoint of head-mounted cameras, the human body is often
+clipped out of view, making traditional image-based egocentric pose estimation
+challenging. On the other hand, headset poses provide valuable information
+about overall body motion, but lack fine-grained details about the hands and
+feet. To synergize headset poses with cameras, we control a humanoid to track
+headset movement while analyzing input images to decide body movement. When
+body parts are seen, the movements of hands and feet will be guided by the
+images; when unseen, the laws of physics guide the controller to generate
+plausible motion. We design an end-to-end method that does not rely on any
+intermediate representations and learns to directly map from images and headset
+poses to humanoid control signals. To train our method, we also propose a
+large-scale synthetic dataset created using camera configurations compatible
+with a commercially available VR headset (Quest 2) and show promising results
+on real-world captures. To demonstrate the applicability of our framework, we
+also test it on an AR headset with a forward-facing camera.",cs.CV,"['cs.CV', 'cs.GR', 'cs.RO']"
+Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Driving,Mozhgan Pourkeshavarz · Mohammad Sabokrou · Amir Rasouli, ,https://arxiv.org/abs/2306.15755,,2306.15755.pdf,Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Driving,"In autonomous driving, behavior prediction is fundamental for safe motion
+planning, hence the security and robustness of prediction models against
+adversarial attacks are of paramount importance. We propose a novel adversarial
+backdoor attack against trajectory prediction models as a means of studying
+their potential vulnerabilities. Our attack affects the victim at training time
+via naturalistic, hence stealthy, poisoned samples crafted using a novel
+two-step approach. First, the triggers are crafted by perturbing the trajectory
+of attacking vehicle and then disguised by transforming the scene using a
+bi-level optimization technique. The proposed attack does not depend on a
+particular model architecture and operates in a black-box manner, thus can be
+effective without any knowledge of the victim model. We conduct extensive
+empirical studies using state-of-the-art prediction models on two benchmark
+datasets using metrics customized for trajectory prediction. We show that the
+proposed attack is highly effective, as it can significantly hinder the
+performance of prediction models, unnoticeable by the victims, and efficient as
+it forces the victim to generate malicious behavior even under constrained
+conditions. Via ablative studies, we analyze the impact of different attack
+design choices followed by an evaluation of existing defence mechanisms against
+the proposed attack.",cs.CV,['cs.CV']
+MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labeling,Xuzhe Zhang · Yuhao Wu · Elsa Angelini · Ang Li · Jia Guo · Jerod Rasmussen · Thomas O'Connor · Pathik Wadhwa · Andrea Jackowski · Hai Li · Jonathan Posner · Andrew Laine · YUN WANG · Yun Wang,https://github.com/XuzheZ/MAPSeg,,https://www.researchgate.net/publication/378738417_MAPSeg_Unified_Unsupervised_Domain_Adaptation_for_Heterogeneous_Medical_Image_Segmentation_Based_on_3D_Masked_Autoencoding_and_Pseudo-Labeling,,,,,nan
+KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling,Yu Wang · Xin Li · Shengzhao Wen · gang zhang · Haixiao Yue · Haocheng Feng · Junyu Han · Errui Ding, ,https://arxiv.org/abs/2311.13657,,2311.13657.pdf,Efficient Transformer Knowledge Distillation: A Performance Review,"As pretrained transformer language models continue to achieve
+state-of-the-art performance, the Natural Language Processing community has
+pushed for advances in model compression and efficient attention mechanisms to
+address high computational requirements and limited input sequence length.
+Despite these separate efforts, no investigation has been done into the
+intersection of these two fields. In this work, we provide an evaluation of
+model compression via knowledge distillation on efficient attention
+transformers. We provide cost-performance trade-offs for the compression of
+state-of-the-art efficient attention architectures and the gains made in
+performance in comparison to their full attention counterparts. Furthermore, we
+introduce a new long-context Named Entity Recognition dataset, GONERD, to train
+and test the performance of NER models on long sequences. We find that
+distilled efficient attention transformers can preserve a significant amount of
+original model performance, preserving up to 98.6% across short-context tasks
+(GLUE, SQUAD, CoNLL-2003), up to 94.6% across long-context
+Question-and-Answering tasks (HotpotQA, TriviaQA), and up to 98.8% on
+long-context Named Entity Recognition (GONERD), while decreasing inference
+times by up to 57.8%. We find that, for most models on most tasks, performing
+knowledge distillation is an effective method to yield high-performing
+efficient attention models with low costs.",cs.CL,"['cs.CL', 'cs.LG']"
+Point-VOS: Pointing Up Video Object Segmentation,Sabarinath Mahadevan · Idil Esen Zulfikar · Paul Voigtlaender · Bastian Leibe, ,https://arxiv.org/abs/2402.05917v1,,2402.05917v1.pdf,Point-VOS: Pointing Up Video Object Segmentation,"Current state-of-the-art Video Object Segmentation (VOS) methods rely on
+dense per-object mask annotations both during training and testing. This
+requires time-consuming and costly video annotation mechanisms. We propose a
+novel Point-VOS task with a spatio-temporally sparse point-wise annotation
+scheme that substantially reduces the annotation effort. We apply our
+annotation scheme to two large-scale video datasets with text descriptions and
+annotate over 19M points across 133K objects in 32K videos. Based on our
+annotations, we propose a new Point-VOS benchmark, and a corresponding
+point-based training mechanism, which we use to establish strong baseline
+results. We show that existing VOS methods can easily be adapted to leverage
+our point annotations during training, and can achieve results close to the
+fully-supervised performance when trained on pseudo-masks generated from these
+points. In addition, we show that our data can be used to improve models that
+connect vision and language, by evaluating it on the Video Narrative Grounding
+(VNG) task. We will make our code and annotations available at
+https://pointvos.github.io.",cs.CV,['cs.CV']
+DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing,Yujun Shi · Chuhui Xue · Jun Hao Liew · Jiachun Pan · Hanshu Yan · Wenqing Zhang · Vincent Y. F. Tan · Song Bai, ,https://arxiv.org/abs/2306.14435,,2306.14435.pdf,DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing,"Accurate and controllable image editing is a challenging task that has
+attracted significant attention recently. Notably, DragGAN is an interactive
+point-based image editing framework that achieves impressive editing results
+with pixel-level precision. However, due to its reliance on generative
+adversarial networks (GANs), its generality is limited by the capacity of
+pretrained GAN models. In this work, we extend this editing framework to
+diffusion models and propose a novel approach DragDiffusion. By harnessing
+large-scale pretrained diffusion models, we greatly enhance the applicability
+of interactive point-based editing on both real and diffusion-generated images.
+Our approach involves optimizing the diffusion latents to achieve precise
+spatial control. The supervision signal of this optimization process is from
+the diffusion model's UNet features, which are known to contain rich semantic
+and geometric information. Moreover, we introduce two additional techniques,
+namely LoRA fine-tuning and latent-MasaCtrl, to further preserve the identity
+of the original image. Lastly, we present a challenging benchmark dataset
+called DragBench -- the first benchmark to evaluate the performance of
+interactive point-based image editing methods. Experiments across a wide range
+of challenging cases (e.g., images with multiple objects, diverse object
+categories, various styles, etc.) demonstrate the versatility and generality of
+DragDiffusion. Code: https://github.com/Yujun-Shi/DragDiffusion.",cs.CV,"['cs.CV', 'cs.LG']"
+Revisiting Adversarial Training at Scale,Zeyu Wang · Xianhang li · Hongru Zhu · Cihang Xie, ,https://arxiv.org/abs/2401.04727,,2401.04727.pdf,Revisiting Adversarial Training at Scale,"The machine learning community has witnessed a drastic change in the training
+pipeline, pivoted by those ''foundation models'' with unprecedented scales.
+However, the field of adversarial training is lagging behind, predominantly
+centered around small model sizes like ResNet-50, and tiny and low-resolution
+datasets like CIFAR-10. To bridge this transformation gap, this paper provides
+a modern re-examination with adversarial training, investigating its potential
+benefits when applied at scale. Additionally, we introduce an efficient and
+effective training strategy to enable adversarial training with giant models
+and web-scale data at an affordable computing cost. We denote this newly
+introduced framework as AdvXL.
+  Empirical results demonstrate that AdvXL establishes new state-of-the-art
+robust accuracy records under AutoAttack on ImageNet-1K. For example, by
+training on DataComp-1B dataset, our AdvXL empowers a vanilla ViT-g model to
+substantially surpass the previous records of $l_{\infty}$-, $l_{2}$-, and
+$l_{1}$-robust accuracy by margins of 11.4%, 14.2% and 12.9%, respectively.
+This achievement posits AdvXL as a pioneering approach, charting a new
+trajectory for the efficient training of robust visual representations at
+significantly larger scales. Our code is available at
+https://github.com/UCSC-VLAA/AdvXL.",cs.CV,['cs.CV']
+Seeing Motion at Nighttime with an Event Camera,Haoyue Liu · Shihan Peng · Lin Zhu · Yi Chang · Hanyu Zhou · Luxin Yan,https://github.com/Liu-haoyue/NER-Net,https://arxiv.org/abs/2404.11884,,2404.11884.pdf,Seeing Motion at Nighttime with an Event Camera,"We focus on a very challenging task: imaging at nighttime dynamic scenes.
+Most previous methods rely on the low-light enhancement of a conventional RGB
+camera. However, they would inevitably face a dilemma between the long exposure
+time of nighttime and the motion blur of dynamic scenes. Event cameras react to
+dynamic changes with higher temporal resolution (microsecond) and higher
+dynamic range (120dB), offering an alternative solution. In this work, we
+present a novel nighttime dynamic imaging method with an event camera.
+Specifically, we discover that the event at nighttime exhibits temporal
+trailing characteristics and spatial non-stationary distribution. Consequently,
+we propose a nighttime event reconstruction network (NER-Net) which mainly
+includes a learnable event timestamps calibration module (LETC) to align the
+temporal trailing events and a non-uniform illumination aware module (NIAM) to
+stabilize the spatiotemporal distribution of events. Moreover, we construct a
+paired real low-light event dataset (RLED) through a co-axial imaging system,
+including 64,200 spatially and temporally aligned image GTs and low-light
+events. Extensive experiments demonstrate that the proposed method outperforms
+state-of-the-art methods in terms of visual quality and generalization ability
+on real-world nighttime datasets. The project are available at:
+https://github.com/Liu-haoyue/NER-Net.",cs.CV,['cs.CV']
+Generative Unlearning for Any Identity,Juwon Seo · Sung-Hoon Lee · Tae-Young Lee · SeungJun Moon · Gyeong-Moon Park, ,https://arxiv.org/abs/2405.09879,,2405.09879.pdf,Generative Unlearning for Any Identity,"Recent advances in generative models trained on large-scale datasets have
+made it possible to synthesize high-quality samples across various domains.
+Moreover, the emergence of strong inversion networks enables not only a
+reconstruction of real-world images but also the modification of attributes
+through various editing methods. However, in certain domains related to privacy
+issues, e.g., human faces, advanced generative models along with strong
+inversion methods can lead to potential misuses. In this paper, we propose an
+essential yet under-explored task called generative identity unlearning, which
+steers the model not to generate an image of a specific identity. In the
+generative identity unlearning, we target the following objectives: (i)
+preventing the generation of images with a certain identity, and (ii)
+preserving the overall quality of the generative model. To satisfy these goals,
+we propose a novel framework, Generative Unlearning for Any Identity (GUIDE),
+which prevents the reconstruction of a specific identity by unlearning the
+generator with only a single image. GUIDE consists of two parts: (i) finding a
+target point for optimization that un-identifies the source latent code and
+(ii) novel loss functions that facilitate the unlearning procedure while less
+affecting the learned distribution. Our extensive experiments demonstrate that
+our proposed method achieves state-of-the-art performance in the generative
+machine unlearning task. The code is available at
+https://github.com/KHU-AGI/GUIDE.",cs.CV,"['cs.CV', 'cs.AI']"
+OmniMedVQA: A New Large-Scale Comprehensive  Evaluation Benchmark for Medical LVLM,Yutao Hu · Yutao Hu · Tianbin · Quanfeng Lu · Wenqi Shao · Junjun He · Yu Qiao · Ping Luo, ,https://arxiv.org/abs/2402.09181,,2402.09181.pdf,OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM,"Large Vision-Language Models (LVLMs) have demonstrated remarkable
+capabilities in various multimodal tasks. However, their potential in the
+medical domain remains largely unexplored. A significant challenge arises from
+the scarcity of diverse medical images spanning various modalities and
+anatomical regions, which is essential in real-world medical applications. To
+solve this problem, in this paper, we introduce OmniMedVQA, a novel
+comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark
+is collected from 73 different medical datasets, including 12 different
+modalities and covering more than 20 distinct anatomical regions. Importantly,
+all images in this benchmark are sourced from authentic medical scenarios,
+ensuring alignment with the requirements of the medical field and suitability
+for evaluating LVLMs. Through our extensive experiments, we have found that
+existing LVLMs struggle to address these medical VQA problems effectively.
+Moreover, what surprises us is that medical-specialized LVLMs even exhibit
+inferior performance to those general-domain models, calling for a more
+versatile and robust LVLM in the biomedical field. The evaluation results not
+only reveal the current limitations of LVLM in understanding real medical
+images but also highlight our dataset's significance. Our code with dataset are
+available at https://github.com/OpenGVLab/Multi-Modality-Arena.",eess.IV,"['eess.IV', 'cs.CV']"
+Sequential Modeling Enables Scalable Learning for Large Vision Models,Yutong Bai · Xinyang Geng · Xinyang Geng · Karttikeya Mangalam · Amir Bar · Alan L. Yuille · Trevor Darrell · Jitendra Malik · Alexei A. Efros, ,https://arxiv.org/abs/2312.00785,,2312.00785.pdf,Sequential Modeling Enables Scalable Learning for Large Vision Models,"We introduce a novel sequential modeling approach which enables learning a
+Large Vision Model (LVM) without making use of any linguistic data. To do this,
+we define a common format, ""visual sentences"", in which we can represent raw
+images and videos as well as annotated data sources such as semantic
+segmentations and depth reconstructions without needing any meta-knowledge
+beyond the pixels. Once this wide variety of visual data (comprising 420
+billion tokens) is represented as sequences, the model can be trained to
+minimize a cross-entropy loss for next token prediction. By training across
+various scales of model architecture and data diversity, we provide empirical
+evidence that our models scale effectively. Many different vision tasks can be
+solved by designing suitable visual prompts at test time.",cs.CV,['cs.CV']
+An edit friendly ddpm noise space: inversion and manipulations,Inbar Huberman-Spiegelglas · Vladimir Kulikov · Tomer Michaeli, ,https://ar5iv.labs.arxiv.org/html/2307.00522,,2307.00522.pdf,LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance,"Recent large-scale text-guided diffusion models provide powerful
+image-generation capabilities. Currently, a significant effort is given to
+enable the modification of these images using text only as means to offer
+intuitive and versatile editing. However, editing proves to be difficult for
+these generative models due to the inherent nature of editing techniques, which
+involves preserving certain content from the original image. Conversely, in
+text-based models, even minor modifications to the text prompt frequently
+result in an entirely distinct result, making attaining one-shot generation
+that accurately corresponds to the users intent exceedingly challenging. In
+addition, to edit a real image using these state-of-the-art tools, one must
+first invert the image into the pre-trained models domain - adding another
+factor affecting the edit quality, as well as latency. In this exploratory
+report, we propose LEDITS - a combined lightweight approach for real-image
+editing, incorporating the Edit Friendly DDPM inversion technique with Semantic
+Guidance, thus extending Semantic Guidance to real image editing, while
+harnessing the editing capabilities of DDPM inversion as well. This approach
+achieves versatile edits, both subtle and extensive as well as alterations in
+composition and style, while requiring no optimization nor extensions to the
+architecture.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes,Alexandros Delitzas · Ayça Takmaz · Federico Tombari · Robert Sumner · Marc Pollefeys · Francis Engelmann,https://scenefun3d.github.io,https://arxiv.org/html/2404.03650v1,,2404.03650v1.pdf,OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views,"Large visual-language models (VLMs), like CLIP, enable open-set image
+segmentation to segment arbitrary concepts from an image in a zero-shot manner.
+This goes beyond the traditional closed-set assumption, i.e., where models can
+only segment classes from a pre-defined training set. More recently, first
+works on open-set segmentation in 3D scenes have appeared in the literature.
+These methods are heavily influenced by closed-set 3D convolutional approaches
+that process point clouds or polygon meshes. However, these 3D scene
+representations do not align well with the image-based nature of the
+visual-language models. Indeed, point cloud and 3D meshes typically have a
+lower resolution than images and the reconstructed 3D scene geometry might not
+project well to the underlying 2D image sequences used to compute pixel-aligned
+CLIP features. To address these challenges, we propose OpenNeRF which naturally
+operates on posed images and directly encodes the VLM features within the NeRF.
+This is similar in spirit to LERF, however our work shows that using pixel-wise
+VLM features (instead of global CLIP features) results in an overall less
+complex architecture without the need for additional DINO regularization. Our
+OpenNeRF further leverages NeRF's ability to render novel views and extract
+open-set VLM features from areas that are not well observed in the initial
+posed images. For 3D point cloud segmentation on the Replica dataset, OpenNeRF
+outperforms recent open-vocabulary methods such as LERF and OpenScene by at
+least +4.9 mIoU.",cs.CV,['cs.CV']
+Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields,Shijie Zhou · Haoran Chang · Sicheng Jiang · Zhiwen Fan · Zehao Zhu · Dejia Xu · Dejia Xu · Pradyumna Chari · Suya You · Zhangyang Wang · Achuta Kadambi, ,https://arxiv.org/abs/2312.03203,,2312.03203.pdf,Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields,"3D scene representations have gained immense popularity in recent years.
+Methods that use Neural Radiance fields are versatile for traditional tasks
+such as novel view synthesis. In recent times, some work has emerged that aims
+to extend the functionality of NeRF beyond view synthesis, for semantically
+aware tasks such as editing and segmentation using 3D feature field
+distillation from 2D foundation models. However, these methods have two major
+limitations: (a) they are limited by the rendering speed of NeRF pipelines, and
+(b) implicitly represented feature fields suffer from continuity artifacts
+reducing feature quality. Recently, 3D Gaussian Splatting has shown
+state-of-the-art performance on real-time radiance field rendering. In this
+work, we go one step further: in addition to radiance field rendering, we
+enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D
+foundation model distillation. This translation is not straightforward: naively
+incorporating feature fields in the 3DGS framework encounters significant
+challenges, notably the disparities in spatial resolution and channel
+consistency between RGB images and feature maps. We propose architectural and
+training changes to efficiently avert this problem. Our proposed method is
+general, and our experiments showcase novel view semantic segmentation,
+language-guided editing and segment anything through learning feature fields
+from state-of-the-art 2D foundation models such as SAM and CLIP-LSeg. Across
+experiments, our distillation method is able to provide comparable or better
+results, while being significantly faster to both train and render.
+Additionally, to the best of our knowledge, we are the first method to enable
+point and bounding-box prompting for radiance field manipulation, by leveraging
+the SAM model. Project website at: https://feature-3dgs.github.io/",cs.CV,['cs.CV']
+Taming Mode Collapse in Score Distillation for Text-to-3D Generation,Peihao Wang · Dejia Xu · Dejia Xu · Zhiwen Fan · Dilin Wang · Sreyas Mohan · Forrest Iandola · Rakesh Ranjan · Yilei Li · Qiang Liu · Zhangyang Wang · Vikas Chandra, ,https://arxiv.org/abs/2401.00909,,2401.00909.pdf,Taming Mode Collapse in Score Distillation for Text-to-3D Generation,"Despite the remarkable performance of score distillation in text-to-3D
+generation, such techniques notoriously suffer from view inconsistency issues,
+also known as ""Janus"" artifact, where the generated objects fake each view with
+multiple front faces. Although empirically effective methods have approached
+this problem via score debiasing or prompt engineering, a more rigorous
+perspective to explain and tackle this problem remains elusive. In this paper,
+we reveal that the existing score distillation-based text-to-3D generation
+frameworks degenerate to maximal likelihood seeking on each view independently
+and thus suffer from the mode collapse problem, manifesting as the Janus
+artifact in practice. To tame mode collapse, we improve score distillation by
+re-establishing the entropy term in the corresponding variational objective,
+which is applied to the distribution of rendered images. Maximizing the entropy
+encourages diversity among different views in generated 3D assets, thereby
+mitigating the Janus problem. Based on this new objective, we derive a new
+update rule for 3D score distillation, dubbed Entropic Score Distillation
+(ESD). We theoretically reveal that ESD can be simplified and implemented by
+just adopting the classifier-free guidance trick upon variational score
+distillation. Although embarrassingly straightforward, our extensive
+experiments successfully demonstrate that ESD can be an effective treatment for
+Janus artifacts in score distillation.",cs.CV,"['cs.CV', 'cs.LG']"
+LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction,Linqing Zhao · Xiuwei Xu · Ziwei Wang · Yunpeng Zhang · Borui Zhang · Wenzhao Zheng · Dalong Du · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2405.17429,,2405.17429.pdf,GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction,"3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and
+semantics of the surrounding scene and is an important task for the robustness
+of vision-centric autonomous driving. Most existing methods employ dense grids
+such as voxels as scene representations, which ignore the sparsity of occupancy
+and the diversity of object scales and thus lead to unbalanced allocation of
+resources. To address this, we propose an object-centric representation to
+describe 3D scenes with sparse 3D semantic Gaussians where each Gaussian
+represents a flexible region of interest and its semantic features. We
+aggregate information from images through the attention mechanism and
+iteratively refine the properties of 3D Gaussians including position,
+covariance, and semantics. We then propose an efficient Gaussian-to-voxel
+splatting method to generate 3D occupancy predictions, which only aggregates
+the neighboring Gaussians for a certain position. We conduct extensive
+experiments on the widely adopted nuScenes and KITTI-360 datasets. Experimental
+results demonstrate that GaussianFormer achieves comparable performance with
+state-of-the-art methods with only 17.8% - 24.8% of their memory consumption.
+Code is available at: https://github.com/huang-yh/GaussianFormer.",cs.CV,"['cs.CV', 'cs.AI']"
+mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration,Qinghao Ye · Haiyang Xu · Jiabo Ye · Ming Yan · Anwen Hu · Haowei Liu · Qi Qian · Ji Zhang · Fei Huang · Fei Huang, ,https://arxiv.org/abs/2311.04257,,2311.04257.pdf,mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration,"Multi-modal Large Language Models (MLLMs) have demonstrated impressive
+instruction abilities across various open-ended tasks. However, previous
+methods primarily focus on enhancing multi-modal capabilities. In this work, we
+introduce a versatile multi-modal large language model, mPLUG-Owl2, which
+effectively leverages modality collaboration to improve performance in both
+text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
+with the language decoder acting as a universal interface for managing
+different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
+modules to facilitate modality collaboration and introduces a modality-adaptive
+module that preserves modality-specific features. Extensive experiments reveal
+that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
+tasks and achieving state-of-the-art performances with a single generic model.
+Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
+collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
+pioneering path in the development of future multi-modal foundation models.",cs.CL,"['cs.CL', 'cs.CV']"
+NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models,Yusuf Dalva · Pinar Yanardag, ,https://arxiv.org/abs/2312.05390,,2312.05390.pdf,NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models,"Generative models have been very popular in the recent years for their image
+generation capabilities. GAN-based models are highly regarded for their
+disentangled latent space, which is a key feature contributing to their success
+in controlled image editing. On the other hand, diffusion models have emerged
+as powerful tools for generating high-quality images. However, the latent space
+of diffusion models is not as thoroughly explored or understood. Existing
+methods that aim to explore the latent space of diffusion models usually relies
+on text prompts to pinpoint specific semantics. However, this approach may be
+restrictive in areas such as art, fashion, or specialized fields like medicine,
+where suitable text prompts might not be available or easy to conceive thus
+limiting the scope of existing work. In this paper, we propose an unsupervised
+method to discover latent semantics in text-to-image diffusion models without
+relying on text prompts. Our method takes a small set of unlabeled images from
+specific domains, such as faces or cats, and a pre-trained diffusion model, and
+discovers diverse semantics in unsupervised fashion using a contrastive
+learning objective. Moreover, the learned directions can be applied
+simultaneously, either within the same domain (such as various types of facial
+edits) or across different domains (such as applying cat and face edits within
+the same image) without interfering with each other. Our extensive experiments
+show that our method achieves highly disentangled edits, outperforming existing
+approaches in both diffusion-based and GAN-based latent space editing methods.",cs.CV,['cs.CV']
+On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving,Kaituo Feng · Changsheng Li · Dongchun Ren · Ye Yuan · Guoren Wang, ,https://arxiv.org/abs/2403.01238,,2403.01238.pdf,On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving,"End-to-end motion planning models equipped with deep neural networks have
+shown great potential for enabling full autonomous driving. However, the
+oversized neural networks render them impractical for deployment on
+resource-constrained systems, which unavoidably requires more computational
+time and resources during reference.To handle this, knowledge distillation
+offers a promising approach that compresses models by enabling a smaller
+student model to learn from a larger teacher model. Nevertheless, how to apply
+knowledge distillation to compress motion planners has not been explored so
+far. In this paper, we propose PlanKD, the first knowledge distillation
+framework tailored for compressing end-to-end motion planners. First,
+considering that driving scenes are inherently complex, often containing
+planning-irrelevant or even noisy information, transferring such information is
+not beneficial for the student planner. Thus, we design an information
+bottleneck based strategy to only distill planning-relevant information, rather
+than transfer all information indiscriminately. Second, different waypoints in
+an output planned trajectory may hold varying degrees of importance for motion
+planning, where a slight deviation in certain crucial waypoints might lead to a
+collision. Therefore, we devise a safety-aware waypoint-attentive distillation
+module that assigns adaptive weights to different waypoints based on the
+importance, to encourage the student to accurately mimic more crucial
+waypoints, thereby improving overall safety. Experiments demonstrate that our
+PlanKD can boost the performance of smaller planners by a large margin, and
+significantly reduce their reference time.",cs.CV,['cs.CV']
+Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning,Zichen Miao · Jiang Wang · Ze Wang · Zhengyuan Yang · Lijuan Wang · Qiang Qiu · Zicheng Liu, ,,https://bair.berkeley.edu/blog/2023/07/14/ddpo/,,,,,nan
+HDQMF: Holographic Feature Decomposition Using Quantum Algorithms,Prathyush Poduval · Zhuowen Zou · Mohsen Imani, ,https://arxiv.org/abs/2403.17444,,,Quantum accelerated cross regression algorithm for multiview feature extraction,"Multi-view Feature Extraction (MvFE) has wide applications in machine
+learning, image processing and other fields. When dealing with massive
+high-dimensional data, the performance of classical computer faces severe
+challenges due to MvFE involves expensive matrix calculation. To address this
+challenge, a quantum-accelerated cross-regression algorithm for MvFE is
+proposed. The main contributions are as follows:(1) a quantum version algorithm
+for MvFE is proposed for the first time, filling the gap of quantum computing
+in the field of MvFE;(2) a quantum algorithm is designed to construct the
+block-encoding of the target data matrix, so that the optimal Hamiltonian
+simulation technology based on the block-encoding framework can be used to
+efficiently realize the quantum simulation of the target data matrix. This
+approach reduces the dependence of the algorithm's on simulation errors to
+enhance algorithm performance;(3) compared with the classical counterpart
+algorithm, the proposed quantum algorithm has a polynomial acceleration in the
+number of data points, the dimension of data points and the number of view
+data.",quant-ph,['quant-ph']
+Leveraging Predicate and Triplet Learning for Scene Graph Generation,Jiankai Li · Yunhong Wang · Xiefan Guo · Ruijie Yang · Weixin Li, ,https://arxiv.org/abs/2309.03542,,2309.03542.pdf,Zero-Shot Scene Graph Generation via Triplet Calibration and Reduction,"Scene Graph Generation (SGG) plays a pivotal role in downstream
+vision-language tasks. Existing SGG methods typically suffer from poor
+compositional generalizations on unseen triplets. They are generally trained on
+incompletely annotated scene graphs that contain dominant triplets and tend to
+bias toward these seen triplets during inference. To address this issue, we
+propose a Triplet Calibration and Reduction (T-CAR) framework in this paper. In
+our framework, a triplet calibration loss is first presented to regularize the
+representations of diverse triplets and to simultaneously excavate the unseen
+triplets in incompletely annotated training scene graphs. Moreover, the unseen
+space of scene graphs is usually several times larger than the seen space since
+it contains a huge number of unrealistic compositions. Thus, we propose an
+unseen space reduction loss to shift the attention of excavation to reasonable
+unseen compositions to facilitate the model training. Finally, we propose a
+contextual encoder to improve the compositional generalizations of unseen
+triplets by explicitly modeling the relative spatial relations between subjects
+and objects. Extensive experiments show that our approach achieves consistent
+improvements for zero-shot SGG over state-of-the-art methods. The code is
+available at https://github.com/jkli1998/T-CAR.",cs.CV,"['cs.CV', 'cs.MM']"
+Open-vocabulary object 6D pose estimation,Jaime Corsetti · Davide Boscaini · Changjae Oh · Andrea Cavallaro · Fabio Poiesi, ,https://arxiv.org/abs/2312.00690v2,,2312.00690v2.pdf,Open-vocabulary object 6D pose estimation,"We introduce the new setting of open-vocabulary object 6D pose estimation, in
+which a textual prompt is used to specify the object of interest. In contrast
+to existing approaches, in our setting (i) the object of interest is specified
+solely through the textual prompt, (ii) no object model (e.g. CAD or video
+sequence) is required at inference, (iii) the object is imaged from two
+different viewpoints of two different scenes, and (iv) the object was not
+observed during the training phase. To operate in this setting, we introduce a
+novel approach that leverages a Vision-Language Model to segment the object of
+interest from two distinct scenes and to estimate its relative 6D pose. The key
+of our approach is a carefully devised strategy to fuse object-level
+information provided by the prompt with local image features, resulting in a
+feature space that can generalize to novel concepts. We validate our approach
+on a new benchmark based on two popular datasets, REAL275 and Toyota-Light,
+which collectively encompass 39 object instances appearing in four thousand
+image pairs. The results demonstrate that our approach outperforms both a
+well-established hand-crafted method and a recent deep learning-based baseline
+in estimating the relative 6D pose of objects in different scenes. Project
+page: https://jcorsetti.github.io/oryon/.",cs.CV,['cs.CV']
+Matching Anything by Segmenting Anything,Siyuan Li · Lei Ke · Martin Danelljan · Luigi Piccinelli · Mattia Segu · Luc Van Gool · Fisher Yu, ,https://arxiv.org/abs/2401.16741v1,,,MESA: Matching Everything by Segmenting Anything,"Feature matching is a crucial task in the field of computer vision, which
+involves finding correspondences between images. Previous studies achieve
+remarkable performance using learning-based feature comparison. However, the
+pervasive presence of matching redundancy between images gives rise to
+unnecessary and error-prone computations in these methods, imposing limitations
+on their accuracy. To address this issue, we propose MESA, a novel approach to
+establish precise area (or region) matches for efficient matching redundancy
+reduction. MESA first leverages the advanced image understanding capability of
+SAM, a state-of-the-art foundation model for image segmentation, to obtain
+image areas with implicit semantic. Then, a multi-relational graph is proposed
+to model the spatial structure of these areas and construct their scale
+hierarchy. Based on graphical models derived from the graph, the area matching
+is reformulated as an energy minimization task and effectively resolved.
+Extensive experiments demonstrate that MESA yields substantial precision
+improvement for multiple point matchers in indoor and outdoor downstream tasks,
+e.g. +13.61% for DKM in indoor pose estimation.",cs.CV,['cs.CV']
+DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation,Xiaoliang Ju · Zhaoyang Huang · Yijin Li · Guofeng Zhang · Yu Qiao · Hongsheng Li, ,https://ar5iv.labs.arxiv.org/html/2311.17261,,,SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors,"We propose SceneTex, a novel method for effectively generating high-quality
+and style-consistent textures for indoor scenes using depth-to-image diffusion
+priors. Unlike previous methods that either iteratively warp 2D views onto a
+mesh surface or distillate diffusion latent features without accurate geometric
+and style cues, SceneTex formulates the texture synthesis task as an
+optimization problem in the RGB space where style and geometry consistency are
+properly reflected. At its core, SceneTex proposes a multiresolution texture
+field to implicitly encode the mesh appearance. We optimize the target texture
+via a score-distillation-based objective function in respective RGB renderings.
+To further secure the style consistency across views, we introduce a
+cross-attention decoder to predict the RGB values by cross-attending to the
+pre-sampled reference locations in each instance. SceneTex enables various and
+accurate texture synthesis for 3D-FRONT scenes, demonstrating significant
+improvements in visual quality and prompt fidelity over the prior texture
+generation methods.",cs.CV,['cs.CV']
+DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection,Yuhao Sun · Lingyun Yu · Hongtao Xie · Jiaming Li · Yongdong Zhang, ,http://export.arxiv.org/abs/2405.09882,,2405.09882.pdf,DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection,"With the rapid development of face recognition (FR) systems, the privacy of
+face images on social media is facing severe challenges due to the abuse of
+unauthorized FR systems. Some studies utilize adversarial attack techniques to
+defend against malicious FR systems by generating adversarial examples.
+However, the generated adversarial examples, i.e., the protected face images,
+tend to suffer from subpar visual quality and low transferability. In this
+paper, we propose a novel face protection approach, dubbed DiffAM, which
+leverages the powerful generative ability of diffusion models to generate
+high-quality protected face images with adversarial makeup transferred from
+reference images. To be specific, we first introduce a makeup removal module to
+generate non-makeup images utilizing a fine-tuned diffusion model with guidance
+of textual prompts in CLIP space. As the inverse process of makeup transfer,
+makeup removal can make it easier to establish the deterministic relationship
+between makeup domain and non-makeup domain regardless of elaborate text
+prompts. Then, with this relationship, a CLIP-based makeup loss along with an
+ensemble attack strategy is introduced to jointly guide the direction of
+adversarial makeup domain, achieving the generation of protected face images
+with natural-looking makeup and high black-box transferability. Extensive
+experiments demonstrate that DiffAM achieves higher visual quality and attack
+success rates with a gain of 12.98% under black-box setting compared with the
+state of the arts. The code will be available at
+https://github.com/HansSunY/DiffAM.",cs.CV,"['cs.CV', 'cs.AI']"
+MoMask: Generative Masked Modeling of 3D Human Motions,chuan guo · Yuxuan Mu · Muhammad Gohar Javed · Sen Wang · Li Cheng, ,https://arxiv.org/abs/2312.00063,,2312.00063.pdf,MoMask: Generative Masked Modeling of 3D Human Motions,"We introduce MoMask, a novel masked modeling framework for text-driven 3D
+human motion generation. In MoMask, a hierarchical quantization scheme is
+employed to represent human motion as multi-layer discrete motion tokens with
+high-fidelity details. Starting at the base layer, with a sequence of motion
+tokens obtained by vector quantization, the residual tokens of increasing
+orders are derived and stored at the subsequent layers of the hierarchy. This
+is consequently followed by two distinct bidirectional transformers. For the
+base-layer motion tokens, a Masked Transformer is designated to predict
+randomly masked motion tokens conditioned on text input at training stage.
+During generation (i.e. inference) stage, starting from an empty sequence, our
+Masked Transformer iteratively fills up the missing tokens; Subsequently, a
+Residual Transformer learns to progressively predict the next-layer tokens
+based on the results from current layer. Extensive experiments demonstrate that
+MoMask outperforms the state-of-art methods on the text-to-motion generation
+task, with an FID of 0.045 (vs e.g. 0.141 of T2M-GPT) on the HumanML3D dataset,
+and 0.228 (vs 0.514) on KIT-ML, respectively. MoMask can also be seamlessly
+applied in related tasks without further model fine-tuning, such as text-guided
+temporal inpainting.",cs.CV,['cs.CV']
+Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement,Zaid Khan · Vijay Kumar BG · Samuel Schulter · Yun Fu · Manmohan Chandraker, ,https://arxiv.org/abs/2404.04627,,2404.04627.pdf,Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement,"Visual program synthesis is a promising approach to exploit the reasoning
+abilities of large language models for compositional computer vision tasks.
+Previous work has used few-shot prompting with frozen LLMs to synthesize visual
+programs. Training an LLM to write better visual programs is an attractive
+prospect, but it is unclear how to accomplish this. No dataset of visual
+programs for training exists, and acquisition of a visual program dataset
+cannot be easily crowdsourced due to the need for expert annotators. To get
+around the lack of direct supervision, we explore improving the program
+synthesis abilities of an LLM using feedback from interactive experience. We
+propose a method where we exploit existing annotations for a vision-language
+task to improvise a coarse reward signal for that task, treat the LLM as a
+policy, and apply reinforced self-training to improve the visual program
+synthesis ability of the LLM for that task. We describe a series of experiments
+on object detection, compositional visual question answering, and image-text
+retrieval, and show that in each case, the self-trained LLM outperforms or
+performs on par with few-shot frozen LLMs that are an order of magnitude
+larger. Website: https://zaidkhan.me/ViReP",cs.CV,['cs.CV']
+Scaling Laws of Synthetic Images for Model Training ... for Now,Lijie Fan · Kaifeng Chen · Dilip Krishnan · Dina Katabi · Phillip Isola · Yonglong Tian,https://github.com/google-research/syn-rep-learn/tree/main/Scaling,https://arxiv.org/abs/2312.04567,,2312.04567.pdf,Scaling Laws of Synthetic Images for Model Training ... for Now,"Recent significant advances in text-to-image models unlock the possibility of
+training vision systems using synthetic images, potentially overcoming the
+difficulty of collecting curated data at scale. It is unclear, however, how
+these models behave at scale, as more synthetic data is added to the training
+set. In this paper we study the scaling laws of synthetic images generated by
+state of the art text-to-image models, for the training of supervised models:
+image classifiers with label supervision, and CLIP with language supervision.
+We identify several factors, including text prompts, classifier-free guidance
+scale, and types of text-to-image models, that significantly affect scaling
+behavior. After tuning these factors, we observe that synthetic images
+demonstrate a scaling trend similar to, but slightly less effective than, real
+images in CLIP training, while they significantly underperform in scaling when
+training supervised image classifiers. Our analysis indicates that the main
+reason for this underperformance is the inability of off-the-shelf
+text-to-image models to generate certain concepts, a limitation that
+significantly impairs the training of image classifiers. Our findings also
+suggest that scaling synthetic data can be particularly effective in scenarios
+such as: (1) when there is a limited supply of real images for a supervised
+problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
+evaluation dataset diverges significantly from the training data, indicating
+the out-of-distribution scenario, or (3) when synthetic data is used in
+conjunction with real images, as demonstrated in the training of CLIP models.",cs.CV,['cs.CV']
+Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning,Fan Qi · Shuai Li, ,,https://ieeexplore.ieee.org/document/10528890,,,,,nan
+DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling,Xiaoyun Zheng · Liwei Liao · Xufeng Li · Jianbo Jiao · Rongjie Wang · Feng Gao · Shiqi Wang · Ronggang Wang,https://pku-dymvhumans.github.io/,https://arxiv.org/abs/2403.16080,,2403.16080.pdf,PKU-DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling,"High-quality human reconstruction and photo-realistic rendering of a dynamic
+scene is a long-standing problem in computer vision and graphics. Despite
+considerable efforts invested in developing various capture systems and
+reconstruction algorithms, recent advancements still struggle with loose or
+oversized clothing and overly complex poses. In part, this is due to the
+challenges of acquiring high-quality human datasets. To facilitate the
+development of these fields, in this paper, we present PKU-DyMVHumans, a
+versatile human-centric dataset for high-fidelity reconstruction and rendering
+of dynamic human scenarios from dense multi-view videos. It comprises 8.2
+million frames captured by more than 56 synchronized cameras across diverse
+scenarios. These sequences comprise 32 human subjects across 45 different
+scenarios, each with a high-detailed appearance and realistic human motion.
+Inspired by recent advancements in neural radiance field (NeRF)-based scene
+representations, we carefully set up an off-the-shelf framework that is easy to
+provide those state-of-the-art NeRF-based implementations and benchmark on
+PKU-DyMVHumans dataset. It is paving the way for various applications like
+fine-grained foreground/background decomposition, high-quality human
+reconstruction and photo-realistic novel view synthesis of a dynamic scene.
+Extensive studies are performed on the benchmark, demonstrating new
+observations and challenges that emerge from using such high-fidelity dynamic
+data.",cs.CV,['cs.CV']
+CrossMAE: Cross Modality Masked Autoencoders For Region-Aware Audio-Visual Pre-Training,Yuxin Guo · Siyang Sun · Shuailei Ma · Kecheng Zheng · Xiaoyi Bao · Shijie Ma · Wei Zou · Yun Zheng, ,https://arxiv.org/abs/2401.14391,,2401.14391.pdf,Rethinking Patch Dependence for Masked Autoencoders,"In this work, we re-examine inter-patch dependencies in the decoding
+mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
+for masked patch reconstruction in MAE into self-attention and cross-attention.
+Our investigations suggest that self-attention between mask patches is not
+essential for learning good representations. To this end, we propose a novel
+pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
+CrossMAE's decoder leverages only cross-attention between masked and visible
+tokens, with no degradation in downstream performance. This design also enables
+decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
+each decoder block can now leverage different encoder features, resulting in
+improved representation learning. CrossMAE matches MAE in performance with 2.5
+to 3.7$\times$ less decoding compute. It also surpasses MAE on ImageNet
+classification and COCO instance segmentation under the same compute. Code and
+models: https://crossmae.github.io",cs.CV,['cs.CV']
+Traceable Federated Continual Learning,Qiang Wang · Bingyan Liu · Yawen Li, ,https://arxiv.org/abs/2312.13500,,2312.13500.pdf,Federated Continual Novel Class Learning,"In a privacy-focused era, Federated Learning (FL) has emerged as a promising
+machine learning technique. However, most existing FL studies assume that the
+data distribution remains nearly fixed over time, while real-world scenarios
+often involve dynamic and continual changes. To equip FL systems with continual
+model evolution capabilities, we focus on an important problem called Federated
+Continual Novel Class Learning (FedCN) in this work. The biggest challenge in
+FedCN is to merge and align novel classes that are discovered and learned by
+different clients without compromising privacy. To address this, we propose a
+Global Alignment Learning (GAL) framework that can accurately estimate the
+global novel class number and provide effective guidance for local training
+from a global perspective, all while maintaining privacy protection.
+Specifically, GAL first locates high-density regions in the representation
+space through a bi-level clustering mechanism to estimate the novel class
+number, with which the global prototypes corresponding to novel classes can be
+constructed. Then, GAL uses a novel semantic weighted loss to capture all
+possible correlations between these prototypes and the training data for
+mitigating the impact of pseudo-label noise and data heterogeneity. Extensive
+experiments on various datasets demonstrate GAL's superior performance over
+state-of-the-art novel class discovery methods. In particular, GAL achieves
+significant improvements in novel-class performance, increasing the accuracy by
+5.1% to 10.6% in the case of one novel class learning stage and by 7.8% to
+17.9% in the case of two novel class learning stages, without sacrificing
+known-class performance. Moreover, GAL is shown to be effective in equipping a
+variety of different mainstream FL algorithms with novel class discovery and
+learning capability, highlighting its potential for many real-world
+applications.",cs.CV,['cs.CV']
+PolarMatte: Fully Computational Ground-Truth-Quality Alpha Matte Extraction for Images and Video using Polarized Screen Matting,Kenji Enomoto · TJ Rhodes · Brian Price · Gavin Miller, ,https://arxiv.org/abs/2311.13535,,2311.13535.pdf,DiffusionMat: Alpha Matting as Sequential Refinement Learning,"In this paper, we introduce DiffusionMat, a novel image matting framework
+that employs a diffusion model for the transition from coarse to refined alpha
+mattes. Diverging from conventional methods that utilize trimaps merely as
+loose guidance for alpha matte prediction, our approach treats image matting as
+a sequential refinement learning process. This process begins with the addition
+of noise to trimaps and iteratively denoises them using a pre-trained diffusion
+model, which incrementally guides the prediction towards a clean alpha matte.
+The key innovation of our framework is a correction module that adjusts the
+output at each denoising step, ensuring that the final result is consistent
+with the input image's structures. We also introduce the Alpha Reliability
+Propagation, a novel technique designed to maximize the utility of available
+guidance by selectively enhancing the trimap regions with confident alpha
+information, thus simplifying the correction task. To train the correction
+module, we devise specialized loss functions that target the accuracy of the
+alpha matte's edges and the consistency of its opaque and transparent regions.
+We evaluate our model across several image matting benchmarks, and the results
+indicate that DiffusionMat consistently outperforms existing methods. Project
+page at~\url{https://cnnlstm.github.io/DiffusionMat",cs.CV,['cs.CV']
+Relightable and Animatable Neural Avatar from Sparse-View Video,Zhen Xu · Sida Peng · Chen Geng · Linzhan Mou · Zihan Yan · Jiaming Sun · Hujun Bao · Xiaowei Zhou,https://zju3dv.github.io/relightable_avatar,https://arxiv.org/abs/2308.07903,,2308.07903.pdf,Relightable and Animatable Neural Avatar from Sparse-View Video,"This paper tackles the challenge of creating relightable and animatable
+neural avatars from sparse-view (or even monocular) videos of dynamic humans
+under unknown illumination. Compared to studio environments, this setting is
+more practical and accessible but poses an extremely challenging ill-posed
+problem. Previous neural human reconstruction methods are able to reconstruct
+animatable avatars from sparse views using deformed Signed Distance Fields
+(SDF) but cannot recover material parameters for relighting. While
+differentiable inverse rendering-based methods have succeeded in material
+recovery of static objects, it is not straightforward to extend them to dynamic
+humans as it is computationally intensive to compute pixel-surface intersection
+and light visibility on deformed SDFs for inverse rendering. To solve this
+challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to
+approximate the world space distances under arbitrary human poses.
+Specifically, we estimate coarse distances based on a parametric human model
+and compute fine distances by exploiting the local deformation invariance of
+SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently
+estimate the surface intersection and light visibility. This allows us to
+develop the first system to recover animatable and relightable neural avatars
+from sparse view (or monocular) inputs. Experiments demonstrate that our
+approach is able to produce superior results compared to state-of-the-art
+methods. Our code will be released for reproducibility.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+DeepCache: Accelerating Diffusion Models for Free,Xinyin Ma · Gongfan Fang · Xinchao Wang, ,https://arxiv.org/abs/2312.00858,,2312.00858.pdf,DeepCache: Accelerating Diffusion Models for Free,"Diffusion models have recently gained unprecedented attention in the field of
+image synthesis due to their remarkable generative capabilities.
+Notwithstanding their prowess, these models often incur substantial
+computational costs, primarily attributed to the sequential denoising process
+and cumbersome model size. Traditional methods for compressing diffusion models
+typically involve extensive retraining, presenting cost and feasibility
+challenges. In this paper, we introduce DeepCache, a novel training-free
+paradigm that accelerates diffusion models from the perspective of model
+architecture. DeepCache capitalizes on the inherent temporal redundancy
+observed in the sequential denoising steps of diffusion models, which caches
+and retrieves features across adjacent denoising stages, thereby curtailing
+redundant computations. Utilizing the property of the U-Net, we reuse the
+high-level features while updating the low-level features in a very cheap way.
+This innovative strategy, in turn, enables a speedup factor of 2.3$\times$ for
+Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1$\times$
+for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments
+also demonstrate DeepCache's superiority over existing pruning and distillation
+methods that necessitate retraining and its compatibility with current sampling
+techniques. Furthermore, we find that under the same throughput, DeepCache
+effectively achieves comparable or even marginally improved results with DDIM
+or PLMS. The code is available at https://github.com/horseee/DeepCache",cs.CV,"['cs.CV', 'cs.AI']"
+Unsupervised Occupancy Learning from Sparse Point Cloud,Amine Ouasfi · Adnane Boukhayma, ,https://arxiv.org/abs/2404.02759,,2404.02759.pdf,Unsupervised Occupancy Learning from Sparse Point Cloud,"Implicit Neural Representations have gained prominence as a powerful
+framework for capturing complex data modalities, encompassing a wide range from
+3D shapes to images and audio. Within the realm of 3D shape representation,
+Neural Signed Distance Functions (SDF) have demonstrated remarkable potential
+in faithfully encoding intricate shape geometry. However, learning SDFs from 3D
+point clouds in the absence of ground truth supervision remains a very
+challenging task. In this paper, we propose a method to infer occupancy fields
+instead of SDFs as they are easier to learn from sparse inputs. We leverage a
+margin-based uncertainty measure to differentially sample from the decision
+boundary of the occupancy function and supervise the sampled boundary points
+using the input point cloud. We further stabilize the optimization process at
+the early stages of the training by biasing the occupancy function towards
+minimal entropy fields while maximizing its entropy at the input point cloud.
+Through extensive experiments and evaluations, we illustrate the efficacy of
+our proposed method, highlighting its capacity to improve implicit shape
+inference with respect to baselines and the state-of-the-art using synthetic
+and real data.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+SGC-Occ: Semantic-Geometry Consistent 3D Occupancy Prediction for Autonomous Driving,Zhiwen Yang · Xiangteng He · Yuxin Peng, ,https://arxiv.org/abs/2403.08748,,2403.08748.pdf,Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution,"In autonomous vehicles, understanding the surrounding 3D environment of the
+ego vehicle in real-time is essential. A compact way to represent scenes while
+encoding geometric distances and semantic object information is via 3D semantic
+occupancy maps. State of the art 3D mapping methods leverage transformers with
+cross-attention mechanisms to elevate 2D vision-centric camera features into
+the 3D domain. However, these methods encounter significant challenges in
+real-time applications due to their high computational demands during
+inference. This limitation is particularly problematic in autonomous vehicles,
+where GPU resources must be shared with other tasks such as localization and
+planning. In this paper, we introduce an approach that extracts features from
+front-view 2D camera images and LiDAR scans, then employs a sparse convolution
+network (Minkowski Engine), for 3D semantic occupancy prediction. Given that
+outdoor scenes in autonomous driving scenarios are inherently sparse, the
+utilization of sparse convolution is particularly apt. By jointly solving the
+problems of 3D scene completion of sparse scenes and 3D semantic segmentation,
+we provide a more efficient learning framework suitable for real-time
+applications in autonomous vehicles. We also demonstrate competitive accuracy
+on the nuScenes dataset.",cs.RO,"['cs.RO', 'cs.CV']"
+Countering Personalized Text-to-Image Generation with Influence Watermarks,Hanwen Liu · Zhicheng Sun · Yadong Mu, ,https://arxiv.org/abs/2312.15905,,,Cross Initialization for Personalized Text-to-Image Generation,"Recently, there has been a surge in face personalization techniques,
+benefiting from the advanced capabilities of pretrained text-to-image diffusion
+models. Among these, a notable method is Textual Inversion, which generates
+personalized images by inverting given images into textual embeddings. However,
+methods based on Textual Inversion still struggle with balancing the trade-off
+between reconstruction quality and editability. In this study, we examine this
+issue through the lens of initialization. Upon closely examining traditional
+initialization methods, we identified a significant disparity between the
+initial and learned embeddings in terms of both scale and orientation. The
+scale of the learned embedding can be up to 100 times greater than that of the
+initial embedding. Such a significant change in the embedding could increase
+the risk of overfitting, thereby compromising the editability. Driven by this
+observation, we introduce a novel initialization method, termed Cross
+Initialization, that significantly narrows the gap between the initial and
+learned embeddings. This method not only improves both reconstruction and
+editability but also reduces the optimization steps from 5000 to 320.
+Furthermore, we apply a regularization term to keep the learned embedding close
+to the initial embedding. We show that when combined with Cross Initialization,
+this regularization term can effectively improve editability. We provide
+comprehensive empirical evidence to demonstrate the superior performance of our
+method compared to the baseline methods. Notably, in our experiments, Cross
+Initialization is the only method that successfully edits an individual's
+facial expression. Additionally, a fast version of our method allows for
+capturing an input image in roughly 26 seconds, while surpassing the baseline
+methods in terms of both reconstruction and editability. Code will be made
+publicly available.",cs.CV,['cs.CV']
+GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields,Yunsong Wang · Hanlin Chen · Gim Hee Lee, ,https://arxiv.org/abs/2404.00931,,2404.00931.pdf,GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields,"Recent advancements in vision-language foundation models have significantly
+enhanced open-vocabulary 3D scene understanding. However, the generalizability
+of existing methods is constrained due to their framework designs and their
+reliance on 3D data. We address this limitation by introducing Generalizable
+Open-Vocabulary Neural Semantic Fields (GOV-NeSF), a novel approach offering a
+generalizable implicit representation of 3D scenes with open-vocabulary
+semantics. We aggregate the geometry-aware features using a cost volume, and
+propose a Multi-view Joint Fusion module to aggregate multi-view features
+through a cross-view attention mechanism, which effectively predicts
+view-specific blending weights for both colors and open-vocabulary features.
+Remarkably, our GOV-NeSF exhibits state-of-the-art performance in both 2D and
+3D open-vocabulary semantic segmentation, eliminating the need for ground truth
+semantic labels or depth priors, and effectively generalize across scenes and
+datasets without fine-tuning.",cs.CV,['cs.CV']
+NeuRAD: Neural Rendering for Autonomous Driving,Adam Tonderski · Carl Lindström · Georg Hess · William Ljungbergh · Lennart Svensson · Christoffer Petersson,https://research.zenseact.com/publications/neurad/,https://arxiv.org/abs/2311.15260,,2311.15260.pdf,NeuRAD: Neural Rendering for Autonomous Driving,"Neural radiance fields (NeRFs) have gained popularity in the autonomous
+driving (AD) community. Recent methods show NeRFs' potential for closed-loop
+simulation, enabling testing of AD systems, and as an advanced training data
+augmentation technique. However, existing methods often require long training
+times, dense semantic supervision, or lack generalizability. This, in turn,
+hinders the application of NeRFs for AD at scale. In this paper, we propose
+NeuRAD, a robust novel view synthesis method tailored to dynamic AD data. Our
+method features simple network design, extensive sensor modeling for both
+camera and lidar -- including rolling shutter, beam divergence and ray dropping
+-- and is applicable to multiple datasets out of the box. We verify its
+performance on five popular AD datasets, achieving state-of-the-art performance
+across the board. To encourage further development, we will openly release the
+NeuRAD source code. See https://github.com/georghess/NeuRAD .",cs.CV,['cs.CV']
+Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning,Wei Zhang · Chaoqun Wan · Tongliang Liu · Xinmei Tian · Xu Shen · Jieping Ye, ,https://arxiv.org/abs/2404.00801,,2404.00801.pdf,$R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding,"Video temporal grounding (VTG) is a fine-grained video understanding problem
+that aims to ground relevant clips in untrimmed videos given natural language
+queries. Most existing VTG models are built upon frame-wise final-layer CLIP
+features, aided by additional temporal backbones (e.g., SlowFast) with
+sophisticated temporal reasoning mechanisms. In this work, we claim that CLIP
+itself already shows great potential for fine-grained spatial-temporal
+modeling, as each layer offers distinct yet useful information under different
+granularity levels. Motivated by this, we propose Reversed Recurrent Tuning
+($R^2$-Tuning), a parameter- and memory-efficient transfer learning framework
+for video temporal grounding. Our method learns a lightweight $R^2$ Block
+containing only 1.5% of the total parameters to perform progressive
+spatial-temporal modeling. Starting from the last layer of CLIP, $R^2$ Block
+recurrently aggregates spatial features from earlier layers, then refines
+temporal correlation conditioning on the given query, resulting in a
+coarse-to-fine scheme. $R^2$-Tuning achieves state-of-the-art performance
+across three VTG tasks (i.e., moment retrieval, highlight detection, and video
+summarization) on six public benchmarks (i.e., QVHighlights, Charades-STA,
+Ego4D-NLQ, TACoS, YouTube Highlights, and TVSum) even without the additional
+backbone, demonstrating the significance and effectiveness of the proposed
+scheme. Our code is available at https://github.com/yeliudev/R2-Tuning.",cs.CV,['cs.CV']
+Enhancing Vision-Language Pretraining with Rich Supervisions,Yuan Gao · Kunyu Shi · Pengkai Zhu · Edouard Belval · Oren Nuriel · Srikar Appalaraju · Shabnam Ghadar · Zhuowen Tu · Vijay Mahadevan · Stefano Soatto, ,https://arxiv.org/abs/2403.03346,,,Enhancing Vision-Language Pre-training with Rich Supervisions,"We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
+pre-training paradigm for Vision-Language Models using data from large-scale
+web screenshot rendering. Using web screenshots unlocks a treasure trove of
+visual and textual cues that are not present in using image-text pairs. In S4,
+we leverage the inherent tree-structured hierarchy of HTML elements and the
+spatial localization to carefully design 10 pre-training tasks with large scale
+annotated data. These tasks resemble downstream tasks across different domains
+and the annotations are cheap to obtain. We demonstrate that, compared to
+current screenshot pre-training objectives, our innovative pre-training method
+significantly enhances performance of image-to-text model in nine varied and
+popular downstream tasks - up to 76.1% improvements on Table Detection, and at
+least 1% on Widget Captioning.",cs.CV,['cs.CV']
+A Category Agnostic Model for Visual Rearrangement,Yuyi Liu · Xinhang Song · Weijie Li · Xiaohan Wang · Shuqiang Jiang, ,,http://vipl.ict.ac.cn/en/news/researchevents/202403/t20240315_207762.html,,,,,nan
+Polos: Multimodal Metric Learning from Human Feedback for Image Captioning,Yuiga Wada · Kanta Kaneda · Daichi Saito · Komei Sugiura,https://yuiga.dev/polos,https://arxiv.org/abs/2402.18091,,2402.18091.pdf,Polos: Multimodal Metric Learning from Human Feedback for Image Captioning,"Establishing an automatic evaluation metric that closely aligns with human
+judgments is essential for effectively developing image captioning models.
+Recent data-driven metrics have demonstrated a stronger correlation with human
+judgments than classic metrics such as CIDEr; however they lack sufficient
+capabilities to handle hallucinations and generalize across diverse images and
+texts partially because they compute scalar similarities merely using
+embeddings learned from tasks unrelated to image captioning evaluation. In this
+study, we propose Polos, a supervised automatic evaluation metric for image
+captioning models. Polos computes scores from multimodal inputs, using a
+parallel feature extraction mechanism that leverages embeddings trained through
+large-scale contrastive learning. To train Polos, we introduce Multimodal
+Metric Learning from Human Feedback (M$^2$LHF), a framework for developing
+metrics based on human feedback. We constructed the Polaris dataset, which
+comprises 131K human judgments from 550 evaluators, which is approximately ten
+times larger than standard datasets. Our approach achieved state-of-the-art
+performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and
+the Polaris dataset, thereby demonstrating its effectiveness and robustness.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+CLIB-FIQA: Face Image Quality Assessment with Confidence Calibration,Fu-Zhao Ou · Fu-Zhao Ou · Chongyi Li · Shiqi Wang · Sam Kwong, ,https://arxiv.org/abs/2404.12203,,2404.12203.pdf,GraFIQs: Face Image Quality Assessment Using Gradient Magnitudes,"Face Image Quality Assessment (FIQA) estimates the utility of face images for
+automated face recognition (FR) systems. We propose in this work a novel
+approach to assess the quality of face images based on inspecting the required
+changes in the pre-trained FR model weights to minimize differences between
+testing samples and the distribution of the FR training dataset. To achieve
+that, we propose quantifying the discrepancy in Batch Normalization statistics
+(BNS), including mean and variance, between those recorded during FR training
+and those obtained by processing testing samples through the pretrained FR
+model. We then generate gradient magnitudes of pretrained FR weights by
+backpropagating the BNS through the pretrained model. The cumulative absolute
+sum of these gradient magnitudes serves as the FIQ for our approach. Through
+comprehensive experimentation, we demonstrate the effectiveness of our
+training-free and quality labeling-free approach, achieving competitive
+performance to recent state-of-theart FIQA approaches without relying on
+quality labeling, the need to train regression networks, specialized
+architectures, or designing and optimizing specific loss functions.",cs.CV,['cs.CV']
+EVCap: Retrieval-Augmented Image Captioning with External Visual--Name Memory for Open-World Comprehension,Jiaxuan Li · Duc Minh Vo · Akihiro Sugimoto · Hideki Nakayama, ,https://arxiv.org/abs/2311.15879v2,,2311.15879v2.pdf,EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension,"Large language models (LLMs)-based image captioning has the capability of
+describing objects not explicitly observed in training data; yet novel objects
+occur frequently, necessitating the requirement of sustaining up-to-date object
+knowledge for open-world comprehension. Instead of relying on large amounts of
+data and/or scaling up network parameters, we introduce a highly effective
+retrieval-augmented image captioning method that prompts LLMs with object names
+retrieved from External Visual--name memory (EVCap). We build ever-changing
+object knowledge memory using objects' visuals and names, enabling us to (i)
+update the memory at a minimal cost and (ii) effortlessly augment LLMs with
+retrieved object names by utilizing a lightweight and fast-to-train model. Our
+model, which was trained only on the COCO dataset, can adapt to out-of-domain
+without requiring additional fine-tuning or re-training. Our experiments
+conducted on benchmarks and synthetic commonsense-violating data show that
+EVCap, with only 3.97M trainable parameters, exhibits superior performance
+compared to other methods based on frozen pre-trained LLMs. Its performance is
+also competitive to specialist SOTAs that require extensive training.",cs.CV,['cs.CV']
+On Exact Inversion of DPM-Solvers,Seongmin Hong · Kyeonghyun Lee · Suh Yoon Jeon · Hyewon Bae · Se Young Chun,https://smhongok.github.io/inv-dpm.html,https://arxiv.org/abs/2311.18387v1,,2311.18387v1.pdf,On Exact Inversion of DPM-Solvers,"Diffusion probabilistic models (DPMs) are a key component in modern
+generative models. DPM-solvers have achieved reduced latency and enhanced
+quality significantly, but have posed challenges to find the exact inverse
+(i.e., finding the initial noise from the given image). Here we investigate the
+exact inversions for DPM-solvers and propose algorithms to perform them when
+samples are generated by the first-order as well as higher-order DPM-solvers.
+For each explicit denoising step in DPM-solvers, we formulated the inversions
+using implicit methods such as gradient descent or forward step method to
+ensure the robustness to large classifier-free guidance unlike the prior
+approach using fixed-point iteration. Experimental results demonstrated that
+our proposed exact inversion methods significantly reduced the error of both
+image and noise reconstructions, greatly enhanced the ability to distinguish
+invisible watermarks and well prevented unintended background changes
+consistently during image editing. Project page:
+\url{https://smhongok.github.io/inv-dpm.html}.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning Structure-from-Motion with Graph Attention Networks,Lucas Brynte · José Pedro Iglesias · Carl Olsson · Fredrik Kahl,https://github.com/lucasbrynte/gasfm/,https://arxiv.org/abs/2308.15984,,2308.15984.pdf,Learning Structure-from-Motion with Graph Attention Networks,"In this paper we tackle the problem of learning Structure-from-Motion (SfM)
+through the use of graph attention networks. SfM is a classic computer vision
+problem that is solved though iterative minimization of reprojection errors,
+referred to as Bundle Adjustment (BA), starting from a good initialization. In
+order to obtain a good enough initialization to BA, conventional methods rely
+on a sequence of sub-problems (such as pairwise pose estimation, pose averaging
+or triangulation) which provide an initial solution that can then be refined
+using BA. In this work we replace these sub-problems by learning a model that
+takes as input the 2D keypoints detected across multiple views, and outputs the
+corresponding camera poses and 3D keypoint coordinates. Our model takes
+advantage of graph neural networks to learn SfM-specific primitives, and we
+show that it can be used for fast inference of the reconstruction for new and
+unseen sequences. The experimental results show that the proposed model
+outperforms competing learning-based methods, and challenges COLMAP while
+having lower runtime. Our code is available at
+https://github.com/lucasbrynte/gasfm/.",cs.CV,"['cs.CV', 'cs.LG']"
+Plug and Play Active Learning for Object Detection,Chenhongyi Yang · Lichao Huang · Elliot Crowley, ,,https://allainews.com/item/plug-and-play-active-learning-for-object-detection-2024-03-15/,,,,,nan
+MACE: Mass Concept Erasure in Diffusion Models,Shilin Lu · Zilan Wang · Leyang Li · Yanzhu Liu · Adams Wai-Kin Kong,https://github.com/Shilin-LU/MACE,https://arxiv.org/abs/2403.06135,,2403.06135.pdf,MACE: Mass Concept Erasure in Diffusion Models,"The rapid expansion of large-scale text-to-image diffusion models has raised
+growing concerns regarding their potential misuse in creating harmful or
+misleading content. In this paper, we introduce MACE, a finetuning framework
+for the task of mass concept erasure. This task aims to prevent models from
+generating images that embody unwanted concepts when prompted. Existing concept
+erasure methods are typically restricted to handling fewer than five concepts
+simultaneously and struggle to find a balance between erasing concept synonyms
+(generality) and maintaining unrelated concepts (specificity). In contrast,
+MACE differs by successfully scaling the erasure scope up to 100 concepts and
+by achieving an effective balance between generality and specificity. This is
+achieved by leveraging closed-form cross-attention refinement along with LoRA
+finetuning, collectively eliminating the information of undesirable concepts.
+Furthermore, MACE integrates multiple LoRAs without mutual interference. We
+conduct extensive evaluations of MACE against prior methods across four
+different tasks: object erasure, celebrity erasure, explicit content erasure,
+and artistic style erasure. Our results reveal that MACE surpasses prior
+methods in all evaluated tasks. Code is available at
+https://github.com/Shilin-LU/MACE.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Contextual Augmented Global Contrast for Multimodal Intent Recognition,Kaili Sun · Zhiwen Xie · Mang Ye · Huyin Zhang, ,https://arxiv.org/html/2312.14667v1,,2312.14667v1.pdf,Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition,"Multimodal intent recognition aims to leverage diverse modalities such as
+expressions, body movements and tone of speech to comprehend user's intent,
+constituting a critical task for understanding human language and behavior in
+real-world multimodal scenarios. Nevertheless, the majority of existing methods
+ignore potential correlations among different modalities and own limitations in
+effectively learning semantic features from nonverbal modalities. In this
+paper, we introduce a token-level contrastive learning method with
+modality-aware prompting (TCL-MAP) to address the above challenges. To
+establish an optimal multimodal semantic environment for text modality, we
+develop a modality-aware prompting module (MAP), which effectively aligns and
+fuses features from text, video and audio modalities with similarity-based
+modality alignment and cross-modality attention mechanism. Based on the
+modality-aware prompt and ground truth labels, the proposed token-level
+contrastive learning framework (TCL) constructs augmented samples and employs
+NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal
+textual semantic insights derived from intent labels to guide the learning
+processes of other modalities in return. Extensive experiments show that our
+method achieves remarkable improvements compared to state-of-the-art methods.
+Additionally, ablation analyses demonstrate the superiority of the
+modality-aware prompt over the handcrafted prompt, which holds substantial
+significance for multimodal prompt learning. The codes are released at
+https://github.com/thuiar/TCL-MAP.",cs.MM,"['cs.MM', 'cs.LG']"
+Fixed Point Diffusion Models,Luke Melas-Kyriazi · Xingjian Bai, ,https://arxiv.org/abs/2401.08741,,2401.08741.pdf,Fixed Point Diffusion Models,"We introduce the Fixed Point Diffusion Model (FPDM), a novel approach to
+image generation that integrates the concept of fixed point solving into the
+framework of diffusion-based generative modeling. Our approach embeds an
+implicit fixed point solving layer into the denoising network of a diffusion
+model, transforming the diffusion process into a sequence of closely-related
+fixed point problems. Combined with a new stochastic training method, this
+approach significantly reduces model size, reduces memory usage, and
+accelerates training. Moreover, it enables the development of two new
+techniques to improve sampling efficiency: reallocating computation across
+timesteps and reusing fixed point solutions between timesteps. We conduct
+extensive experiments with state-of-the-art models on ImageNet, FFHQ,
+CelebA-HQ, and LSUN-Church, demonstrating substantial improvements in
+performance and efficiency. Compared to the state-of-the-art DiT model, FPDM
+contains 87% fewer parameters, consumes 60% less memory during training, and
+improves image generation quality in situations where sampling computation or
+time is limited. Our code and pretrained models are available at
+https://lukemelas.github.io/fixed-point-diffusion-models.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+High Fidelity Person-centric Subject-to-Image Synthesis,Yibin Wang · Weizhong Zhang · Jianwei Zheng · Cheng Jin, ,https://arxiv.org/abs/2311.10329,,2311.10329.pdf,High-fidelity Person-centric Subject-to-Image Synthesis,"Current subject-driven image generation methods encounter significant
+challenges in person-centric image generation. The reason is that they learn
+the semantic scene and person generation by fine-tuning a common pre-trained
+diffusion, which involves an irreconcilable training imbalance. Precisely, to
+generate realistic persons, they need to sufficiently tune the pre-trained
+model, which inevitably causes the model to forget the rich semantic scene
+prior and makes scene generation over-fit to the training data. Moreover, even
+with sufficient fine-tuning, these methods can still not generate high-fidelity
+persons since joint learning of the scene and person generation also lead to
+quality compromise. In this paper, we propose Face-diffuser, an effective
+collaborative generation pipeline to eliminate the above training imbalance and
+quality compromise. Specifically, we first develop two specialized pre-trained
+diffusion models, i.e., Text-driven Diffusion Model (TDM) and Subject-augmented
+Diffusion Model (SDM), for scene and person generation, respectively. The
+sampling process is divided into three sequential stages, i.e., semantic scene
+construction, subject-scene fusion, and subject enhancement. The first and last
+stages are performed by TDM and SDM respectively. The subject-scene fusion
+stage, that is the collaboration achieved through a novel and highly effective
+mechanism, Saliency-adaptive Noise Fusion (SNF). Specifically, it is based on
+our key observation that there exists a robust link between classifier-free
+guidance responses and the saliency of generated images. In each time step, SNF
+leverages the unique strengths of each model and allows for the spatial
+blending of predicted noises from both models automatically in a saliency-aware
+manner. Extensive experiments confirm the impressive effectiveness and
+robustness of the Face-diffuser.",cs.CV,"['cs.CV', 'cs.AI']"
+On the Content Bias in Fréchet Video Distance,Songwei Ge · Aniruddha Mahapatra · Gaurav Parmar · Jun-Yan Zhu · Jia-Bin Huang, ,https://arxiv.org/abs/2404.12391,,2404.12391.pdf,On the Content Bias in Fréchet Video Distance,"Fr\'echet Video Distance (FVD), a prominent metric for evaluating video
+generation models, is known to conflict with human perception occasionally. In
+this paper, we aim to explore the extent of FVD's bias toward per-frame quality
+over temporal realism and identify its sources. We first quantify the FVD's
+sensitivity to the temporal axis by decoupling the frame and motion quality and
+find that the FVD increases only slightly with large temporal corruption. We
+then analyze the generated videos and show that via careful sampling from a
+large set of generated videos that do not contain motions, one can drastically
+decrease FVD without improving the temporal quality. Both studies suggest FVD's
+bias towards the quality of individual frames. We further observe that the bias
+can be attributed to the features extracted from a supervised video classifier
+trained on the content-biased dataset. We show that FVD with features extracted
+from the recent large-scale self-supervised video models is less biased toward
+image quality. Finally, we revisit a few real-world examples to validate our
+hypothesis.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Implicit Discriminative Knowledge Learning for Visible-Infrared Person Re-Identification,kaijie ren · Lei Zhang, ,https://arxiv.org/abs/2403.11708v2,,2403.11708v2.pdf,Implicit Discriminative Knowledge Learning for Visible-Infrared Person Re-Identification,"Visible-Infrared Person Re-identification (VI-ReID) is a challenging
+cross-modal pedestrian retrieval task, due to significant intra-class
+variations and cross-modal discrepancies among different cameras. Existing
+works mainly focus on embedding images of different modalities into a unified
+space to mine modality-shared features. They only seek distinctive information
+within these shared features, while ignoring the identity-aware useful
+information that is implicit in the modality-specific features. To address this
+issue, we propose a novel Implicit Discriminative Knowledge Learning (IDKL)
+network to uncover and leverage the implicit discriminative information
+contained within the modality-specific. First, we extract modality-specific and
+modality-shared features using a novel dual-stream network. Then, the
+modality-specific features undergo purification to reduce their modality style
+discrepancies while preserving identity-aware discriminative knowledge.
+Subsequently, this kind of implicit knowledge is distilled into the
+modality-shared feature to enhance its distinctiveness. Finally, an alignment
+loss is proposed to minimize modality discrepancy on enhanced modality-shared
+features. Extensive experiments on multiple public datasets demonstrate the
+superiority of IDKL network over the state-of-the-art methods. Code is
+available at https://github.com/1KK077/IDKL.",cs.CV,['cs.CV']
+PointBeV: A Sparse Approach for BeV Predictions,Loick Chambon · Éloi Zablocki · Mickaël Chen · Florent Bartoccioni · Patrick Pérez · Matthieu Cord, ,https://arxiv.org/abs/2312.00703,,2312.00703.pdf,PointBeV: A Sparse Approach to BeV Predictions,"Bird's-eye View (BeV) representations have emerged as the de-facto shared
+space in driving applications, offering a unified space for sensor data fusion
+and supporting various downstream tasks. However, conventional models use grids
+with fixed resolution and range and face computational inefficiencies due to
+the uniform allocation of resources across all cells. To address this, we
+propose PointBeV, a novel sparse BeV segmentation model operating on sparse BeV
+cells instead of dense grids. This approach offers precise control over memory
+usage, enabling the use of long temporal contexts and accommodating
+memory-constrained platforms. PointBeV employs an efficient two-pass strategy
+for training, enabling focused computation on regions of interest. At inference
+time, it can be used with various memory/performance trade-offs and flexibly
+adjusts to new specific use cases. PointBeV achieves state-of-the-art results
+on the nuScenes dataset for vehicle, pedestrian, and lane segmentation,
+showcasing superior performance in static and temporal settings despite being
+trained solely with sparse signals. We will release our code along with two new
+efficient modules used in the architecture: Sparse Feature Pulling, designed
+for the effective extraction of features from images to BeV, and Submanifold
+Attention, which enables efficient temporal modeling. Our code is available at
+https://github.com/valeoai/PointBeV.",cs.CV,['cs.CV']
+Behind the Veil: Enhanced Indoor 3D Scene Reconstruction with Occluded Surfaces Completion,Su Sun · Cheng Zhao · Yuliang Guo · Ruoyu Wang · Xinyu Huang · Yingjie Victor Chen · Liu Ren, ,https://arxiv.org/abs/2404.03070,,2404.03070.pdf,Behind the Veil: Enhanced Indoor 3D Scene Reconstruction with Occluded Surfaces Completion,"In this paper, we present a novel indoor 3D reconstruction method with
+occluded surface completion, given a sequence of depth readings. Prior
+state-of-the-art (SOTA) methods only focus on the reconstruction of the visible
+areas in a scene, neglecting the invisible areas due to the occlusions, e.g.,
+the contact surface between furniture, occluded wall and floor. Our method
+tackles the task of completing the occluded scene surfaces, resulting in a
+complete 3D scene mesh. The core idea of our method is learning 3D geometry
+prior from various complete scenes to infer the occluded geometry of an unseen
+scene from solely depth measurements. We design a coarse-fine hierarchical
+octree representation coupled with a dual-decoder architecture, i.e.,
+Geo-decoder and 3D Inpainter, which jointly reconstructs the complete 3D scene
+geometry. The Geo-decoder with detailed representation at fine levels is
+optimized online for each scene to reconstruct visible surfaces. The 3D
+Inpainter with abstract representation at coarse levels is trained offline
+using various scenes to complete occluded surfaces. As a result, while the
+Geo-decoder is specialized for an individual scene, the 3D Inpainter can be
+generally applied across different scenes. We evaluate the proposed method on
+the 3D Completed Room Scene (3D-CRS) and iTHOR datasets, significantly
+outperforming the SOTA methods by a gain of 16.8% and 24.2% in terms of the
+completeness of 3D reconstruction. 3D-CRS dataset including a complete 3D mesh
+of each scene is provided at project webpage.",cs.CV,['cs.CV']
+VidLA: Video-Language Alignment at Scale,Mamshad Nayeem Rizve · Fan Fei · Jayakrishnan Unnikrishnan · Son Dinh Tran · Benjamin Yao · Belinda Zeng · Mubarak Shah · Trishul Chilimbi, ,https://arxiv.org/abs/2403.14870,,2403.14870.pdf,VidLA: Video-Language Alignment at Scale,"In this paper, we propose VidLA, an approach for video-language alignment at
+scale. There are two major limitations of previous video-language alignment
+approaches. First, they do not capture both short-range and long-range temporal
+dependencies and typically employ complex hierarchical deep network
+architectures that are hard to integrate with existing pretrained image-text
+foundation models. To effectively address this limitation, we instead keep the
+network architecture simple and use a set of data tokens that operate at
+different temporal resolutions in a hierarchical manner, accounting for the
+temporally hierarchical nature of videos. By employing a simple two-tower
+architecture, we are able to initialize our video-language model with
+pretrained image-text foundation models, thereby boosting the final
+performance. Second, existing video-language alignment works struggle due to
+the lack of semantically aligned large-scale training data. To overcome it, we
+leverage recent LLMs to curate the largest video-language dataset to date with
+better visual grounding. Furthermore, unlike existing video-text datasets which
+only contain short clips, our dataset is enriched with video clips of varying
+durations to aid our temporally hierarchical data tokens in extracting better
+representations at varying temporal scales. Overall, empirical results show
+that our proposed approach surpasses state-of-the-art methods on multiple
+retrieval benchmarks, especially on longer videos, and performs competitively
+on classification benchmarks.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+ODIN: A Single Model for 2D and 3D Segmentation,Ayush Jain · Pushkal Katara · Nikolaos Gkanatsios · Adam Harley · Gabriel Sarch · Kriti Aggarwal · Vishrav Chaudhary · Katerina Fragkiadaki, ,https://arxiv.org/abs/2401.02416,,2401.02416.pdf,ODIN: A Single Model for 2D and 3D Segmentation,"State-of-the-art models on contemporary 3D segmentation benchmarks like
+ScanNet consume and label dataset-provided 3D point clouds, obtained through
+post processing of sensed multiview RGB-D images. They are typically trained
+in-domain, forego large-scale 2D pre-training and outperform alternatives that
+featurize the posed RGB-D multiview images instead. The gap in performance
+between methods that consume posed images versus post-processed 3D point clouds
+has fueled the belief that 2D and 3D perception require distinct model
+architectures. In this paper, we challenge this view and propose ODIN
+(Omni-Dimensional INstance segmentation), a model that can segment and label
+both 2D RGB images and 3D point clouds, using a transformer architecture that
+alternates between 2D within-view and 3D cross-view information fusion. Our
+model differentiates 2D and 3D feature operations through the positional
+encodings of the tokens involved, which capture pixel coordinates for 2D patch
+tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
+performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
+benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
+outperforms all previous works by a wide margin when the sensed 3D point cloud
+is used in place of the point cloud sampled from 3D mesh. When used as the 3D
+perception engine in an instructable embodied agent architecture, it sets a new
+state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
+checkpoints can be found at the project website (https://odin-seg.github.io).",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding,Syed Talal Wasim · Muzammal Naseer · Salman Khan · Ming-Hsuan Yang · Fahad Shahbaz Khan, ,https://arxiv.org/abs/2401.00901,,2401.00901.pdf,Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding,"Video grounding aims to localize a spatio-temporal section in a video
+corresponding to an input text query. This paper addresses a critical
+limitation in current video grounding methodologies by introducing an
+Open-Vocabulary Spatio-Temporal Video Grounding task. Unlike prevalent
+closed-set approaches that struggle with open-vocabulary scenarios due to
+limited training data and predefined vocabularies, our model leverages
+pre-trained representations from foundational spatial grounding models. This
+empowers it to effectively bridge the semantic gap between natural language and
+diverse visual content, achieving strong performance in closed-set and
+open-vocabulary settings. Our contributions include a novel spatio-temporal
+video grounding model, surpassing state-of-the-art results in closed-set
+evaluations on multiple datasets and demonstrating superior performance in
+open-vocabulary scenarios. Notably, the proposed model outperforms
+state-of-the-art methods in closed-set settings on VidSTG (Declarative and
+Interrogative) and HC-STVG (V1 and V2) datasets. Furthermore, in
+open-vocabulary evaluations on HC-STVG V1 and YouCook-Interactions, our model
+surpasses the recent best-performing models by $4.88$ m_vIoU and $1.83\%$
+accuracy, demonstrating its efficacy in handling diverse linguistic and visual
+concepts for improved video understanding. Our codes will be publicly released.",cs.CV,['cs.CV']
+Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts,Jiawen Zhu · Guansong Pang, ,https://arxiv.org/abs/2403.06495,,2403.06495.pdf,Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts,"This paper explores the problem of Generalist Anomaly Detection (GAD), aiming
+to train one single detection model that can generalize to detect anomalies in
+diverse datasets from different application domains without any further
+training on the target data. Some recent studies have shown that large
+pre-trained Visual-Language Models (VLMs) like CLIP have strong generalization
+capabilities on detecting industrial defects from various datasets, but their
+methods rely heavily on handcrafted text prompts about defects, making them
+difficult to generalize to anomalies in other applications, e.g., medical image
+anomalies or semantic anomalies in natural images. In this work, we propose to
+train a GAD model with few-shot normal images as sample prompts for AD on
+diverse datasets on the fly. To this end, we introduce a novel approach that
+learns an in-context residual learning model for GAD, termed InCTRL. It is
+trained on an auxiliary dataset to discriminate anomalies from normal samples
+based on a holistic evaluation of the residuals between query images and
+few-shot normal sample prompts. Regardless of the datasets, per definition of
+anomaly, larger residuals are expected for anomalies than normal samples,
+thereby enabling InCTRL to generalize across different domains without further
+training. Comprehensive experiments on nine AD datasets are performed to
+establish a GAD benchmark that encapsulate the detection of industrial defect
+anomalies, medical anomalies, and semantic anomalies in both one-vs-all and
+multi-class setting, on which InCTRL is the best performer and significantly
+outperforms state-of-the-art competing methods. Code is available at
+https://github.com/mala-lab/InCTRL.",cs.CV,['cs.CV']
+ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting,Chen Duan · Pei Fu · Shan Guo · Qianyi Jiang · Xiaoming Wei, ,https://arxiv.org/abs/2403.00303,,2403.00303.pdf,ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting,"In recent years, text-image joint pre-training techniques have shown
+promising results in various tasks. However, in Optical Character Recognition
+(OCR) tasks, aligning text instances with their corresponding text regions in
+images poses a challenge, as it requires effective alignment between text and
+OCR-Text (referring to the text in images as OCR-Text to distinguish from the
+text in natural language) rather than a holistic understanding of the overall
+image content. In this paper, we propose a new pre-training method called
+OCR-Text Destylization Modeling (ODM) that transfers diverse styles of text
+found in images to a uniform style based on the text prompt. With ODM, we
+achieve better alignment between text and OCR-Text and enable pre-trained
+models to adapt to the complex and diverse styles of scene text detection and
+spotting tasks. Additionally, we have designed a new labeling generation method
+specifically for ODM and combined it with our proposed Text-Controller module
+to address the challenge of annotation costs in OCR tasks, allowing a larger
+amount of unlabeled data to participate in pre-training. Extensive experiments
+on multiple public datasets demonstrate that our method significantly improves
+performance and outperforms current pre-training methods in scene text
+detection and spotting tasks. Code is available at
+https://github.com/PriNing/ODM.",cs.CV,['cs.CV']
+"LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning",Sijin Chen · Xin Chen · Chi Zhang · Mingsheng Li · Gang Yu · Hao Fei · Hongyuan Zhu · Jiayuan Fan · Tao Chen, ,https://arxiv.org/abs/2311.18651,,2311.18651.pdf,"LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning","Recent advances in Large Multimodal Models (LMM) have made it possible for
+various applications in human-machine interactions. However, developing LMMs
+that can comprehend, reason, and plan in complex and diverse 3D environments
+remains a challenging topic, especially considering the demand for
+understanding permutation-invariant point cloud 3D representations of the 3D
+scene. Existing works seek help from multi-view images, and project 2D features
+to 3D space as 3D scene representations. This, however, leads to huge
+computational overhead and performance degradation. In this paper, we present
+LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and
+respond to both textual-instructions and visual-prompts. This help LMMs better
+comprehend human interactions and further help to remove the ambiguities in
+cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results,
+and surpasses various 3D vision-language models on both 3D Dense Captioning and
+3D Question Answering.",cs.CV,['cs.CV']
+UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation,Hong Li · Yutang Feng · Song Xue · Xuhui Liu · Boyu Liu · Bohan Zeng · Shanglin Li · Jianzhuang Liu · Shumin Han · Baochang Zhang, ,https://arxiv.org/abs/2403.19235,,,DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation,"While large-scale pre-trained text-to-image models can synthesize diverse and
+high-quality human-centered images, novel challenges arise with a nuanced task
+of ""identity fine editing"": precisely modifying specific features of a subject
+while maintaining its inherent identity and context. Existing personalization
+methods either require time-consuming optimization or learning additional
+encoders, adept in ""identity re-contextualization"". However, they often
+struggle with detailed and sensitive tasks like human face editing. To address
+these challenges, we introduce DreamSalon, a noise-guided, staged-editing
+framework, uniquely focusing on detailed image manipulations and
+identity-context preservation. By discerning editing and boosting stages via
+the frequency and gradient of predicted noises, DreamSalon first performs
+detailed manipulations on specific features in the editing stage, guided by
+high-frequency information, and then employs stochastic denoising in the
+boosting stage to improve image quality. For more precise editing, DreamSalon
+semantically mixes source and target textual prompts, guided by differences in
+their embedding covariances, to direct the model's focus on specific
+manipulation areas. Our experiments demonstrate DreamSalon's ability to
+efficiently and faithfully edit fine details on human faces, outperforming
+existing methods both qualitatively and quantitatively.",cs.CV,['cs.CV']
+Text2QR:  Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation,Guangyang Wu · Xiaohong Liu · Jun Jia · Xuehao Cui · Guangtao Zhai, ,https://arxiv.org/abs/2403.06452,,2403.06452.pdf,Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation,"In the digital era, QR codes serve as a linchpin connecting virtual and
+physical realms. Their pervasive integration across various applications
+highlights the demand for aesthetically pleasing codes without compromised
+scannability. However, prevailing methods grapple with the intrinsic challenge
+of balancing customization and scannability. Notably, stable-diffusion models
+have ushered in an epoch of high-quality, customizable content generation. This
+paper introduces Text2QR, a pioneering approach leveraging these advancements
+to address a fundamental challenge: concurrently achieving user-defined
+aesthetics and scanning robustness. To ensure stable generation of aesthetic QR
+codes, we introduce the QR Aesthetic Blueprint (QAB) module, generating a
+blueprint image exerting control over the entire generation process.
+Subsequently, the Scannability Enhancing Latent Refinement (SELR) process
+refines the output iteratively in the latent space, enhancing scanning
+robustness. This approach harnesses the potent generation capabilities of
+stable-diffusion models, navigating the trade-off between image aesthetics and
+QR code scannability. Our experiments demonstrate the seamless fusion of visual
+appeal with the practical utility of aesthetic QR codes, markedly outperforming
+prior methods. Codes are available at \url{https://github.com/mulns/Text2QR}",cs.CV,['cs.CV']
+Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers,Zi-Xin Zou · Zhipeng Yu · Yuan-Chen Guo · Yangguang Li · Yan-Pei Cao · Ding Liang · Song-Hai Zhang,https://zouzx.github.io/TriplaneGaussian/,https://arxiv.org/abs/2312.09147,,2312.09147.pdf,Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers,"Recent advancements in 3D reconstruction from single images have been driven
+by the evolution of generative models. Prominent among these are methods based
+on Score Distillation Sampling (SDS) and the adaptation of diffusion models in
+the 3D domain. Despite their progress, these techniques often face limitations
+due to slow optimization or rendering processes, leading to extensive training
+and optimization times. In this paper, we introduce a novel approach for
+single-view reconstruction that efficiently generates a 3D model from a single
+image via feed-forward inference. Our method utilizes two transformer-based
+networks, namely a point decoder and a triplane decoder, to reconstruct 3D
+objects using a hybrid Triplane-Gaussian intermediate representation. This
+hybrid representation strikes a balance, achieving a faster rendering speed
+compared to implicit representations while simultaneously delivering superior
+rendering quality than explicit representations. The point decoder is designed
+for generating point clouds from single images, offering an explicit
+representation which is then utilized by the triplane decoder to query Gaussian
+features for each point. This design choice addresses the challenges associated
+with directly regressing explicit 3D Gaussian attributes characterized by their
+non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to
+enable rapid rendering through splatting. Both decoders are built upon a
+scalable, transformer-based architecture and have been efficiently trained on
+large-scale 3D datasets. The evaluations conducted on both synthetic datasets
+and real-world images demonstrate that our method not only achieves higher
+quality but also ensures a faster runtime in comparison to previous
+state-of-the-art techniques. Please see our project page at
+https://zouzx.github.io/TriplaneGaussian/.",cs.CV,['cs.CV']
+Active Object Detection with Knowledge Aggregation and Distillation from Large Models,Dejie Yang · Yang Liu, ,https://arxiv.org/abs/2405.12509,,2405.12509.pdf,Active Object Detection with Knowledge Aggregation and Distillation from Large Models,"Accurately detecting active objects undergoing state changes is essential for
+comprehending human interactions and facilitating decision-making. The existing
+methods for active object detection (AOD) primarily rely on visual appearance
+of the objects within input, such as changes in size, shape and relationship
+with hands. However, these visual changes can be subtle, posing challenges,
+particularly in scenarios with multiple distracting no-change instances of the
+same category. We observe that the state changes are often the result of an
+interaction being performed upon the object, thus propose to use informed
+priors about object related plausible interactions (including semantics and
+visual appearance) to provide more reliable cues for AOD. Specifically, we
+propose a knowledge aggregation procedure to integrate the aforementioned
+informed priors into oracle queries within the teacher decoder, offering more
+object affordance commonsense to locate the active object. To streamline the
+inference process and reduce extra knowledge inputs, we propose a knowledge
+distillation approach that encourages the student decoder to mimic the
+detection capabilities of the teacher decoder using the oracle query by
+replicating its predictions and attention. Our proposed framework achieves
+state-of-the-art performance on four datasets, namely Ego4D, Epic-Kitchens,
+MECCANO, and 100DOH, which demonstrates the effectiveness of our approach in
+improving AOD.",cs.CV,['cs.CV']
+Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs,shiyu xuan · Qingpei Guo · Ming Yang · Shiliang Zhang, ,https://arxiv.org/abs/2310.00582,,,Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs,"Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities
+in various multi-modal tasks. Nevertheless, their performance in fine-grained
+image understanding tasks is still limited. To address this issue, this paper
+proposes a new framework to enhance the fine-grained image understanding
+abilities of MLLMs. Specifically, we present a new method for constructing the
+instruction tuning dataset at a low cost by leveraging annotations in existing
+datasets. A self-consistent bootstrapping method is also introduced to extend
+existing dense object annotations into high-quality
+referring-expression-bounding-box pairs. These methods enable the generation of
+high-quality instruction data which includes a wide range of fundamental
+abilities essential for fine-grained image perception. Moreover, we argue that
+the visual encoder should be tuned during instruction tuning to mitigate the
+gap between full image perception and fine-grained image perception.
+Experimental results demonstrate the superior performance of our method. For
+instance, our model exhibits a 5.2% accuracy improvement over Qwen-VL on GQA
+and surpasses the accuracy of Kosmos-2 by 24.7% on RefCOCO_val. We have also
+attained the top rank on the leaderboard of MMBench. This promising performance
+is achieved by training on only publicly available data, making it easily
+reproducible. The models, datasets, and codes are publicly available at
+https://github.com/SY-Xuan/Pink.",cs.CV,"['cs.CV', 'cs.AI']"
+PoseIRM: Enhance 3D Human Pose Estimation on Unseen  Camera Settings via Invariant Risk Minimization,Yanlu Cai · Weizhong Zhang · Yuan Wu · Cheng Jin, ,https://arxiv.org/abs/2405.05216,,,FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models,"The 3D Human Pose Estimation (3D HPE) task uses 2D images or videos to
+predict human joint coordinates in 3D space. Despite recent advancements in
+deep learning-based methods, they mostly ignore the capability of coupling
+accessible texts and naturally feasible knowledge of humans, missing out on
+valuable implicit supervision to guide the 3D HPE task. Moreover, previous
+efforts often study this task from the perspective of the whole human body,
+neglecting fine-grained guidance hidden in different body parts. To this end,
+we present a new Fine-Grained Prompt-Driven Denoiser based on a diffusion model
+for 3D HPE, named \textbf{FinePOSE}. It consists of three core blocks enhancing
+the reverse process of the diffusion model: (1) Fine-grained Part-aware Prompt
+learning (FPP) block constructs fine-grained part-aware prompts via coupling
+accessible texts and naturally feasible knowledge of body parts with learnable
+prompts to model implicit guidance. (2) Fine-grained Prompt-pose Communication
+(FPC) block establishes fine-grained communications between learned part-aware
+prompts and poses to improve the denoising quality. (3) Prompt-driven Timestamp
+Stylization (PTS) block integrates learned prompt embedding and temporal
+information related to the noise level to enable adaptive adjustment at each
+denoising step. Extensive experiments on public single-human pose estimation
+datasets show that FinePOSE outperforms state-of-the-art methods. We further
+extend FinePOSE to multi-human pose estimation. Achieving 34.3mm average MPJPE
+on the EgoHumans dataset demonstrates the potential of FinePOSE to deal with
+complex multi-human scenarios. Code is available at
+https://github.com/PKU-ICST-MIPL/FinePOSE_CVPR2024.",cs.CV,['cs.CV']
+Transcriptomics-guided Slide Representation Learning in Computational Pathology,Guillaume Jaume · Lukas Oldenburg · Anurag Vaidya · Richard J. Chen · Drew F. K. Williamson · Thomas Peeters · Andrew Song · Faisal Mahmood,https://github.com/mahmoodlab/TANGLE,https://arxiv.org/abs/2405.11618,,2405.11618.pdf,Transcriptomics-guided Slide Representation Learning in Computational Pathology,"Self-supervised learning (SSL) has been successful in building patch
+embeddings of small histology images (e.g., 224x224 pixels), but scaling these
+models to learn slide embeddings from the entirety of giga-pixel whole-slide
+images (WSIs) remains challenging. Here, we leverage complementary information
+from gene expression profiles to guide slide representation learning using
+multimodal pre-training. Expression profiles constitute highly detailed
+molecular descriptions of a tissue that we hypothesize offer a strong
+task-agnostic training signal for learning slide embeddings. Our slide and
+expression (S+E) pre-training strategy, called Tangle, employs
+modality-specific encoders, the outputs of which are aligned via contrastive
+learning. Tangle was pre-trained on samples from three different organs: liver
+(n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different
+species (Homo sapiens and Rattus norvegicus). Across three independent test
+datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver
+WSIs, Tangle shows significantly better few-shot performance compared to
+supervised and SSL baselines. When assessed using prototype-based
+classification and slide retrieval, Tangle also shows a substantial performance
+improvement over all baselines. Code available at
+https://github.com/mahmoodlab/TANGLE.",cs.CV,"['cs.CV', 'cs.AI']"
+Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households,Zhihao Cao · ZiDong Wang · Siwen Xie · Anji Liu · Lifeng Fan,https://github.com/bigai-ai/smart-help,https://arxiv.org/abs/2404.09001,,2404.09001.pdf,Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households,"Despite the significant demand for assistive technology among vulnerable
+groups (e.g., the elderly, children, and the disabled) in daily tasks, research
+into advanced AI-driven assistive solutions that genuinely accommodate their
+diverse needs remains sparse. Traditional human-machine interaction tasks often
+require machines to simply help without nuanced consideration of human
+abilities and feelings, such as their opportunity for practice and learning,
+sense of self-improvement, and self-esteem. Addressing this gap, we define a
+pivotal and novel challenge Smart Help, which aims to provide proactive yet
+adaptive support to human agents with diverse disabilities and dynamic goals in
+various tasks and environments. To establish this challenge, we leverage
+AI2-THOR to build a new interactive 3D realistic household environment for the
+Smart Help task. We introduce an innovative opponent modeling module that
+provides a nuanced understanding of the main agent's capabilities and goals, in
+order to optimize the assisting agent's helping policy. Rigorous experiments
+validate the efficacy of our model components and show the superiority of our
+holistic approach against established baselines. Our findings illustrate the
+potential of AI-imbued assistive robots in improving the well-being of
+vulnerable groups.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV']"
+FocSAM: Delving Deeply into Focused Objects in Segmenting Anything,You Huang · Zongyu Lan · Liujuan Cao · Xianming Lin · Shengchuan Zhang · Guannan Jiang · Rongrong Ji, ,https://arxiv.org/abs/2405.18706,,2405.18706.pdf,FocSAM: Delving Deeply into Focused Objects in Segmenting Anything,"The Segment Anything Model (SAM) marks a notable milestone in segmentation
+models, highlighted by its robust zero-shot capabilities and ability to handle
+diverse prompts. SAM follows a pipeline that separates interactive segmentation
+into image preprocessing through a large encoder and interactive inference via
+a lightweight decoder, ensuring efficient real-time performance. However, SAM
+faces stability issues in challenging samples upon this pipeline. These issues
+arise from two main factors. Firstly, the image preprocessing disables SAM from
+dynamically using image-level zoom-in strategies to refocus on the target
+object during interaction. Secondly, the lightweight decoder struggles to
+sufficiently integrate interactive information with image embeddings. To
+address these two limitations, we propose FocSAM with a pipeline redesigned on
+two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention
+(Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object.
+Dwin-MSA localizes attention computations around the target object, enhancing
+object-related embeddings with minimal computational overhead. Second, we
+propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of
+interactive information from a few initial clicks that have significant impacts
+on the overall segmentation results. Experimentally, FocSAM augments SAM's
+interactive segmentation performance to match the existing state-of-the-art
+method in segmentation quality, requiring only about 5.6% of this method's
+inference time on CPUs.",cs.CV,['cs.CV']
+Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation,Xiaoyang Wang · Huihui Bai · Limin Yu · Yao Zhao · Jimin Xiao, ,https://arxiv.org/abs/2403.06462v2,,2403.06462v2.pdf,Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation,"Semi-supervised semantic segmentation allows model to mine effective
+supervision from unlabeled data to complement label-guided training. Recent
+research has primarily focused on consistency regularization techniques,
+exploring perturbation-invariant training at both the image and feature levels.
+In this work, we proposed a novel feature-level consistency learning framework
+named Density-Descending Feature Perturbation (DDFP). Inspired by the
+low-density separation assumption in semi-supervised learning, our key insight
+is that feature density can shed a light on the most promising direction for
+the segmentation classifier to explore, which is the regions with lower
+density. We propose to shift features with confident predictions towards
+lower-density regions by perturbation injection. The perturbed features are
+then supervised by the predictions on the original features, thereby compelling
+the classifier to explore less dense regions to effectively regularize the
+decision boundary. Central to our method is the estimation of feature density.
+To this end, we introduce a lightweight density estimator based on normalizing
+flow, allowing for efficient capture of the feature density distribution in an
+online manner. By extracting gradients from the density estimator, we can
+determine the direction towards less dense regions for each feature. The
+proposed DDFP outperforms other designs on feature-level perturbations and
+shows state of the art performances on both Pascal VOC and Cityscapes dataset
+under various partition protocols. The project is available at
+https://github.com/Gavinwxy/DDFP.",cs.CV,['cs.CV']
+Rethinking the Region Classification in Open-Vocabulary Semantic Segmentation: An Image-to-Image View,Yuan Wang · Rui Sun · Naisong Luo · Yuwen Pan · Tianzhu Zhang, ,https://arxiv.org/abs/2404.00262,,2404.00262.pdf,Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation,"Open-vocabulary semantic segmentation (OVS) aims to segment images of
+arbitrary categories specified by class labels or captions. However, most
+previous best-performing methods, whether pixel grouping methods or region
+recognition methods, suffer from false matches between image features and
+category labels. We attribute this to the natural gap between the textual
+features and visual features. In this work, we rethink how to mitigate false
+matches from the perspective of image-to-image matching and propose a novel
+relation-aware intra-modal matching (RIM) framework for OVS based on visual
+foundation models. RIM achieves robust region classification by firstly
+constructing diverse image-modal reference features and then matching them with
+region features based on relation-aware ranking distribution. The proposed RIM
+enjoys several merits. First, the intra-modal reference features are better
+aligned, circumventing potential ambiguities that may arise in cross-modal
+matching. Second, the ranking-based matching process harnesses the structure
+information implicit in the inter-class relationships, making it more robust
+than comparing individually. Extensive experiments on three benchmarks
+demonstrate that RIM outperforms previous state-of-the-art methods by large
+margins, obtaining a lead of more than 10% in mIoU on PASCAL VOC benchmark.",cs.CV,['cs.CV']
+Addressing Background Context Bias in Few-Shot Segmentation through Iterative Modulation,Lanyun Zhu · Tianrun Chen · Jianxiong Yin · Simon See · Jun Liu, ,https://arxiv.org/abs/2401.08407,,2401.08407.pdf,Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining,"Cross-Domain Few-Shot Segmentation (CD-FSS) poses the challenge of segmenting
+novel categories from a distinct domain using only limited exemplars. In this
+paper, we undertake a comprehensive study of CD-FSS and uncover two crucial
+insights: (i) the necessity of a fine-tuning stage to effectively transfer the
+learned meta-knowledge across domains, and (ii) the overfitting risk during the
+na\""ive fine-tuning due to the scarcity of novel category examples. With these
+insights, we propose a novel cross-domain fine-tuning strategy that addresses
+the challenging CD-FSS tasks. We first design Bi-directional Few-shot
+Prediction (BFP), which establishes support-query correspondence in a
+bi-directional manner, crafting augmented supervision to reduce the overfitting
+risk. Then we further extend BFP into Iterative Few-shot Adaptor (IFA), which
+is a recursive framework to capture the support-query correspondence
+iteratively, targeting maximal exploitation of supervisory signals from the
+sparse novel category samples. Extensive empirical evaluations show that our
+method significantly outperforms the state-of-the-arts (+7.8\%), which verifies
+that IFA tackles the cross-domain challenges and mitigates the overfitting
+simultaneously. The code is available at: https://github.com/niejiahao1998/IFA.",cs.CV,['cs.CV']
+GeoChat: Grounded Large Vision-Language Model for Remote Sensing,Kartik Kuckreja · Muhammad Sohail Danish · Muzammal Naseer · Abhijit Das · Salman Khan · Fahad Shahbaz Khan, ,https://arxiv.org/abs/2311.15826,,2311.15826.pdf,GeoChat: Grounded Large Vision-Language Model for Remote Sensing,"Recent advancements in Large Vision-Language Models (VLMs) have shown great
+promise in natural image domains, allowing users to hold a dialogue about given
+visual content. However, such general-domain VLMs perform poorly for Remote
+Sensing (RS) scenarios, leading to inaccurate or fabricated information when
+presented with RS domain-specific queries. Such a behavior emerges due to the
+unique challenges introduced by RS imagery. For example, to handle
+high-resolution RS imagery with diverse scale changes across categories and
+many small objects, region-level reasoning is necessary alongside holistic
+scene interpretation. Furthermore, the lack of domain-specific multimodal
+instruction following data as well as strong backbone models for RS make it
+hard for the models to align their behavior with user queries. To address these
+limitations, we propose GeoChat - the first versatile remote sensing VLM that
+offers multitask conversational capabilities with high-resolution RS images.
+Specifically, GeoChat can not only answer image-level queries but also accepts
+region inputs to hold region-specific dialogue. Furthermore, it can visually
+ground objects in its responses by referring to their spatial coordinates. To
+address the lack of domain-specific datasets, we generate a novel RS multimodal
+instruction-following dataset by extending image-text pairs from existing
+diverse RS datasets. We establish a comprehensive benchmark for RS multitask
+conversations and compare with a number of baseline methods. GeoChat
+demonstrates robust zero-shot performance on various RS tasks, e.g., image and
+region captioning, visual question answering, scene classification, visually
+grounded conversations and referring detection. Our code is available at
+https://github.com/mbzuai-oryx/geochat.",cs.CV,"['cs.CV', 'cs.AI']"
+RankMatch: Exploring the Better Consistency Regularization for Semi-supervised Semantic Segmentation,Huayu Mai · Rui Sun · Tianzhu Zhang · Feng Wu, ,https://arxiv.org/abs/2312.08631,,2312.08631.pdf,Semi-supervised Semantic Segmentation Meets Masked Modeling:Fine-grained Locality Learning Matters in Consistency Regularization,"Semi-supervised semantic segmentation aims to utilize limited labeled images
+and abundant unlabeled images to achieve label-efficient learning, wherein the
+weak-to-strong consistency regularization framework, popularized by FixMatch,
+is widely used as a benchmark scheme. Despite its effectiveness, we observe
+that such scheme struggles with satisfactory segmentation for the local
+regions. This can be because it originally stems from the image classification
+task and lacks specialized mechanisms to capture fine-grained local semantics
+that prioritizes in dense prediction. To address this issue, we propose a novel
+framework called \texttt{MaskMatch}, which enables fine-grained locality
+learning to achieve better dense segmentation. On top of the original
+teacher-student framework, we design a masked modeling proxy task that
+encourages the student model to predict the segmentation given the unmasked
+image patches (even with 30\% only) and enforces the predictions to be
+consistent with pseudo-labels generated by the teacher model using the complete
+image. Such design is motivated by the intuition that if the predictions are
+more consistent given insufficient neighboring information, stronger
+fine-grained locality perception is achieved. Besides, recognizing the
+importance of reliable pseudo-labels in the above locality learning and the
+original consistency learning scheme, we design a multi-scale ensembling
+strategy that considers context at different levels of abstraction for
+pseudo-label generation. Extensive experiments on benchmark datasets
+demonstrate the superiority of our method against previous approaches and its
+plug-and-play flexibility.",cs.CV,['cs.CV']
+Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment,Alireza Ganjdanesh · Shangqian Gao · Heng Huang, ,https://arxiv.org/abs/2403.19490,,2403.19490.pdf,Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment,"Structural model pruning is a prominent approach used for reducing the
+computational cost of Convolutional Neural Networks (CNNs) before their
+deployment on resource-constrained devices. Yet, the majority of proposed ideas
+require a pretrained model before pruning, which is costly to secure. In this
+paper, we propose a novel structural pruning approach to jointly learn the
+weights and structurally prune architectures of CNN models. The core element of
+our method is a Reinforcement Learning (RL) agent whose actions determine the
+pruning ratios of the CNN model's layers, and the resulting model's accuracy
+serves as its reward. We conduct the joint training and pruning by iteratively
+training the model's weights and the agent's policy, and we regularize the
+model's weights to align with the selected structure by the agent. The evolving
+model's weights result in a dynamic reward function for the agent, which
+prevents using prominent episodic RL methods with stationary environment
+assumption for our purpose. We address this challenge by designing a mechanism
+to model the complex changing dynamics of the reward function and provide a
+representation of it to the RL agent. To do so, we take a learnable embedding
+for each training epoch and employ a recurrent model to calculate a
+representation of the changing environment. We train the recurrent model and
+embeddings using a decoder model to reconstruct observed rewards. Such a design
+empowers our agent to effectively leverage episodic observations along with the
+environment representations to learn a proper policy to determine performant
+sub-networks of the CNN model. Our extensive experiments on CIFAR-10 and
+ImageNet using ResNets and MobileNets demonstrate the effectiveness of our
+method.",cs.CV,['cs.CV']
+WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion,Soyong Shin · Juyong Kim · Eni Halilaj · Michael J. Black,https://wham.is.tue.mpg.de/,https://arxiv.org/abs/2312.07531,,2312.07531.pdf,WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion,"The estimation of 3D human motion from video has progressed rapidly but
+current methods still have several key limitations. First, most methods
+estimate the human in camera coordinates. Second, prior work on estimating
+humans in global coordinates often assumes a flat ground plane and produces
+foot sliding. Third, the most accurate methods rely on computationally
+expensive optimization pipelines, limiting their use to offline applications.
+Finally, existing video-based methods are surprisingly less accurate than
+single-frame methods. We address these limitations with WHAM (World-grounded
+Humans with Accurate Motion), which accurately and efficiently reconstructs 3D
+human motion in a global coordinate system from video. WHAM learns to lift 2D
+keypoint sequences to 3D using motion capture data and fuses this with video
+features, integrating motion context and visual information. WHAM exploits
+camera angular velocity estimated from a SLAM method together with human motion
+to estimate the body's global trajectory. We combine this with a contact-aware
+trajectory refinement method that lets WHAM capture human motion in diverse
+conditions, such as climbing stairs. WHAM outperforms all existing 3D human
+motion recovery methods across multiple in-the-wild benchmarks. Code will be
+available for research purposes at http://wham.is.tue.mpg.de/",cs.CV,['cs.CV']
+StrokeFaceNeRF: Stroke-based Facial Appearance Editing in Neural Radiance Field,Xiao-juan Li · Dingxi Zhang · Shu-Yu Chen · Feng-Lin Liu, ,https://arxiv.org/abs/2312.09913,,,LAENeRF: Local Appearance Editing for Neural Radiance Fields,"Due to the omnipresence of Neural Radiance Fields (NeRFs), the interest
+towards editable implicit 3D representations has surged over the last years.
+However, editing implicit or hybrid representations as used for NeRFs is
+difficult due to the entanglement of appearance and geometry encoded in the
+model parameters. Despite these challenges, recent research has shown first
+promising steps towards photorealistic and non-photorealistic appearance edits.
+The main open issues of related work include limited interactivity, a lack of
+support for local edits and large memory requirements, rendering them less
+useful in practice. We address these limitations with LAENeRF, a unified
+framework for photorealistic and non-photorealistic appearance editing of
+NeRFs. To tackle local editing, we leverage a voxel grid as starting point for
+region selection. We learn a mapping from expected ray terminations to final
+output color, which can optionally be supervised by a style loss, resulting in
+a framework which can perform photorealistic and non-photorealistic appearance
+editing of selected regions. Relying on a single point per ray for our mapping,
+we limit memory requirements and enable fast optimization. To guarantee
+interactivity, we compose the output color using a set of learned, modifiable
+base colors, composed with additive layer mixing. Compared to concurrent work,
+LAENeRF enables recoloring and stylization while keeping processing time low.
+Furthermore, we demonstrate that our approach surpasses baseline methods both
+quantitatively and qualitatively.",cs.CV,['cs.CV']
+Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D,Mukund Varma T · Peihao Wang · Zhiwen Fan · Zhangyang Wang · Hao Su · Ravi Ramamoorthi,https://mukundvarmat.github.io/Lift3D/,https://arxiv.org/abs/2403.18922,,2403.18922.pdf,Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D,"In recent years, there has been an explosion of 2D vision models for numerous
+tasks such as semantic segmentation, style transfer or scene editing, enabled
+by large-scale 2D image datasets. At the same time, there has been renewed
+interest in 3D scene representations such as neural radiance fields from
+multi-view images. However, the availability of 3D or multiview data is still
+substantially limited compared to 2D image datasets, making extending 2D vision
+models to 3D data highly desirable but also very challenging. Indeed, extending
+a single 2D vision operator like scene editing to 3D typically requires a
+highly creative method specialized to that task and often requires per-scene
+optimization. In this paper, we ask the question of whether any 2D vision model
+can be lifted to make 3D consistent predictions. We answer this question in the
+affirmative; our new Lift3D method trains to predict unseen views on feature
+spaces generated by a few visual models (i.e. DINO and CLIP), but then
+generalizes to novel vision operators and tasks, such as style transfer,
+super-resolution, open vocabulary segmentation and image colorization; for some
+of these tasks, there is no comparable previous 3D method. In many cases, we
+even outperform state-of-the-art methods specialized for the task in question.
+Moreover, Lift3D is a zero-shot method, in the sense that it requires no
+task-specific training, nor scene-specific optimization.",cs.CV,['cs.CV']
+Querying as Prompt: Parameter-Efficient Learning for Multimodal Language Model,Tian Liang · Jing Huang · Ming Kong · Luyuan Chen · Qiang Zhu, ,https://arxiv.org/html/2405.20654v1,,2405.20654v1.pdf,Passage-specific Prompt Tuning for Passage Reranking in Question Answering with Large Language Models,"Effective passage retrieval and reranking methods have been widely utilized
+to identify suitable candidates in open-domain question answering tasks, recent
+studies have resorted to LLMs for reranking the retrieved passages by the
+log-likelihood of the question conditioned on each passage. Although these
+methods have demonstrated promising results, the performance is notably
+sensitive to the human-written prompt (or hard prompt), and fine-tuning LLMs
+can be computationally intensive and time-consuming. Furthermore, this approach
+limits the leverage of question-passage relevance pairs and passage-specific
+knowledge to enhance the ranking capabilities of LLMs. In this paper, we
+propose passage-specific prompt tuning for reranking in open-domain question
+answering (PSPT): a parameter-efficient method that fine-tunes learnable
+passage-specific soft prompts, incorporating passage-specific knowledge from a
+limited set of question-passage relevance pairs. The method involves ranking
+retrieved passages based on the log-likelihood of the model generating the
+question conditioned on each passage and the learned soft prompt. We conducted
+extensive experiments utilizing the Llama-2-chat-7B model across three publicly
+available open-domain question answering datasets and the results demonstrate
+the effectiveness of the proposed approach.",cs.CL,"['cs.CL', 'cs.IR']"
+PairDETR : Joint Detection and Association of Human Bodies and Faces,Ammar Ali · Georgii Gaikov · Denis Rybalchenko · Alexander Chigorin · Ivan Laptev · Sergey Zagoruyko, ,https://arxiv.org/abs/2404.08450,,2404.08450.pdf,Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues,"Face recognition systems are frequently subjected to a variety of physical
+and digital attacks of different types. Previous methods have achieved
+satisfactory performance in scenarios that address physical attacks and digital
+attacks, respectively. However, few methods are considered to integrate a model
+that simultaneously addresses both physical and digital attacks, implying the
+necessity to develop and maintain multiple models. To jointly detect physical
+and digital attacks within a single model, we propose an innovative approach
+that can adapt to any network architecture. Our approach mainly contains two
+types of data augmentation, which we call Simulated Physical Spoofing Clues
+augmentation (SPSC) and Simulated Digital Spoofing Clues augmentation (SDSC).
+SPSC and SDSC augment live samples into simulated attack samples by simulating
+spoofing clues of physical and digital attacks, respectively, which
+significantly improve the capability of the model to detect ""unseen"" attack
+types. Extensive experiments show that SPSC and SDSC can achieve
+state-of-the-art generalization in Protocols 2.1 and 2.2 of the UniAttackData
+dataset, respectively. Our method won first place in ""Unified Physical-Digital
+Face Attack Detection"" of the 5th Face Anti-spoofing Challenge@CVPR2024. Our
+final submission obtains 3.75% APCER, 0.93% BPCER, and 2.34% ACER,
+respectively. Our code is available at
+https://github.com/Xianhua-He/cvpr2024-face-anti-spoofing-challenge.",cs.CV,['cs.CV']
+Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis,Zicheng Zhang · RUOBING ZHENG · Bonan Li · Congying Han · Tianqi Li · Meng Wang · Tiande Guo · Jingdong Chen · Ziwen Liu · Ming Yang, ,https://arxiv.org/abs/2402.17364v1,,2402.17364v1.pdf,Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis,"Recent works in implicit representations, such as Neural Radiance Fields
+(NeRF), have advanced the generation of realistic and animatable head avatars
+from video sequences. These implicit methods are still confronted by visual
+artifacts and jitters, since the lack of explicit geometric constraints poses a
+fundamental challenge in accurately modeling complex facial deformations. In
+this paper, we introduce Dynamic Tetrahedra (DynTet), a novel hybrid
+representation that encodes explicit dynamic meshes by neural networks to
+ensure geometric consistency across various motions and viewpoints. DynTet is
+parameterized by the coordinate-based networks which learn signed distance,
+deformation, and material texture, anchoring the training data into a
+predefined tetrahedra grid. Leveraging Marching Tetrahedra, DynTet efficiently
+decodes textured meshes with a consistent topology, enabling fast rendering
+through a differentiable rasterizer and supervision via a pixel loss. To
+enhance training efficiency, we incorporate classical 3D Morphable Models to
+facilitate geometry learning and define a canonical space for simplifying
+texture learning. These advantages are readily achievable owing to the
+effective geometric representation employed in DynTet. Compared with prior
+works, DynTet demonstrates significant improvements in fidelity, lip
+synchronization, and real-time performance according to various metrics. Beyond
+producing stable and visually appealing synthesis videos, our method also
+outputs the dynamic meshes which is promising to enable many emerging
+applications.",cs.CV,['cs.CV']
+SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering,Antoine Guédon · Vincent Lepetit,https://anttwo.github.io/sugar/,,https://huggingface.co/papers/2311.12775,,,,,nan
+Composed Video Retrieval via Enriched Context and Discriminative Embeddings,Omkar Thawakar · Muzammal Naseer · Rao Anwer · Salman Khan · Michael Felsberg · Mubarak Shah · Fahad Shahbaz Khan, ,https://arxiv.org/abs/2403.16997,,2403.16997.pdf,Composed Video Retrieval via Enriched Context and Discriminative Embeddings,"Composed video retrieval (CoVR) is a challenging problem in computer vision
+which has recently highlighted the integration of modification text with visual
+queries for more sophisticated video search in large databases. Existing works
+predominantly rely on visual queries combined with modification text to
+distinguish relevant videos. However, such a strategy struggles to fully
+preserve the rich query-specific context in retrieved target videos and only
+represents the target video using visual embedding. We introduce a novel CoVR
+framework that leverages detailed language descriptions to explicitly encode
+query-specific contextual information and learns discriminative embeddings of
+vision only, text only and vision-text for better alignment to accurately
+retrieve matched target videos. Our proposed framework can be flexibly employed
+for both composed video (CoVR) and image (CoIR) retrieval tasks. Experiments on
+three datasets show that our approach obtains state-of-the-art performance for
+both CovR and zero-shot CoIR tasks, achieving gains as high as around 7% in
+terms of recall@K=1 score. Our code, models, detailed language descriptions for
+WebViD-CoVR dataset are available at
+\url{https://github.com/OmkarThawakar/composed-video-retrieval}",cs.CV,['cs.CV']
+Distilling Vision-Language Models on Millions of Videos,Yue Zhao · Long Zhao · Xingyi Zhou · Jialin Wu · Chun-Te Chu · Hui Miao · Florian Schroff · Hartwig Adam · Ting Liu · Boqing Gong · Philipp Krähenbühl · Liangzhe Yuan, ,https://arxiv.org/abs/2401.06129,,2401.06129.pdf,Distilling Vision-Language Models on Millions of Videos,"The recent advance in vision-language models is largely attributed to the
+abundance of image-text data. We aim to replicate this success for
+video-language models, but there simply is not enough human-curated video-text
+data available. We thus resort to fine-tuning a video-language model from a
+strong image-language baseline with synthesized instructional data. The
+resulting video model by video-instruction-tuning (VIIT) is then used to
+auto-label millions of videos to generate high-quality captions. We show the
+adapted video-language model performs well on a wide range of video-language
+benchmarks. For instance, it surpasses the best prior result on open-ended
+NExT-QA by 2.8%. Besides, our model generates detailed descriptions for
+previously unseen videos, which provide better textual supervision than
+existing methods. Experiments show that a video-language dual-encoder model
+contrastively trained on these auto-generated captions is 3.8% better than the
+strongest baseline that also leverages vision-language models. Our best model
+outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video
+retrieval by 6%. As a side product, we generate the largest video caption
+dataset to date.",cs.CV,['cs.CV']
+ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing,Junkun Chen · Samuel Rota Bulò · Norman Müller · Lorenzo Porzi · Peter Kontschieder · Yu-Xiong Wang, ,https://arxiv.org/abs/2308.13223,,,EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior,"While image diffusion models have made significant progress in text-driven 3D
+content creation, they often fail to accurately capture the intended meaning of
+text prompts, especially for view information. This limitation leads to the
+Janus problem, where multi-faced 3D models are generated under the guidance of
+such diffusion models. In this paper, we propose a robust high-quality 3D
+content generation pipeline by exploiting orthogonal-view image guidance.
+First, we introduce a novel 2D diffusion model that generates an image
+consisting of four orthogonal-view sub-images based on the given text prompt.
+Then, the 3D content is created using this diffusion model. Notably, the
+generated orthogonal-view image provides strong geometric structure priors and
+thus improves 3D consistency. As a result, it effectively resolves the Janus
+problem and significantly enhances the quality of 3D content creation.
+Additionally, we present a 3D synthesis fusion network that can further improve
+the details of the generated 3D contents. Both quantitative and qualitative
+evaluations demonstrate that our method surpasses previous text-to-3D
+techniques. Project page: https://efficientdreamer.github.io.",cs.CV,['cs.CV']
+Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness,Guangzhi Wang · Yangyang Guo · Ziwei Xu · Mohan Kankanhalli, ,https://arxiv.org/abs/2307.10499,,,Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detection,"Human-Object Interaction Detection is a crucial aspect of human-centric scene
+understanding, with important applications in various domains. Despite recent
+progress in this field, recognizing subtle and detailed interactions remains
+challenging. Existing methods try to use human-related clues to alleviate the
+difficulty, but rely heavily on external annotations or knowledge, limiting
+their practical applicability in real-world scenarios. In this work, we propose
+a novel Part Semantic Network (PSN) to solve this problem. The core of PSN is a
+Conditional Part Attention (CPA) mechanism, where human features are taken as
+keys and values, and the object feature is used as query for the computation in
+a cross-attention mechanism. In this way, our model learns to automatically
+focus on the most informative human parts conditioned on the involved object,
+generating more semantically meaningful features for interaction recognition.
+Additionally, we propose an Occluded Part Extrapolation (OPE) strategy to
+facilitate interaction recognition under occluded scenarios, which teaches the
+model to extrapolate detailed features from partially occluded ones. Our method
+consistently outperforms prior approaches on the V-COCO and HICO-DET datasets,
+without external data or extra annotations. Additional ablation studies
+validate the effectiveness of each component of our proposed method.",cs.CV,['cs.CV']
+Multi-modal learning for geospatial vegetation forecasting,Vitus Benson · Claire Robin · Christian Requena-Mesa · LAZARO ALONSO SILVA · Mélanie Weynants · Nora Linscheid · Jose Cortes · Zhihan Gao · Nuno Carvalhais · Markus Reichstein, ,https://arxiv.org/html/2405.20161v1,,2405.20161v1.pdf,Landslide mapping from Sentinel-2 imagery through change detection,"Landslides are one of the most critical and destructive geohazards.
+Widespread development of human activities and settlements combined with the
+effects of climate change on weather are resulting in a high increase in the
+frequency and destructive power of landslides, making them a major threat to
+human life and the economy. In this paper, we explore methodologies to map
+newly-occurred landslides using Sentinel-2 imagery automatically. All
+approaches presented are framed as a bi-temporal change detection problem,
+requiring only a pair of Sentinel-2 images, taken respectively before and after
+a landslide-triggering event. Furthermore, we introduce a novel deep learning
+architecture for fusing Sentinel-2 bi-temporal image pairs with Digital
+Elevation Model (DEM) data, showcasing its promising performances w.r.t. other
+change detection models in the literature. As a parallel task, we address
+limitations in existing datasets by creating a novel geodatabase, which
+includes manually validated open-access landslide inventories over
+heterogeneous ecoregions of the world. We release both code and dataset with an
+open-source license.",cs.CV,"['cs.CV', 'eess.IV']"
+LISA: Reasoning Segmentation via Large Language Model,Xin Lai · Zhuotao Tian · Yukang Chen · Yanwei Li · Yuhui Yuan · Shu Liu · Jiaya Jia, ,https://arxiv.org/abs/2308.00692,,2308.00692.pdf,LISA: Reasoning Segmentation via Large Language Model,"Although perception systems have made remarkable advancements in recent
+years, they still rely on explicit human instruction or pre-defined categories
+to identify the target objects before executing visual recognition tasks. Such
+systems cannot actively reason and comprehend implicit user intention. In this
+work, we propose a new segmentation task -- reasoning segmentation. The task is
+designed to output a segmentation mask given a complex and implicit query text.
+Furthermore, we establish a benchmark comprising over one thousand
+image-instruction-mask data samples, incorporating intricate reasoning and
+world knowledge for evaluation purposes. Finally, we present LISA: large
+Language Instructed Segmentation Assistant, which inherits the language
+generation capabilities of multimodal Large Language Models (LLMs) while also
+possessing the ability to produce segmentation masks. We expand the original
+vocabulary with a <SEG> token and propose the embedding-as-mask paradigm to
+unlock the segmentation capability. Remarkably, LISA can handle cases involving
+complex reasoning and world knowledge. Also, it demonstrates robust zero-shot
+capability when trained exclusively on reasoning-free datasets. In addition,
+fine-tuning the model with merely 239 reasoning segmentation data samples
+results in further performance enhancement. Both quantitative and qualitative
+experiments show our method effectively unlocks new reasoning segmentation
+capabilities for multimodal LLMs. Code, models, and data are available at
+https://github.com/dvlab-research/LISA.",cs.CV,['cs.CV']
+Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion,Linzhan Mou · Junkun Chen · Yu-Xiong Wang, ,https://arxiv.org/abs/2306.09551,,2306.09551.pdf,Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model,"Recent research has demonstrated that the combination of pretrained diffusion
+models with neural radiance fields (NeRFs) has emerged as a promising approach
+for text-to-3D generation. Simply coupling NeRF with diffusion models will
+result in cross-view inconsistency and degradation of stylized view syntheses.
+To address this challenge, we propose the Edit-DiffNeRF framework, which is
+composed of a frozen diffusion model, a proposed delta module to edit the
+latent semantic space of the diffusion model, and a NeRF. Instead of training
+the entire diffusion for each scene, our method focuses on editing the latent
+semantic space in frozen pretrained diffusion models by the delta module. This
+fundamental change to the standard diffusion framework enables us to make
+fine-grained modifications to the rendered views and effectively consolidate
+these instructions in a 3D scene via NeRF training. As a result, we are able to
+produce an edited 3D scene that faithfully aligns to input text instructions.
+Furthermore, to ensure semantic consistency across different viewpoints, we
+propose a novel multi-view semantic consistency loss that extracts a latent
+semantic embedding from the input view as a prior, and aim to reconstruct it in
+different views. Our proposed method has been shown to effectively edit
+real-world 3D scenes, resulting in 25% improvement in the alignment of the
+performed 3D edits with text instructions compared to prior work.",cs.CV,['cs.CV']
+RepAn: Enhanced Annealing through Re-parameterization,Xiang Fei · Xiawu Zheng · Yan Wang · Fei Chao · Chenglin Wu · Liujuan Cao, ,,https://dilithjay.com/blog/the-reparameterization-trick-clearly-explained,,,,,nan
+EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything,Yunyang Xiong · Balakrishnan Varadarajan · Lemeng Wu · Xiaoyu Xiang · Fanyi Xiao · Chenchen Zhu · Xiaoliang Dai · Dilin Wang · Fei Sun · Forrest Iandola · Raghuraman Krishnamoorthi · Vikas Chandra, ,https://arxiv.org/abs/2312.00863,,2312.00863.pdf,EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything,"Segment Anything Model (SAM) has emerged as a powerful tool for numerous
+vision applications. A key component that drives the impressive performance for
+zero-shot transfer and high versatility is a super large Transformer model
+trained on the extensive high-quality SA-1B dataset. While beneficial, the huge
+computation cost of SAM model has limited its applications to wider real-world
+applications. To address this limitation, we propose EfficientSAMs,
+light-weight SAM models that exhibits decent performance with largely reduced
+complexity. Our idea is based on leveraging masked image pretraining, SAMI,
+which learns to reconstruct features from SAM image encoder for effective
+visual representation learning. Further, we take SAMI-pretrained light-weight
+image encoders and mask decoder to build EfficientSAMs, and finetune the models
+on SA-1B for segment anything task. We perform evaluations on multiple vision
+tasks including image classification, object detection, instance segmentation,
+and semantic object detection, and find that our proposed pretraining method,
+SAMI, consistently outperforms other masked image pretraining methods. On
+segment anything task such as zero-shot instance segmentation, our
+EfficientSAMs with SAMI-pretrained lightweight image encoders perform favorably
+with a significant gain (e.g., ~4 AP on COCO/LVIS) over other fast SAM models.",cs.CV,['cs.CV']
+Strong Transferable Adversarial Attacks via Ensembled Asymptotically Normal Distribution Learning,Zhengwei Fang · Rui Wang · Tao Huang · Liping Jing, ,https://arxiv.org/abs/2308.02897,,2308.02897.pdf,An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability,"While the transferability property of adversarial examples allows the
+adversary to perform black-box attacks (i.e., the attacker has no knowledge
+about the target model), the transfer-based adversarial attacks have gained
+great attention. Previous works mostly study gradient variation or image
+transformations to amplify the distortion on critical parts of inputs. These
+methods can work on transferring across models with limited differences, i.e.,
+from CNNs to CNNs, but always fail in transferring across models with wide
+differences, such as from CNNs to ViTs. Alternatively, model ensemble
+adversarial attacks are proposed to fuse outputs from surrogate models with
+diverse architectures to get an ensemble loss, making the generated adversarial
+example more likely to transfer to other models as it can fool multiple models
+concurrently. However, existing ensemble attacks simply fuse the outputs of the
+surrogate models evenly, thus are not efficacious to capture and amplify the
+intrinsic transfer information of adversarial examples. In this paper, we
+propose an adaptive ensemble attack, dubbed AdaEA, to adaptively control the
+fusion of the outputs from each model, via monitoring the discrepancy ratio of
+their contributions towards the adversarial objective. Furthermore, an extra
+disparity-reduced filter is introduced to further synchronize the update
+direction. As a result, we achieve considerable improvement over the existing
+ensemble attacks on various datasets, and the proposed AdaEA can also boost
+existing transfer-based attacks, which further demonstrates its efficacy and
+versatility.",cs.CV,['cs.CV']
+AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning,Duojun Huang · Xinyu Xiong · Jie Ma · Jichang Li · Zequn Jie · Lin Ma · Guanbin Li, ,https://arxiv.org/abs/2312.03628,,2312.03628.pdf,Boosting Segment Anything Model Towards Open-Vocabulary Learning,"The recent Segment Anything Model (SAM) has emerged as a new paradigmatic
+vision foundation model, showcasing potent zero-shot generalization and
+flexible prompting. Despite SAM finding applications and adaptations in various
+domains, its primary limitation lies in the inability to grasp object
+semantics. In this paper, we present Sambor to seamlessly integrate SAM with
+the open-vocabulary object detector in an end-to-end framework. While retaining
+all the remarkable capabilities inherent to SAM, we enhance it with the
+capacity to detect arbitrary objects based on human inputs like category names
+or reference expressions. To accomplish this, we introduce a novel SideFormer
+module that extracts SAM features to facilitate zero-shot object localization
+and inject comprehensive semantic information for open-vocabulary recognition.
+In addition, we devise an open-set region proposal network (Open-set RPN),
+enabling the detector to acquire the open-set proposals generated by SAM.
+Sambor demonstrates superior zero-shot performance across benchmarks, including
+COCO and LVIS, proving highly competitive against previous SoTA methods. We
+aspire for this work to serve as a meaningful endeavor in endowing SAM to
+recognize diverse object categories and advancing open-vocabulary learning with
+the support of vision foundation models.",cs.CV,['cs.CV']
+Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering,Zaid Khan · Yun Fu, ,https://arxiv.org/abs/2404.10193,,2404.10193.pdf,Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering,"The goal of selective prediction is to allow an a model to abstain when it
+may not be able to deliver a reliable prediction, which is important in
+safety-critical contexts. Existing approaches to selective prediction typically
+require access to the internals of a model, require retraining a model or study
+only unimodal models. However, the most powerful models (e.g. GPT-4) are
+typically only available as black boxes with inaccessible internals, are not
+retrainable by end-users, and are frequently used for multimodal tasks. We
+study the possibility of selective prediction for vision-language models in a
+realistic, black-box setting. We propose using the principle of
+\textit{neighborhood consistency} to identify unreliable responses from a
+black-box vision-language model in question answering tasks. We hypothesize
+that given only a visual question and model response, the consistency of the
+model's responses over the neighborhood of a visual question will indicate
+reliability. It is impossible to directly sample neighbors in feature space in
+a black-box setting. Instead, we show that it is possible to use a smaller
+proxy model to approximately sample from the neighborhood. We find that
+neighborhood consistency can be used to identify model responses to visual
+questions that are likely unreliable, even in adversarial settings or settings
+that are out-of-distribution to the proxy model.",cs.CV,['cs.CV']
+Distribution-aware Knowledge Prototyping for Non-exemplar Lifelong Person Re-identification,Kunlun Xu · Xu Zou · Yuxin Peng · Jiahuan Zhou, ,https://arxiv.org/abs/2405.19005,,2405.19005.pdf,Auto-selected Knowledge Adapters for Lifelong Person Re-identification,"Lifelong Person Re-Identification (LReID) extends traditional ReID by
+requiring systems to continually learn from non-overlapping datasets across
+different times and locations, adapting to new identities while preserving
+knowledge of previous ones. Existing approaches, either rehearsal-free or
+rehearsal-based, still suffer from the problem of catastrophic forgetting since
+they try to cram diverse knowledge into one fixed model. To overcome this
+limitation, we introduce a novel framework AdalReID, that adopts knowledge
+adapters and a parameter-free auto-selection mechanism for lifelong learning.
+Concretely, we incrementally build distinct adapters to learn domain-specific
+knowledge at each step, which can effectively learn and preserve knowledge
+across different datasets. Meanwhile, the proposed auto-selection strategy
+adaptively calculates the knowledge similarity between the input set and the
+adapters. On the one hand, the appropriate adapters are selected for the inputs
+to process ReID, and on the other hand, the knowledge interaction and fusion
+between adapters are enhanced to improve the generalization ability of the
+model. Extensive experiments are conducted to demonstrate the superiority of
+our AdalReID, which significantly outperforms SOTAs by about 10$\sim$20\% mAP
+on both seen and unseen domains.",cs.CV,['cs.CV']
+Looking 3D: Anomaly Detection with 2D-3D Alignment,Ankan Kumar Bhunia · Changjian Li · Hakan Bilen,https://github.com/VICO-UoE/Looking3D,https://arxiv.org/abs/2311.14897,,2311.14897.pdf,Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network,"Recently, 3D anomaly detection, a crucial problem involving fine-grained
+geometry discrimination, is getting more attention. However, the lack of
+abundant real 3D anomaly data limits the scalability of current models. To
+enable scalable anomaly data collection, we propose a 3D anomaly synthesis
+pipeline to adapt existing large-scale 3Dmodels for 3D anomaly detection.
+Specifically, we construct a synthetic dataset, i.e., Anomaly-ShapeNet, basedon
+ShapeNet. Anomaly-ShapeNet consists of 1600 point cloud samples under 40
+categories, which provides a rich and varied collection of data, enabling
+efficient training and enhancing adaptability to industrial scenarios.
+Meanwhile,to enable scalable representation learning for 3D anomaly
+localization, we propose a self-supervised method, i.e., Iterative Mask
+Reconstruction Network (IMRNet). During training, we propose a geometry-aware
+sample module to preserve potentially anomalous local regions during point
+cloud down-sampling. Then, we randomly mask out point patches and sent the
+visible patches to a transformer for reconstruction-based self-supervision.
+During testing, the point cloud repeatedly goes through the Mask Reconstruction
+Network, with each iteration's output becoming the next input. By merging and
+contrasting the final reconstructed point cloud with the initial input, our
+method successfully locates anomalies. Experiments show that IMRNet outperforms
+previous state-of-the-art methods, achieving 66.1% in I-AUC on Anomaly-ShapeNet
+dataset and 72.5% in I-AUC on Real3D-AD dataset. Our dataset will be released
+at https://github.com/Chopper-233/Anomaly-ShapeNet",cs.CV,['cs.CV']
+Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network,wenqiao Li · Xiaohao Xu · Yao Gu · BoZhong Zheng · Shenghua Gao · Yingna Wu, ,https://arxiv.org/abs/2311.14897,,,Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network,"Recently, 3D anomaly detection, a crucial problem involving fine-grained
+geometry discrimination, is getting more attention. However, the lack of
+abundant real 3D anomaly data limits the scalability of current models. To
+enable scalable anomaly data collection, we propose a 3D anomaly synthesis
+pipeline to adapt existing large-scale 3Dmodels for 3D anomaly detection.
+Specifically, we construct a synthetic dataset, i.e., Anomaly-ShapeNet, basedon
+ShapeNet. Anomaly-ShapeNet consists of 1600 point cloud samples under 40
+categories, which provides a rich and varied collection of data, enabling
+efficient training and enhancing adaptability to industrial scenarios.
+Meanwhile,to enable scalable representation learning for 3D anomaly
+localization, we propose a self-supervised method, i.e., Iterative Mask
+Reconstruction Network (IMRNet). During training, we propose a geometry-aware
+sample module to preserve potentially anomalous local regions during point
+cloud down-sampling. Then, we randomly mask out point patches and sent the
+visible patches to a transformer for reconstruction-based self-supervision.
+During testing, the point cloud repeatedly goes through the Mask Reconstruction
+Network, with each iteration's output becoming the next input. By merging and
+contrasting the final reconstructed point cloud with the initial input, our
+method successfully locates anomalies. Experiments show that IMRNet outperforms
+previous state-of-the-art methods, achieving 66.1% in I-AUC on Anomaly-ShapeNet
+dataset and 72.5% in I-AUC on Real3D-AD dataset. Our dataset will be released
+at https://github.com/Chopper-233/Anomaly-ShapeNet",cs.CV,['cs.CV']
+Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,Lihe Yang · Bingyi Kang · Zilong Huang · Xiaogang Xu · Jiashi Feng · Hengshuang Zhao, ,https://arxiv.org/abs/2401.10891,,2401.10891.pdf,Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,"This work presents Depth Anything, a highly practical solution for robust
+monocular depth estimation. Without pursuing novel technical modules, we aim to
+build a simple yet powerful foundation model dealing with any images under any
+circumstances. To this end, we scale up the dataset by designing a data engine
+to collect and automatically annotate large-scale unlabeled data (~62M), which
+significantly enlarges the data coverage and thus is able to reduce the
+generalization error. We investigate two simple yet effective strategies that
+make data scaling-up promising. First, a more challenging optimization target
+is created by leveraging data augmentation tools. It compels the model to
+actively seek extra visual knowledge and acquire robust representations.
+Second, an auxiliary supervision is developed to enforce the model to inherit
+rich semantic priors from pre-trained encoders. We evaluate its zero-shot
+capabilities extensively, including six public datasets and randomly captured
+photos. It demonstrates impressive generalization ability. Further, through
+fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs
+are set. Our better depth model also results in a better depth-conditioned
+ControlNet. Our models are released at
+https://github.com/LiheYoung/Depth-Anything.",cs.CV,['cs.CV']
+SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World,Kiana Ehsani · Tanmay Gupta · Rose Hendrix · Jordi Salvador · Luca Weihs · Kuo-Hao Zeng · Kunal Singh Singh · Yejin Kim · Winson Han · Alvaro Herrasti · Ranjay Krishna · Dustin Schwenk · Eli VanderBilt · Aniruddha Kembhavi, ,https://arxiv.org/abs/2312.02976,,2312.02976.pdf,Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World,"Reinforcement learning (RL) with dense rewards and imitation learning (IL)
+with human-generated trajectories are the most widely used approaches for
+training modern embodied agents. RL requires extensive reward shaping and
+auxiliary losses and is often too slow and ineffective for long-horizon tasks.
+While IL with human supervision is effective, collecting human trajectories at
+scale is extremely expensive. In this work, we show that imitating
+shortest-path planners in simulation produces agents that, given a language
+instruction, can proficiently navigate, explore, and manipulate objects in both
+simulation and in the real world using only RGB sensors (no depth map or GPS
+coordinates). This surprising result is enabled by our end-to-end,
+transformer-based, SPOC architecture, powerful visual encoders paired with
+extensive image augmentation, and the dramatic scale and diversity of our
+training data: millions of frames of shortest-path-expert trajectories
+collected inside approximately 200,000 procedurally generated houses containing
+40,000 unique 3D assets. Our models, data, training code, and newly proposed
+10-task benchmarking suite CHORES will be open-sourced.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV']"
+A Unified and Interpretable Emotion Representation and Expression Generation,Reni Paskaleva · Mykyta Holubakha · Andela Ilic · Saman Motamed · Luc Van Gool · Danda Paudel,https://emotion-diffusion.github.io/,https://arxiv.org/abs/2404.01243,,2404.01243.pdf,A Unified and Interpretable Emotion Representation and Expression Generation,"Canonical emotions, such as happy, sad, and fearful, are easy to understand
+and annotate. However, emotions are often compound, e.g. happily surprised, and
+can be mapped to the action units (AUs) used for expressing emotions, and
+trivially to the canonical ones. Intuitively, emotions are continuous as
+represented by the arousal-valence (AV) model. An interpretable unification of
+these four modalities - namely, Canonical, Compound, AUs, and AV - is highly
+desirable, for a better representation and understanding of emotions. However,
+such unification remains to be unknown in the current literature. In this work,
+we propose an interpretable and unified emotion model, referred as C2A2. We
+also develop a method that leverages labels of the non-unified models to
+annotate the novel unified one. Finally, we modify the text-conditional
+diffusion models to understand continuous numbers, which are then used to
+generate continuous expressions using our unified emotion model. Through
+quantitative and qualitative experiments, we show that our generated images are
+rich and capture subtle expressions. Our work allows a fine-grained generation
+of expressions in conjunction with other textual inputs and offers a new label
+space for emotions at the same time.",cs.CV,['cs.CV']
+Regularized Parameter Uncertainty for Improving Generalization in Reinforcement Learning,Pehuen Moure · Longbiao Cheng · Joachim Ott · Zuowen Wang · Shih-Chii Liu, ,,https://arxiv.org/pdf/2207.02016v4,,,,,nan
+Understanding Video Transfomers via Universal Concept Discovery,Matthew Kowal · Achal Dave · Rares Andrei Ambrus · Adrien Gaidon · Kosta Derpanis · Pavel Tokmakov,https://yorkucvil.github.io/VTCD/,https://arxiv.org/abs/2401.10831,,,Understanding Video Transformers via Universal Concept Discovery,"This paper studies the problem of concept-based interpretability of
+transformer representations for videos. Concretely, we seek to explain the
+decision-making process of video transformers based on high-level,
+spatiotemporal concepts that are automatically discovered. Prior research on
+concept-based interpretability has concentrated solely on image-level tasks.
+Comparatively, video models deal with the added temporal dimension, increasing
+complexity and posing challenges in identifying dynamic concepts over time. In
+this work, we systematically address these challenges by introducing the first
+Video Transformer Concept Discovery (VTCD) algorithm. To this end, we propose
+an efficient approach for unsupervised identification of units of video
+transformer representations - concepts, and ranking their importance to the
+output of a model. The resulting concepts are highly interpretable, revealing
+spatio-temporal reasoning mechanisms and object-centric representations in
+unstructured video models. Performing this analysis jointly over a diverse set
+of supervised and self-supervised representations, we discover that some of
+these mechanism are universal in video transformers. Finally, we show that VTCD
+can be used for fine-grained action recognition and video object segmentation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO']"
+HIPTrack: Visual Tracking with Historical Prompts,Wenrui Cai · Qingjie Liu · Yunhong Wang, ,https://arxiv.org/abs/2311.02072,,2311.02072.pdf,HIPTrack: Visual Tracking with Historical Prompts,"Trackers that follow Siamese paradigm utilize similarity matching between
+template and search region features for tracking. Many methods have been
+explored to enhance tracking performance by incorporating tracking history to
+better handle scenarios involving target appearance variations such as
+deformation and occlusion. However, the utilization of historical information
+in existing methods is insufficient and incomprehensive, which typically
+requires repetitive training and introduces a large amount of computation. In
+this paper, we show that by providing a tracker that follows Siamese paradigm
+with precise and updated historical information, a significant performance
+improvement can be achieved with completely unchanged parameters. Based on
+this, we propose a historical prompt network that uses refined historical
+foreground masks and historical visual features of the target to provide
+comprehensive and precise prompts for the tracker. We build a novel tracker
+called HIPTrack based on the historical prompt network, which achieves
+considerable performance improvements without the need to retrain the entire
+model. We conduct experiments on seven datasets and experimental results
+demonstrate that our method surpasses the current state-of-the-art trackers on
+LaSOT, LaSOText, GOT-10k and NfS. Furthermore, the historical prompt network
+can seamlessly integrate as a plug-and-play module into existing trackers,
+providing performance enhancements. The source code is available at
+https://github.com/WenRuiCai/HIPTrack.",cs.CV,['cs.CV']
+Self-supervised Representation Learning from Arbitrary Scenarios,Zhaowen Li · Yousong Zhu · Zhiyang Chen · Zongxin Gao · Rui Zhao · Chaoyang Zhao · Ming Tang · Jinqiao Wang, ,https://arxiv.org/abs/2403.03740,,2403.03740.pdf,Self-supervised Photographic Image Layout Representation Learning,"In the domain of image layout representation learning, the critical process
+of translating image layouts into succinct vector forms is increasingly
+significant across diverse applications, such as image retrieval, manipulation,
+and generation. Most approaches in this area heavily rely on costly labeled
+datasets and notably lack in adapting their modeling and learning methods to
+the specific nuances of photographic image layouts. This shortfall makes the
+learning process for photographic image layouts suboptimal. In our research, we
+directly address these challenges. We innovate by defining basic layout
+primitives that encapsulate various levels of layout information and by mapping
+these, along with their interconnections, onto a heterogeneous graph structure.
+This graph is meticulously engineered to capture the intricate layout
+information within the pixel domain explicitly. Advancing further, we introduce
+novel pretext tasks coupled with customized loss functions, strategically
+designed for effective self-supervised learning of these layout graphs.
+Building on this foundation, we develop an autoencoder-based network
+architecture skilled in compressing these heterogeneous layout graphs into
+precise, dimensionally-reduced layout representations. Additionally, we
+introduce the LODB dataset, which features a broader range of layout categories
+and richer semantics, serving as a comprehensive benchmark for evaluating the
+effectiveness of layout representation learning methods. Our extensive
+experimentation on this dataset demonstrates the superior performance of our
+approach in the realm of photographic image layout representation learning.",cs.CV,"['cs.CV', 'cs.MM']"
+Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation,Yunhao Ge · Xiaohui Zeng · Jacob Huffman · Tsung-Yi Lin · Ming-Yu Liu · Yin Cui, ,https://arxiv.org/abs/2404.19752,,2404.19752.pdf,Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation,"Existing automatic captioning methods for visual content face challenges such
+as lack of detail, content hallucination, and poor instruction following. In
+this work, we propose VisualFactChecker (VFC), a flexible training-free
+pipeline that generates high-fidelity and detailed captions for both 2D images
+and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text
+captioning models propose multiple initial captions; 2) verification, where a
+large language model (LLM) utilizes tools such as object detection and VQA
+models to fact-check proposed captions; 3) captioning, where an LLM generates
+the final caption by summarizing caption proposals and the fact check
+verification results. In this step, VFC can flexibly generate captions in
+various styles following complex instructions. We conduct comprehensive
+captioning evaluations using four metrics: 1) CLIP-Score for image-text
+similarity; 2) CLIP-Image-Score for measuring the image-image similarity
+between the original and the reconstructed image generated by a text-to-image
+model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V
+for fine-grained evaluation. Evaluation results show that VFC outperforms
+state-of-the-art open-sourced captioning methods for 2D images on the COCO
+dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by
+combining open-source models into a pipeline, we can attain captioning
+capability comparable to proprietary models such as GPT-4V, despite being over
+10x smaller in model size.",cs.CV,['cs.CV']
+NViST: In the Wild New View Synthesis from a Single Image with Transformers,Wonbong Jang · Lourdes Agapito, ,https://arxiv.org/abs/2312.08568,,2312.08568.pdf,NViST: In the Wild New View Synthesis from a Single Image with Transformers,"We propose NViST, a transformer-based model for efficient and generalizable
+novel-view synthesis from a single image for real-world scenes. In contrast to
+many methods that are trained on synthetic data, object-centred scenarios, or
+in a category-specific manner, NViST is trained on MVImgNet, a large-scale
+dataset of casually-captured real-world videos of hundreds of object categories
+with diverse backgrounds. NViST transforms image inputs directly into a
+radiance field, conditioned on camera parameters via adaptive layer
+normalisation. In practice, NViST exploits fine-tuned masked autoencoder (MAE)
+features and translates them to 3D output tokens via cross-attention, while
+addressing occlusions with self-attention. To move away from object-centred
+datasets and enable full scene synthesis, NViST adopts a 6-DOF camera pose
+model and only requires relative pose, dropping the need for canonicalization
+of the training data, which removes a substantial barrier to it being used on
+casually captured datasets. We show results on unseen objects and categories
+from MVImgNet and even generalization to casual phone captures. We conduct
+qualitative and quantitative evaluations on MVImgNet and ShapeNet to show that
+our model represents a step forward towards enabling true in-the-wild
+generalizable novel-view synthesis from a single image. Project webpage:
+https://wbjang.github.io/nvist_webpage.",cs.CV,['cs.CV']
+Space-time Diffusion Features for Zero-shot Text-driven Motion Transfer,Rafail Fridman · Danah Yatim · Omer Bar-Tal · Yoni Kasten · Tali Dekel,https://diffusion-motion-transfer.github.io/,https://arxiv.org/abs/2311.17009,,2311.17009.pdf,Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer,"We present a new method for text-driven motion transfer - synthesizing a
+video that complies with an input text prompt describing the target objects and
+scene while maintaining an input video's motion and scene layout. Prior methods
+are confined to transferring motion across two subjects within the same or
+closely related object categories and are applicable for limited domains (e.g.,
+humans). In this work, we consider a significantly more challenging setting in
+which the target and source objects differ drastically in shape and
+fine-grained motion characteristics (e.g., translating a jumping dog into a
+dolphin). To this end, we leverage a pre-trained and fixed text-to-video
+diffusion model, which provides us with generative and motion priors. The
+pillar of our method is a new space-time feature loss derived directly from the
+model. This loss guides the generation process to preserve the overall motion
+of the input video while complying with the target object in terms of shape and
+fine-grained motion traits.",cs.CV,['cs.CV']
+COLMAP-Free 3D Gaussian Splatting,Yang Fu · Sifei Liu · Amey Kulkarni · Jan Kautz · Alexei A. Efros · Xiaolong Wang, ,https://arxiv.org/abs/2312.07504,,2312.07504.pdf,COLMAP-Free 3D Gaussian Splatting,"While neural rendering has led to impressive advances in scene reconstruction
+and novel view synthesis, it relies heavily on accurately pre-computed camera
+poses. To relax this constraint, multiple efforts have been made to train
+Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the
+implicit representations of NeRFs provide extra challenges to optimize the 3D
+structure and camera poses at the same time. On the other hand, the recently
+proposed 3D Gaussian Splatting provides new opportunities given its explicit
+point cloud representations. This paper leverages both the explicit geometric
+representation and the continuity of the input video stream to perform novel
+view synthesis without any SfM preprocessing. We process the input frames in a
+sequential manner and progressively grow the 3D Gaussians set by taking one
+input frame at a time, without the need to pre-compute the camera poses. Our
+method significantly improves over previous approaches in view synthesis and
+camera pose estimation under large motion changes. Our project page is
+https://oasisyang.github.io/colmap-free-3dgs",cs.CV,['cs.CV']
+Scaling Laws for Data Filtering: Data Curation cannot be Compute Agnostic,Sachin Goyal · Pratyush Maini · Zachary Lipton · Aditi Raghunathan · Zico Kolter, ,https://arxiv.org/abs/2404.07177v1,,2404.07177v1.pdf,Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic,"Vision-language models (VLMs) are trained for thousands of GPU hours on
+carefully curated web datasets. In recent times, data curation has gained
+prominence with several works developing strategies to retain 'high-quality'
+subsets of 'raw' scraped data. For instance, the LAION public dataset retained
+only 10% of the total crawled data. However, these strategies are typically
+developed agnostic of the available compute for training. In this paper, we
+first demonstrate that making filtering decisions independent of training
+compute is often suboptimal: the limited high-quality data rapidly loses its
+utility when repeated, eventually requiring the inclusion of 'unseen' but
+'lower-quality' data. To address this quality-quantity tradeoff
+($\texttt{QQT}$), we introduce neural scaling laws that account for the
+non-homogeneous nature of web data, an angle ignored in existing literature.
+Our scaling laws (i) characterize the $\textit{differing}$ 'utility' of various
+quality subsets of web data; (ii) account for how utility diminishes for a data
+point at its 'nth' repetition; and (iii) formulate the mutual interaction of
+various data pools when combined, enabling the estimation of model performance
+on a combination of multiple data pools without ever jointly training on them.
+Our key message is that data curation $\textit{cannot}$ be agnostic of the
+total compute that a model will be trained for. Our scaling laws allow us to
+curate the best possible pool for achieving top performance on Datacomp at
+various compute budgets, carving out a pareto-frontier for data curation. Code
+is available at https://github.com/locuslab/scaling_laws_data_filtering.",cs.LG,['cs.LG']
+GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces,Yingwenqi Jiang · Jiadong Tu · Yuan Liu · Xifeng Gao · Xiaoxiao Long · Wenping Wang · Yuexin Ma, ,https://arxiv.org/abs/2311.17977,,2311.17977.pdf,GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces,"The advent of neural 3D Gaussians has recently brought about a revolution in
+the field of neural rendering, facilitating the generation of high-quality
+renderings at real-time speeds. However, the explicit and discrete
+representation encounters challenges when applied to scenes featuring
+reflective surfaces. In this paper, we present GaussianShader, a novel method
+that applies a simplified shading function on 3D Gaussians to enhance the
+neural rendering in scenes with reflective surfaces while preserving the
+training and rendering efficiency. The main challenge in applying the shading
+function lies in the accurate normal estimation on discrete 3D Gaussians.
+Specifically, we proposed a novel normal estimation framework based on the
+shortest axis directions of 3D Gaussians with a delicately designed loss to
+make the consistency between the normals and the geometries of Gaussian
+spheres. Experiments show that GaussianShader strikes a commendable balance
+between efficiency and visual quality. Our method surpasses Gaussian Splatting
+in PSNR on specular object datasets, exhibiting an improvement of 1.57dB. When
+compared to prior works handling reflective surfaces, such as Ref-NeRF, our
+optimization time is significantly accelerated (23h vs. 0.58h). Please click on
+our project website to see more results.",cs.CV,['cs.CV']
+BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation,Qihang Zhang · Yinghao Xu · Yujun Shen · Bo Dai · Bolei Zhou · Ceyuan Yang, ,https://arxiv.org/abs/2312.02136,,2312.02136.pdf,BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation,"Generating large-scale 3D scenes cannot simply apply existing 3D object
+synthesis technique since 3D scenes usually hold complex spatial configurations
+and consist of a number of objects at varying scales. We thus propose a
+practical and efficient 3D representation that incorporates an equivariant
+radiance field with the guidance of a bird's-eye view (BEV) map. Concretely,
+objects of synthesized 3D scenes could be easily manipulated through steering
+the corresponding BEV maps. Moreover, by adequately incorporating positional
+encoding and low-pass filters into the generator, the representation becomes
+equivariant to the given BEV map. Such equivariance allows us to produce
+large-scale, even infinite-scale, 3D scenes via synthesizing local scenes and
+then stitching them with smooth consistency. Extensive experiments on 3D scene
+datasets demonstrate the effectiveness of our approach. Our project website is
+at https://zqh0253.github.io/BerfScene/.",cs.CV,['cs.CV']
+L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream,Jingtao Sun · Yaonan Wang · Mingtao Feng · Yulan Guo · Ajmal Mian · Mike Zheng Shou, ,https://arxiv.org/abs/2403.12728,,2403.12728.pdf,Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation,"Fully-supervised category-level pose estimation aims to determine the 6-DoF
+poses of unseen instances from known categories, requiring expensive mannual
+labeling costs. Recently, various self-supervised category-level pose
+estimation methods have been proposed to reduce the requirement of the
+annotated datasets. However, most methods rely on synthetic data or 3D CAD
+model for self-supervised training, and they are typically limited to
+addressing single-object pose problems without considering multi-objective
+tasks or shape reconstruction. To overcome these challenges and limitations, we
+introduce a diffusion-driven self-supervised network for multi-object shape
+reconstruction and categorical pose estimation, only leveraging the shape
+priors. Specifically, to capture the SE(3)-equivariant pose features and 3D
+scale-invariant shape information, we present a Prior-Aware Pyramid 3D Point
+Transformer in our network. This module adopts a point convolutional layer with
+radial-kernels for pose-aware learning and a 3D scale-invariant graph
+convolution layer for object-level shape representation, respectively.
+Furthermore, we introduce a pretrain-to-refine self-supervised training
+paradigm to train our network. It enables proposed network to capture the
+associations between shape priors and observations, addressing the challenge of
+intra-class shape variations by utilising the diffusion mechanism. Extensive
+experiments conducted on four public datasets and a self-built dataset
+demonstrate that our method significantly outperforms state-of-the-art
+self-supervised category-level baselines and even surpasses some
+fully-supervised instance-level and category-level methods.",cs.CV,['cs.CV']
+Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection,Jiawen Zhu · Choubo Ding · Yu Tian · Guansong Pang, ,https://arxiv.org/abs/2310.12790,,2310.12790.pdf,Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection,"Open-set supervised anomaly detection (OSAD) - a recently emerging anomaly
+detection area - aims at utilizing a few samples of anomaly classes seen during
+training to detect unseen anomalies (i.e., samples from open-set anomaly
+classes), while effectively identifying the seen anomalies. Benefiting from the
+prior knowledge illustrated by the seen anomalies, current OSAD methods can
+often largely reduce false positive errors. However, these methods are trained
+in a closed-set setting and treat the anomaly examples as from a homogeneous
+distribution, rendering them less effective in generalizing to unseen anomalies
+that can be drawn from any distribution. This paper proposes to learn
+heterogeneous anomaly distributions using the limited anomaly examples to
+address this issue. To this end, we introduce a novel approach, namely Anomaly
+Heterogeneity Learning (AHL), that simulates a diverse set of heterogeneous
+anomaly distributions and then utilizes them to learn a unified heterogeneous
+abnormality model in surrogate open-set environments. Further, AHL is a generic
+framework that existing OSAD models can plug and play for enhancing their
+abnormality modeling. Extensive experiments on nine real-world anomaly
+detection datasets show that AHL can 1) substantially enhance different
+state-of-the-art OSAD models in detecting seen and unseen anomalies, and 2)
+effectively generalize to unseen anomalies in new domains. Code is available at
+https://github.com/mala-lab/AHL.",cs.CV,['cs.CV']
+"1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness",Bernd Prach · Fabio Brau · Giorgio Buttazzo · Christoph Lampert, ,https://arxiv.org/abs/2311.16833,,2311.16833.pdf,"1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness","The robustness of neural networks against input perturbations with bounded
+magnitude represents a serious concern in the deployment of deep learning
+models in safety-critical systems. Recently, the scientific community has
+focused on enhancing certifiable robustness guarantees by crafting 1-Lipschitz
+neural networks that leverage Lipschitz bounded dense and convolutional layers.
+Although different methods have been proposed in the literature to achieve this
+goal, understanding the performance of such methods is not straightforward,
+since different metrics can be relevant (e.g., training time, memory usage,
+accuracy, certifiable robustness) for different applications. For this reason,
+this work provides a thorough theoretical and empirical comparison between
+methods by evaluating them in terms of memory usage, speed, and certifiable
+robust accuracy. The paper also provides some guidelines and recommendations to
+support the user in selecting the methods that work best depending on the
+available resources. We provide code at
+https://github.com/berndprach/1LipschitzLayersCompared.",cs.LG,"['cs.LG', 'cs.CV', 'cs.NE']"
+Bootstrapping Autonomous Driving Radars with Self-Supervised Learning,Yiduo Hao · Sohrab Madani · Junfeng Guan · Mo Alloulah · Saurabh Gupta · Haitham Al Hassanieh, ,https://arxiv.org/abs/2312.04519,,2312.04519.pdf,Bootstrapping Autonomous Driving Radars with Self-Supervised Learning,"The perception of autonomous vehicles using radars has attracted increased
+research interest due its ability to operate in fog and bad weather. However,
+training radar models is hindered by the cost and difficulty of annotating
+large-scale radar data. To overcome this bottleneck, we propose a
+self-supervised learning framework to leverage the large amount of unlabeled
+radar data to pre-train radar-only embeddings for self-driving perception
+tasks. The proposed method combines radar-to-radar and radar-to-vision
+contrastive losses to learn a general representation from unlabeled radar
+heatmaps paired with their corresponding camera images. When used for
+downstream object detection, we demonstrate that the proposed self-supervision
+framework can improve the accuracy of state-of-the-art supervised baselines by
+$5.8\%$ in mAP. Code is available at \url{https://github.com/yiduohao/Radical}.",cs.CV,['cs.CV']
+CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models,Tuna Han Salih Meral · Enis Simsar · Federico Tombari · Pinar Yanardag, ,https://arxiv.org/abs/2312.06059v1,,2312.06059v1.pdf,CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models,"Images produced by text-to-image diffusion models might not always faithfully
+represent the semantic intent of the provided text prompt, where the model
+might overlook or entirely fail to produce certain objects. Existing solutions
+often require customly tailored functions for each of these problems, leading
+to sub-optimal results, especially for complex prompts. Our work introduces a
+novel perspective by tackling this challenge in a contrastive context. Our
+approach intuitively promotes the segregation of objects in attention maps
+while also maintaining that pairs of related attributes are kept close to each
+other. We conduct extensive experiments across a wide variety of scenarios,
+each involving unique combinations of objects, attributes, and scenes. These
+experiments effectively showcase the versatility, efficiency, and flexibility
+of our method in working with both latent and pixel-based diffusion models,
+including Stable Diffusion and Imagen. Moreover, we publicly share our source
+code to facilitate further research.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model,Dian Zheng · Xiao-Ming Wu · Shuzhou Yang · Jian Zhang · Jian-Fang Hu · Wei-Shi Zheng, ,https://arxiv.org/abs/2403.11157,,2403.11157.pdf,Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model,"Universal image restoration is a practical and potential computer vision task
+for real-world applications. The main challenge of this task is handling the
+different degradation distributions at once. Existing methods mainly utilize
+task-specific conditions (e.g., prompt) to guide the model to learn different
+distributions separately, named multi-partite mapping. However, it is not
+suitable for universal model learning as it ignores the shared information
+between different tasks. In this work, we propose an advanced selective
+hourglass mapping strategy based on diffusion model, termed DiffUIR. Two novel
+considerations make our DiffUIR non-trivial. Firstly, we equip the model with
+strong condition guidance to obtain accurate generation direction of diffusion
+model (selective). More importantly, DiffUIR integrates a flexible shared
+distribution term (SDT) into the diffusion algorithm elegantly and naturally,
+which gradually maps different distributions into a shared one. In the reverse
+process, combined with SDT and strong condition guidance, DiffUIR iteratively
+guides the shared distribution to the task-specific distribution with high
+image quality (hourglass). Without bells and whistles, by only modifying the
+mapping strategy, we achieve state-of-the-art performance on five image
+restoration tasks, 22 benchmarks in the universal setting and zero-shot
+generalization setting. Surprisingly, by only using a lightweight model (only
+0.89M), we could achieve outstanding performance. The source code and
+pre-trained models are available at https://github.com/iSEE-Laboratory/DiffUIR",cs.CV,['cs.CV']
+ActiveDC: Distribution Calibration for Active Finetuning,Wenshuai Xu · Zhenghui Hu · Yu Lu · Jinzhou Meng · Qingjie Liu · Yunhong Wang, ,https://arxiv.org/abs/2311.07634,,2311.07634.pdf,ActiveDC: Distribution Calibration for Active Finetuning,"The pretraining-finetuning paradigm has gained popularity in various computer
+vision tasks. In this paradigm, the emergence of active finetuning arises due
+to the abundance of large-scale data and costly annotation requirements. Active
+finetuning involves selecting a subset of data from an unlabeled pool for
+annotation, facilitating subsequent finetuning. However, the use of a limited
+number of training samples can lead to a biased distribution, potentially
+resulting in model overfitting. In this paper, we propose a new method called
+ActiveDC for the active finetuning tasks. Firstly, we select samples for
+annotation by optimizing the distribution similarity between the subset to be
+selected and the entire unlabeled pool in continuous space. Secondly, we
+calibrate the distribution of the selected samples by exploiting implicit
+category information in the unlabeled pool. The feature visualization provides
+an intuitive sense of the effectiveness of our approach to distribution
+calibration. We conducted extensive experiments on three image classification
+datasets with different sampling ratios. The results indicate that ActiveDC
+consistently outperforms the baseline performance in all image classification
+tasks. The improvement is particularly significant when the sampling ratio is
+low, with performance gains of up to 10%. Our code will be released.",cs.CV,['cs.CV']
+Extreme Point Supervised Instance Segmentation,Hyeonjun Lee · Sehyun Hwang · Suha Kwak, ,https://arxiv.org/abs/2405.20729,,2405.20729.pdf,Extreme Point Supervised Instance Segmentation,"This paper introduces a novel approach to learning instance segmentation
+using extreme points, i.e., the topmost, leftmost, bottommost, and rightmost
+points, of each object. These points are readily available in the modern
+bounding box annotation process while offering strong clues for precise
+segmentation, and thus allows to improve performance at the same annotation
+cost with box-supervised methods. Our work considers extreme points as a part
+of the true instance mask and propagates them to identify potential foreground
+and background points, which are all together used for training a pseudo label
+generator. Then pseudo labels given by the generator are in turn used for
+supervised learning of our final model. On three public benchmarks, our method
+significantly outperforms existing box-supervised methods, further narrowing
+the gap with its fully supervised counterpart. In particular, our model
+generates high-quality masks when a target object is separated into multiple
+parts, where previous box-supervised methods often fail.",cs.CV,['cs.CV']
+Towards Robust 3D Pose Transfer with Adversarial Learning,Haoyu Chen · Hao Tang · Ehsan Adeli · Guoying Zhao, ,https://arxiv.org/abs/2404.02242,,2404.02242.pdf,Towards Robust 3D Pose Transfer with Adversarial Learning,"3D pose transfer that aims to transfer the desired pose to a target mesh is
+one of the most challenging 3D generation tasks. Previous attempts rely on
+well-defined parametric human models or skeletal joints as driving pose
+sources. However, to obtain those clean pose sources, cumbersome but necessary
+pre-processing pipelines are inevitable, hindering implementations of the
+real-time applications. This work is driven by the intuition that the
+robustness of the model can be enhanced by introducing adversarial samples into
+the training, leading to a more invulnerable model to the noisy inputs, which
+even can be further extended to directly handling the real-world data like raw
+point clouds/scans without intermediate processing. Furthermore, we propose a
+novel 3D pose Masked Autoencoder (3D-PoseMAE), a customized MAE that
+effectively learns 3D extrinsic presentations (i.e., pose). 3D-PoseMAE
+facilitates learning from the aspect of extrinsic attributes by simultaneously
+generating adversarial samples that perturb the model and learning the
+arbitrary raw noisy poses via a multi-scale masking strategy. Both qualitative
+and quantitative studies show that the transferred meshes given by our network
+result in much better quality. Besides, we demonstrate the strong
+generalizability of our method on various poses, different domains, and even
+raw scans. Experimental results also show meaningful insights that the
+intermediate adversarial samples generated in the training can successfully
+attack the existing pose transfer models.",cs.CV,['cs.CV']
+Improving Image Restoration through Removing Degradations in Textual Representations,Jingbo Lin · Zhilu Zhang · Yuxiang Wei · Dongwei Ren · Dongsheng Jiang · Qi Tian · Wangmeng Zuo, ,https://arxiv.org/abs/2312.17334,,2312.17334.pdf,Improving Image Restoration through Removing Degradations in Textual Representations,"In this paper, we introduce a new perspective for improving image restoration
+by removing degradation in the textual representations of a given degraded
+image. Intuitively, restoration is much easier on text modality than image one.
+For example, it can be easily conducted by removing degradation-related words
+while keeping the content-aware words. Hence, we combine the advantages of
+images in detail description and ones of text in degradation removal to perform
+restoration. To address the cross-modal assistance, we propose to map the
+degraded images into textual representations for removing the degradations, and
+then convert the restored textual representations into a guidance image for
+assisting image restoration. In particular, We ingeniously embed an
+image-to-text mapper and text restoration module into CLIP-equipped
+text-to-image models to generate the guidance. Then, we adopt a simple
+coarse-to-fine approach to dynamically inject multi-scale information from
+guidance to image restoration networks. Extensive experiments are conducted on
+various image restoration tasks, including deblurring, dehazing, deraining, and
+denoising, and all-in-one image restoration. The results showcase that our
+method outperforms state-of-the-art ones across all these tasks. The codes and
+models are available at \url{https://github.com/mrluin/TextualDegRemoval}.",cs.CV,['cs.CV']
+Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution,Longguang Wang · Juncheng Li · Yingqian Wang · Qingyong Hu · Yulan Guo, ,,https://link.springer.com/article/10.1007/s11760-023-02936-x,,,,,nan
+Dispersed Structured Light for Hyperspectral 3D Imaging,Suhyun Shin · Seokjun Choi · Felix Heide · Seung-Hwan Baek, ,https://arxiv.org/abs/2311.18287,,2311.18287.pdf,Dispersed Structured Light for Hyperspectral 3D Imaging,"Hyperspectral 3D imaging aims to acquire both depth and spectral information
+of a scene. However, existing methods are either prohibitively expensive and
+bulky or compromise on spectral and depth accuracy. In this work, we present
+Dispersed Structured Light (DSL), a cost-effective and compact method for
+accurate hyperspectral 3D imaging. DSL modifies a traditional projector-camera
+system by placing a sub-millimeter thick diffraction grating film front of the
+projector. The grating disperses structured light based on light wavelength. To
+utilize the dispersed structured light, we devise a model for dispersive
+projection image formation and a per-pixel hyperspectral 3D reconstruction
+method. We validate DSL by instantiating a compact experimental prototype. DSL
+achieves spectral accuracy of 18.8nm full-width half-maximum (FWHM) and depth
+error of 1mm. We demonstrate that DSL outperforms prior work on practical
+hyperspectral 3D imaging. DSL promises accurate and practical hyperspectral 3D
+imaging for diverse application domains, including computer vision and
+graphics, cultural heritage, geology, and biology.",eess.IV,"['eess.IV', 'cs.CV', 'cs.GR']"
+MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric,Haokun Lin · Haoli Bai · Zhili Liu · Lu Hou · Muyi Sun · Linqi Song · Ying Wei · Zhenan Sun, ,,https://paperswithcode.com/paper/mope-clip-structured-pruning-for-efficient,,,,,nan
+"TurboSL: Dense, Accurate and Fast 3D by Neural Inverse Structured Light",Parsa Mirdehghan · Maxx Wu · Wenzheng Chen · Wenzheng Chen · David B. Lindell · Kiriakos Kutulakos, ,https://arxiv.org/abs/2306.13361,,2306.13361.pdf,Neural 360$^\circ$ Structured Light with Learned Metasurfaces,"Structured light has proven instrumental in 3D imaging, LiDAR, and
+holographic light projection. Metasurfaces, comprised of sub-wavelength-sized
+nanostructures, facilitate 180$^\circ$ field-of-view (FoV) structured light,
+circumventing the restricted FoV inherent in traditional optics like
+diffractive optical elements. However, extant metasurface-facilitated
+structured light exhibits sub-optimal performance in downstream tasks, due to
+heuristic pattern designs such as periodic dots that do not consider the
+objectives of the end application. In this paper, we present neural 360$^\circ$
+structured light, driven by learned metasurfaces. We propose a differentiable
+framework, that encompasses a computationally-efficient 180$^\circ$ wave
+propagation model and a task-specific reconstructor, and exploits both
+transmission and reflection channels of the metasurface. Leveraging a
+first-order optimizer within our differentiable framework, we optimize the
+metasurface design, thereby realizing neural 360$^\circ$ structured light. We
+have utilized neural 360$^\circ$ structured light for holographic light
+projection and 3D imaging. Specifically, we demonstrate the first 360$^\circ$
+light projection of complex patterns, enabled by our propagation model that can
+be computationally evaluated 50,000$\times$ faster than the Rayleigh-Sommerfeld
+propagation. For 3D imaging, we improve depth-estimation accuracy by
+5.09$\times$ in RMSE compared to the heuristically-designed structured light.
+Neural 360$^\circ$ structured light promises robust 360$^\circ$ imaging and
+display for robotics, extended-reality systems, and human-computer
+interactions.",physics.optics,"['physics.optics', 'cs.CV', 'eess.IV']"
+Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning,wenlong deng · Christos Thrampoulidis · Xiaoxiao Li, ,https://arxiv.org/abs/2310.18285,,2310.18285.pdf,Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning,"Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve
+state-of-the-art performance with improved efficiency in various computer
+vision tasks. This suggests a promising paradigm shift of adapting pre-trained
+ViT models to Federated Learning (FL) settings. However, the challenge of data
+heterogeneity among FL clients presents a significant hurdle in effectively
+deploying ViT models. Existing Generalized FL (GFL) and Personalized FL (PFL)
+methods have limitations in balancing performance across both global and local
+data distributions. In this paper, we present a novel algorithm, SGPT, that
+integrates GFL and PFL approaches by employing a unique combination of both
+shared and group-specific prompts. This design enables SGPT to capture both
+common and group-specific features. A key feature of SGPT is its prompt
+selection module, which facilitates the training of a single global model
+capable of automatically adapting to diverse local client data distributions
+without the need for local fine-tuning. To effectively train the prompts, we
+utilize block coordinate descent (BCD), learning from common feature
+information (shared prompts), and then more specialized knowledge (group
+prompts) iteratively. Theoretically, we justify that learning the proposed
+prompts can reduce the gap between global and local performance. Empirically,
+we conduct experiments on both label and feature heterogeneity settings in
+comparison with state-of-the-art baselines, along with extensive ablation
+studies, to substantiate the superior performance of SGPT.",cs.LG,"['cs.LG', 'cs.CV']"
+Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models,Shitian Zhao · Zhuowan Li · YadongLu · Alan L. Yuille · Yan Wang, ,https://arxiv.org/abs/2312.06685,,2312.06685.pdf,Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models,"While Multi-modal Language Models (MLMs) demonstrate impressive multimodal
+ability, they still struggle on providing factual and precise responses for
+tasks like visual question answering (VQA). In this paper, we address this
+challenge from the perspective of contextual information. We propose Causal
+Context Generation, Causal-CoG, which is a prompting strategy that engages
+contextual information to enhance precise VQA during inference. Specifically,
+we prompt MLMs to generate contexts, i.e, text description of an image, and
+engage the generated contexts for question answering. Moreover, we investigate
+the advantage of contexts on VQA from a causality perspective, introducing
+causality filtering to select samples for which contextual information is
+helpful. To show the effectiveness of Causal-CoG, we run extensive experiments
+on 10 multimodal benchmarks and show consistent improvements, e.g., +6.30% on
+POPE, +13.69% on Vizwiz and +6.43% on VQAv2 compared to direct decoding,
+surpassing existing methods. We hope Casual-CoG inspires explorations of
+context knowledge in multimodal models, and serves as a plug-and-play strategy
+for MLM decoding.",cs.AI,['cs.AI']
+Person-in-WiFi 3D: End-to-End Multi-Person 3D Pose Estimation with Wi-Fi,Kangwei Yan · Fei Wang · Bo Qian · Han Ding · Jinsong Han · Xing Wei, ,https://arxiv.org/abs/2404.02041,,2404.02041.pdf,SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation,"We present a new self-supervised approach, SelfPose3d, for estimating 3d
+poses of multiple persons from multiple camera views. Unlike current
+state-of-the-art fully-supervised methods, our approach does not require any 2d
+or 3d ground-truth poses and uses only the multi-view input images from a
+calibrated camera setup and 2d pseudo poses generated from an off-the-shelf 2d
+human pose estimator. We propose two self-supervised learning objectives:
+self-supervised person localization in 3d space and self-supervised 3d pose
+estimation. We achieve self-supervised 3d person localization by training the
+model on synthetically generated 3d points, serving as 3d person root
+positions, and on the projected root-heatmaps in all the views. We then model
+the 3d poses of all the localized persons with a bottleneck representation, map
+them onto all views obtaining 2d joints, and render them using 2d Gaussian
+heatmaps in an end-to-end differentiable manner. Afterwards, we use the
+corresponding 2d joints and heatmaps from the pseudo 2d poses for learning. To
+alleviate the intrinsic inaccuracy of the pseudo labels, we propose an adaptive
+supervision attention mechanism to guide the self-supervision. Our experiments
+and analysis on three public benchmark datasets, including Panoptic, Shelf, and
+Campus, show the effectiveness of our approach, which is comparable to
+fully-supervised methods. Code is available at
+\url{https://github.com/CAMMA-public/SelfPose3D}",cs.CV,['cs.CV']
+Multi-agent Collaborative Perception via Motion-aware Robust Communication Network,Shixin Hong · Yu LIU · Zhi Li · Shaohui Li · You He, ,https://arxiv.org/abs/2401.12694,,2401.12694.pdf,Pragmatic Communication in Multi-Agent Collaborative Perception,"Collaborative perception allows each agent to enhance its perceptual
+abilities by exchanging messages with others. It inherently results in a
+trade-off between perception ability and communication costs. Previous works
+transmit complete full-frame high-dimensional feature maps among agents,
+resulting in substantial communication costs. To promote communication
+efficiency, we propose only transmitting the information needed for the
+collaborator's downstream task. This pragmatic communication strategy focuses
+on three key aspects: i) pragmatic message selection, which selects
+task-critical parts from the complete data, resulting in spatially and
+temporally sparse feature vectors; ii) pragmatic message representation, which
+achieves pragmatic approximation of high-dimensional feature vectors with a
+task-adaptive dictionary, enabling communicating with integer indices; iii)
+pragmatic collaborator selection, which identifies beneficial collaborators,
+pruning unnecessary communication links. Following this strategy, we first
+formulate a mathematical optimization framework for the
+perception-communication trade-off and then propose PragComm, a multi-agent
+collaborative perception system with two key components: i) single-agent
+detection and tracking and ii) pragmatic collaboration. The proposed PragComm
+promotes pragmatic communication and adapts to a wide range of communication
+conditions. We evaluate PragComm for both collaborative 3D object detection and
+tracking tasks in both real-world, V2V4Real, and simulation datasets, OPV2V and
+V2X-SIM2.0. PragComm consistently outperforms previous methods with more than
+32.7K times lower communication volume on OPV2V. Code is available at
+github.com/PhyllisH/PragComm.",cs.CV,['cs.CV']
+Dense Optical Tracking: Connecting the Dots,Guillaume Le Moing · Jean Ponce · Cordelia Schmid,https://github.com/16lemoing/dot,https://arxiv.org/abs/2312.00786,,2312.00786.pdf,Dense Optical Tracking: Connecting the Dots,"Recent approaches to point tracking are able to recover the trajectory of any
+scene point through a large portion of a video despite the presence of
+occlusions. They are, however, too slow in practice to track every point
+observed in a single frame in a reasonable amount of time. This paper
+introduces DOT, a novel, simple and efficient method for solving this problem.
+It first extracts a small set of tracks from key regions at motion boundaries
+using an off-the-shelf point tracking algorithm. Given source and target
+frames, DOT then computes rough initial estimates of a dense flow field and
+visibility mask through nearest-neighbor interpolation, before refining them
+using a learnable optical flow estimator that explicitly handles occlusions and
+can be trained on synthetic data with ground-truth correspondences. We show
+that DOT is significantly more accurate than current optical flow techniques,
+outperforms sophisticated ""universal"" trackers like OmniMotion, and is on par
+with, or better than, the best point tracking algorithms like CoTracker while
+being at least two orders of magnitude faster. Quantitative and qualitative
+experiments with synthetic and real videos validate the promise of the proposed
+approach. Code, data, and videos showcasing the capabilities of our approach
+are available in the project webpage: https://16lemoing.github.io/dot .",cs.CV,['cs.CV']
+Enhancing Post-training Quantization Calibration through Contrastive Learning,Yuzhang Shang · Gaowen Liu · Ramana Kompella · Yan Yan, ,https://arxiv.org/abs/2311.06322,,2311.06322.pdf,Post-training Quantization with Progressive Calibration and Activation Relaxing for Text-to-Image Diffusion Models,"Diffusion models have achieved great success due to their remarkable
+generation ability. However, their high computational overhead is still a
+troublesome problem. Recent studies have leveraged post-training quantization
+(PTQ) to compress diffusion models. However, most of them only focus on
+unconditional models, leaving the quantization of widely used large pretrained
+text-to-image models, e.g., Stable Diffusion, largely unexplored. In this
+paper, we propose a novel post-training quantization method PCR (Progressive
+Calibration and Relaxing) for text-to-image diffusion models, which consists of
+a progressive calibration strategy that considers the accumulated quantization
+error across timesteps, and an activation relaxing strategy that improves the
+performance with negligible cost. Additionally, we demonstrate the previous
+metrics for text-to-image diffusion model quantization are not accurate due to
+the distribution gap. To tackle the problem, we propose a novel QDiffBench
+benchmark, which utilizes data in the same domain for more accurate evaluation.
+Besides, QDiffBench also considers the generalization performance of the
+quantized model outside the calibration dataset. Extensive experiments on
+Stable Diffusion and Stable Diffusion XL demonstrate the superiority of our
+method and benchmark. Moreover, we are the first to achieve quantization for
+Stable Diffusion XL while maintaining the performance.",cs.CV,"['cs.CV', 'cs.LG']"
+PanoPose: Self-supervised Relative Pose Estimation for Panoramic Images,Diantao Tu · Hainan Cui · Xianwei Zheng · Shuhan Shen, ,https://arxiv.org/abs/2404.02041,,,SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation,"We present a new self-supervised approach, SelfPose3d, for estimating 3d
+poses of multiple persons from multiple camera views. Unlike current
+state-of-the-art fully-supervised methods, our approach does not require any 2d
+or 3d ground-truth poses and uses only the multi-view input images from a
+calibrated camera setup and 2d pseudo poses generated from an off-the-shelf 2d
+human pose estimator. We propose two self-supervised learning objectives:
+self-supervised person localization in 3d space and self-supervised 3d pose
+estimation. We achieve self-supervised 3d person localization by training the
+model on synthetically generated 3d points, serving as 3d person root
+positions, and on the projected root-heatmaps in all the views. We then model
+the 3d poses of all the localized persons with a bottleneck representation, map
+them onto all views obtaining 2d joints, and render them using 2d Gaussian
+heatmaps in an end-to-end differentiable manner. Afterwards, we use the
+corresponding 2d joints and heatmaps from the pseudo 2d poses for learning. To
+alleviate the intrinsic inaccuracy of the pseudo labels, we propose an adaptive
+supervision attention mechanism to guide the self-supervision. Our experiments
+and analysis on three public benchmark datasets, including Panoptic, Shelf, and
+Campus, show the effectiveness of our approach, which is comparable to
+fully-supervised methods. Code is available at
+\url{https://github.com/CAMMA-public/SelfPose3D}",cs.CV,['cs.CV']
+Semantics-aware Motion Retargeting with Vision-Language Models,Haodong Zhang · ZhiKe Chen · Haocheng Xu · Lei Hao · Xiaofei Wu · Songcen Xu · Zhensong Zhang · Yue Wang · Rong Xiong, ,https://arxiv.org/abs/2312.01964,,2312.01964.pdf,Semantics-aware Motion Retargeting with Vision-Language Models,"Capturing and preserving motion semantics is essential to motion retargeting
+between animation characters. However, most of the previous works neglect the
+semantic information or rely on human-designed joint-level representations.
+Here, we present a novel Semantics-aware Motion reTargeting (SMT) method with
+the advantage of vision-language models to extract and maintain meaningful
+motion semantics. We utilize a differentiable module to render 3D motions. Then
+the high-level motion semantics are incorporated into the motion retargeting
+process by feeding the vision-language model with the rendered images and
+aligning the extracted semantic embeddings. To ensure the preservation of
+fine-grained motion details and high-level semantics, we adopt a two-stage
+pipeline consisting of skeleton-aware pre-training and fine-tuning with
+semantics and geometry constraints. Experimental results show the effectiveness
+of the proposed method in producing high-quality motion retargeting results
+while accurately preserving motion semantics.",cs.CV,"['cs.CV', 'cs.GR']"
+HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances,Supreeth Narasimhaswamy · Uttaran Bhattacharya · Xiang Chen · Ishita Dasgupta · Saayan Mitra · Minh Hoai, ,https://arxiv.org/abs/2403.01693,,2403.01693.pdf,HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances,"Text-to-image generative models can generate high-quality humans, but realism
+is lost when generating hands. Common artifacts include irregular hand poses,
+shapes, incorrect numbers of fingers, and physically implausible finger
+orientations. To generate images with realistic hands, we propose a novel
+diffusion-based architecture called HanDiffuser that achieves realism by
+injecting hand embeddings in the generative process. HanDiffuser consists of
+two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and
+MANO-Hand parameters from input text prompts, and a Text-Guided
+Hand-Params-to-Image diffusion model to synthesize images by conditioning on
+the prompts and hand parameters generated by the previous component. We
+incorporate multiple aspects of hand representation, including 3D shapes and
+joint-level finger positions, orientations and articulations, for robust
+learning and reliable performance during inference. We conduct extensive
+quantitative and qualitative experiments and perform user studies to
+demonstrate the efficacy of our method in generating images with high-quality
+hands.",cs.CV,"['cs.CV', 'cs.AI']"
+Communication-Efficient Collaborative Perception via Information Filling with Codebook,Yue Hu · Juntong Peng · Sifei Liu · Junhao Ge · Si Liu · Siheng Chen, ,https://arxiv.org/abs/2405.04966,,2405.04966.pdf,Communication-Efficient Collaborative Perception via Information Filling with Codebook,"Collaborative perception empowers each agent to improve its perceptual
+ability through the exchange of perceptual messages with other agents. It
+inherently results in a fundamental trade-off between perception ability and
+communication cost. To address this bottleneck issue, our core idea is to
+optimize the collaborative messages from two key aspects: representation and
+selection. The proposed codebook-based message representation enables the
+transmission of integer codes, rather than high-dimensional feature maps. The
+proposed information-filling-driven message selection optimizes local messages
+to collectively fill each agent's information demand, preventing information
+overflow among multiple agents. By integrating these two designs, we propose
+CodeFilling, a novel communication-efficient collaborative perception system,
+which significantly advances the perception-communication trade-off and is
+inclusive to both homogeneous and heterogeneous collaboration settings. We
+evaluate CodeFilling in both a real-world dataset, DAIR-V2X, and a new
+simulation dataset, OPV2VH+. Results show that CodeFilling outperforms previous
+SOTA Where2comm on DAIR-V2X/OPV2VH+ with 1,333/1,206 times lower communication
+volume. Our code is available at https://github.com/PhyllisH/CodeFilling.",cs.IT,"['cs.IT', 'cs.CV', 'cs.MA', 'math.IT']"
+Adversarial Score Distillation: When score distillation meets GAN,Min Wei · Jingkai Zhou · Junyao Sun · Xuesong Zhang, ,https://arxiv.org/abs/2312.00739,,2312.00739.pdf,Adversarial Score Distillation: When score distillation meets GAN,"Existing score distillation methods are sensitive to classifier-free guidance
+(CFG) scale: manifested as over-smoothness or instability at small CFG scales,
+while over-saturation at large ones. To explain and analyze these issues, we
+revisit the derivation of Score Distillation Sampling (SDS) and decipher
+existing score distillation with the Wasserstein Generative Adversarial Network
+(WGAN) paradigm. With the WGAN paradigm, we find that existing score
+distillation either employs a fixed sub-optimal discriminator or conducts
+incomplete discriminator optimization, resulting in the scale-sensitive issue.
+We propose the Adversarial Score Distillation (ASD), which maintains an
+optimizable discriminator and updates it using the complete optimization
+objective. Experiments show that the proposed ASD performs favorably in 2D
+distillation and text-to-3D tasks against existing methods. Furthermore, to
+explore the generalization ability of our WGAN paradigm, we extend ASD to the
+image editing task, which achieves competitive results. The project page and
+code are at https://github.com/2y7c3/ASD.",cs.CV,['cs.CV']
+Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks,Bin Xiao · Haiping Wu · Weijian Xu · Xiyang Dai · Houdong Hu · Yumao Lu · Michael Zeng · Ce Liu · Lu Yuan, ,https://arxiv.org/abs/2311.06242,,2311.06242.pdf,Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks,"We introduce Florence-2, a novel vision foundation model with a unified,
+prompt-based representation for a variety of computer vision and
+vision-language tasks. While existing large vision models excel in transfer
+learning, they struggle to perform a diversity of tasks with simple
+instructions, a capability that implies handling the complexity of various
+spatial hierarchy and semantic granularity. Florence-2 was designed to take
+text-prompt as task instructions and generate desirable results in text forms,
+whether it be captioning, object detection, grounding or segmentation. This
+multi-task learning setup demands large-scale, high-quality annotated data. To
+this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive
+visual annotations on 126 million images, using an iterative strategy of
+automated image annotation and model refinement. We adopted a
+sequence-to-sequence structure to train Florence-2 to perform versatile and
+comprehensive vision tasks. Extensive evaluations on numerous tasks
+demonstrated Florence-2 to be a strong vision foundation model contender with
+unprecedented zero-shot and fine-tuning capabilities.",cs.CV,['cs.CV']
+GOAT-Bench: A Benchmark for Multi-modal Lifelong Navigation,Mukul Khanna · Ram Ramrakhya · Gunjan Chhablani · Sriram Yenamandra · Theo Gervet · Matthew Chang · Zsolt Kira · Devendra Singh Chaplot · Dhruv Batra · Roozbeh Mottaghi, ,https://arxiv.org/abs/2404.06609,,2404.06609.pdf,GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation,"The Embodied AI community has made significant strides in visual navigation
+tasks, exploring targets from 3D coordinates, objects, language descriptions,
+and images. However, these navigation models often handle only a single input
+modality as the target. With the progress achieved so far, it is time to move
+towards universal navigation models capable of handling various goal types,
+enabling more effective user interaction with robots. To facilitate this goal,
+we propose GOAT-Bench, a benchmark for the universal navigation task referred
+to as GO to AnyThing (GOAT). In this task, the agent is directed to navigate to
+a sequence of targets specified by the category name, language description, or
+image in an open-vocabulary fashion. We benchmark monolithic RL and modular
+methods on the GOAT task, analyzing their performance across modalities, the
+role of explicit and implicit scene memories, their robustness to noise in goal
+specifications, and the impact of memory in lifelong scenarios.",cs.AI,"['cs.AI', 'cs.RO']"
+LLaFS: When Large Language Models Meet Few-Shot Segmentation,Lanyun Zhu · Tianrun Chen · Deyi Ji · Deyi Ji · Jieping Ye · Jun Liu, ,https://arxiv.org/abs/2311.16926,,2311.16926.pdf,LLaFS: When Large Language Models Meet Few-Shot Segmentation,"This paper proposes LLaFS, the first attempt to leverage large language
+models (LLMs) in few-shot segmentation. In contrast to the conventional
+few-shot segmentation methods that only rely on the limited and biased
+information from the annotated support images, LLaFS leverages the vast prior
+knowledge gained by LLM as an effective supplement and directly uses the LLM to
+segment images in a few-shot manner. To enable the text-based LLM to handle
+image-related tasks, we carefully design an input instruction that allows the
+LLM to produce segmentation results represented as polygons, and propose a
+region-attribute table to simulate the human visual mechanism and provide
+multi-modal guidance. We also synthesize pseudo samples and use curriculum
+learning for pretraining to augment data and achieve better optimization. LLaFS
+achieves state-of-the-art results on multiple datasets, showing the potential
+of using LLMs for few-shot computer vision tasks.",cs.CV,['cs.CV']
+MVCPS-NeuS: Multi-view Constrained Photometric Stereo for Neural Surface Reconstruction,Hiroaki Santo · Fumio Okura · Yasuyuki Matsushita,https://github.com/hiroaki-santo/mvcps-neus,https://arxiv.org/abs/2405.12057,,2405.12057.pdf,NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo,"In this work we present a novel multi-view photometric stereo (PS) method.
+Like many works in 3D reconstruction we are leveraging neural shape
+representations and learnt renderers. However, our work differs from the
+state-of-the-art multi-view PS methods such as PS-NeRF or SuperNormal we
+explicity leverage per-pixel intensity renderings rather than relying mainly on
+estimated normals.
+  We model point light attenuation and explicitly raytrace cast shadows in
+order to best approximate each points incoming radiance. This is used as input
+to a fully neural material renderer that uses minimal prior assumptions and it
+is jointly optimised with the surface. Finally, estimated normal and
+segmentation maps can also incorporated in order to maximise the surface
+accuracy.
+  Our method is among the first to outperform the classical approach of
+DiLiGenT-MV and achieves average 0.2mm Chamfer distance for objects imaged at
+approx 1.5m distance away with approximate 400x400 resolution. Moreover, we
+show robustness to poor normals in low light count scenario, achieving 0.27mm
+Chamfer distance when pixel rendering is used instead of estimated normals.",cs.CV,['cs.CV']
+FlowTrack: Revisiting Optical Flow for Long-Range Dense Tracking,Seokju Cho · Gabriel Huang · Seungryong Kim · Joon-Young Lee, ,https://arxiv.org/abs/2312.00786,,,Dense Optical Tracking: Connecting the Dots,"Recent approaches to point tracking are able to recover the trajectory of any
+scene point through a large portion of a video despite the presence of
+occlusions. They are, however, too slow in practice to track every point
+observed in a single frame in a reasonable amount of time. This paper
+introduces DOT, a novel, simple and efficient method for solving this problem.
+It first extracts a small set of tracks from key regions at motion boundaries
+using an off-the-shelf point tracking algorithm. Given source and target
+frames, DOT then computes rough initial estimates of a dense flow field and
+visibility mask through nearest-neighbor interpolation, before refining them
+using a learnable optical flow estimator that explicitly handles occlusions and
+can be trained on synthetic data with ground-truth correspondences. We show
+that DOT is significantly more accurate than current optical flow techniques,
+outperforms sophisticated ""universal"" trackers like OmniMotion, and is on par
+with, or better than, the best point tracking algorithms like CoTracker while
+being at least two orders of magnitude faster. Quantitative and qualitative
+experiments with synthetic and real videos validate the promise of the proposed
+approach. Code, data, and videos showcasing the capabilities of our approach
+are available in the project webpage: https://16lemoing.github.io/dot .",cs.CV,['cs.CV']
+MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning,Matteo Farina · Massimiliano Mancini · Elia Cunegatti · Gaowen Liu · Giovanni Iacca · Elisa Ricci,https://github.com/FarinaMatteo/multiflow,https://arxiv.org/abs/2404.05621,,2404.05621.pdf,MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning,"While excellent in transfer learning, Vision-Language models (VLMs) come with
+high computational costs due to their large number of parameters. To address
+this issue, removing parameters via model pruning is a viable solution.
+However, existing techniques for VLMs are task-specific, and thus require
+pruning the network from scratch for each new task of interest. In this work,
+we explore a new direction: Task-Agnostic Vision-Language Pruning (TA-VLP).
+Given a pretrained VLM, the goal is to find a unique pruned counterpart
+transferable to multiple unknown downstream tasks. In this challenging setting,
+the transferable representations already encoded in the pretrained model are a
+key aspect to preserve. Thus, we propose Multimodal Flow Pruning (MULTIFLOW), a
+first, gradient-free, pruning framework for TA-VLP where: (i) the importance of
+a parameter is expressed in terms of its magnitude and its information flow, by
+incorporating the saliency of the neurons it connects; and (ii) pruning is
+driven by the emergent (multimodal) distribution of the VLM parameters after
+pretraining. We benchmark eight state-of-the-art pruning algorithms in the
+context of TA-VLP, experimenting with two VLMs, three vision-language tasks,
+and three pruning ratios. Our experimental results show that MULTIFLOW
+outperforms recent sophisticated, combinatorial competitors in the vast
+majority of the cases, paving the way towards addressing TA-VLP. The code is
+publicly available at https://github.com/FarinaMatteo/multiflow.",cs.CV,['cs.CV']
+In-Context Matting,He Guo · Zixuan Ye · Zhiguo Cao · Hao Lu, ,https://arxiv.org/abs/2403.15789,,2403.15789.pdf,In-Context Matting,"We introduce in-context matting, a novel task setting of image matting. Given
+a reference image of a certain foreground and guided priors such as points,
+scribbles, and masks, in-context matting enables automatic alpha estimation on
+a batch of target images of the same foreground category, without additional
+auxiliary input. This setting marries good performance in auxiliary input-based
+matting and ease of use in automatic matting, which finds a good trade-off
+between customization and automation. To overcome the key challenge of accurate
+foreground matching, we introduce IconMatting, an in-context matting model
+built upon a pre-trained text-to-image diffusion model. Conditioned on inter-
+and intra-similarity matching, IconMatting can make full use of reference
+context to generate accurate target alpha mattes. To benchmark the task, we
+also introduce a novel testing dataset ICM-$57$, covering 57 groups of
+real-world images. Quantitative and qualitative results on the ICM-57 testing
+set show that IconMatting rivals the accuracy of trimap-based matting while
+retaining the automation level akin to automatic matting. Code is available at
+https://github.com/tiny-smart/in-context-matting",cs.CV,['cs.CV']
+Interactive Continual Learning: Fast and Slow Thinking,Biqing Qi · Xinquan Chen · Junqi Gao · Dong Li · Jianxing Liu · Ligang Wu · Bowen Zhou, ,https://arxiv.org/abs/2403.02628,,2403.02628.pdf,Interactive Continual Learning: Fast and Slow Thinking,"Advanced life forms, sustained by the synergistic interaction of neural
+cognitive mechanisms, continually acquire and transfer knowledge throughout
+their lifespan. In contrast, contemporary machine learning paradigms exhibit
+limitations in emulating the facets of continual learning (CL). Nonetheless,
+the emergence of large language models (LLMs) presents promising avenues for
+realizing CL via interactions with these models. Drawing on Complementary
+Learning System theory, this paper presents a novel Interactive Continual
+Learning (ICL) framework, enabled by collaborative interactions among models of
+various sizes. Specifically, we assign the ViT model as System1 and multimodal
+LLM as System2. To enable the memory module to deduce tasks from class
+information and enhance Set2Set retrieval, we propose the Class-Knowledge-Task
+Multi-Head Attention (CKT-MHA). Additionally, to improve memory retrieval in
+System1 through enhanced geometric representation, we introduce the CL-vMF
+mechanism, based on the von Mises-Fisher (vMF) distribution. Meanwhile, we
+introduce the von Mises-Fisher Outlier Detection and Interaction (vMF-ODI)
+strategy to identify hard examples, thus enhancing collaboration between
+System1 and System2 for complex reasoning realization. Comprehensive evaluation
+of our proposed ICL demonstrates significant resistance to forgetting and
+superior performance relative to existing methods. Code is available at
+github.com/ICL.",cs.CV,"['cs.CV', 'cs.LG']"
+The Devil is in the Details: StyleFeatureEditor for Detail-Rich StyleGAN Inversion and High Quality Image Editing,Denis Bobkov · Vadim Titov · Aibek Alanov · Dmitry Vetrov, ,https://ar5iv.labs.arxiv.org/html/2203.08450,,2203.08450.pdf,The Devil Is in the Details: Window-based Attention for Image Compression,"Learned image compression methods have exhibited superior rate-distortion
+performance than classical image compression standards. Most existing learned
+image compression models are based on Convolutional Neural Networks (CNNs).
+Despite great contributions, a main drawback of CNN based model is that its
+structure is not designed for capturing local redundancy, especially the
+non-repetitive textures, which severely affects the reconstruction quality.
+Therefore, how to make full use of both global structure and local texture
+becomes the core problem for learning-based image compression. Inspired by
+recent progresses of Vision Transformer (ViT) and Swin Transformer, we found
+that combining the local-aware attention mechanism with the global-related
+feature learning could meet the expectation in image compression. In this
+paper, we first extensively study the effects of multiple kinds of attention
+mechanisms for local features learning, then introduce a more straightforward
+yet effective window-based local attention block. The proposed window-based
+attention is very flexible which could work as a plug-and-play component to
+enhance CNN and Transformer models. Moreover, we propose a novel Symmetrical
+TransFormer (STF) framework with absolute transformer blocks in the
+down-sampling encoder and up-sampling decoder. Extensive experimental
+evaluations have shown that the proposed method is effective and outperforms
+the state-of-the-art methods. The code is publicly available at
+https://github.com/Googolxx/STF.",eess.IV,"['eess.IV', 'cs.CV']"
+RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos,Hongchi Xia · Yang Fu · Sifei Liu · Xiaolong Wang, ,https://arxiv.org/abs/2401.12592,,2401.12592.pdf,RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos,"We introduce a new RGB-D object dataset captured in the wild called
+WildRGB-D. Unlike most existing real-world object-centric datasets which only
+come with RGB capturing, the direct capture of the depth channel allows better
+3D annotations and broader downstream applications. WildRGB-D comprises
+large-scale category-level RGB-D object videos, which are taken using an iPhone
+to go around the objects in 360 degrees. It contains around 8500 recorded
+objects and nearly 20000 RGB-D videos across 46 common object categories. These
+videos are taken with diverse cluttered backgrounds with three setups to cover
+as many real-world scenarios as possible: (i) a single object in one video;
+(ii) multiple objects in one video; and (iii) an object with a static hand in
+one video. The dataset is annotated with object masks, real-world scale camera
+poses, and reconstructed aggregated point clouds from RGBD videos. We benchmark
+four tasks with WildRGB-D including novel view synthesis, camera pose
+estimation, object 6d pose estimation, and object surface reconstruction. Our
+experiments show that the large-scale capture of RGB-D objects provides a large
+potential to advance 3D object learning. Our project page is
+https://wildrgbd.github.io/.",cs.CV,['cs.CV']
+Learning Continual Compatible Representation for Re-indexing Free Lifelong Person Re-identification,Zhenyu Cui · Jiahuan Zhou · Xun Wang · Manyu Zhu · Yuxin Peng, ,https://arxiv.org/abs/2403.16003,,2403.16003.pdf,Diverse Representation Embedding for Lifelong Person Re-Identification,"Lifelong Person Re-Identification (LReID) aims to continuously learn from
+successive data streams, matching individuals across multiple cameras. The key
+challenge for LReID is how to effectively preserve old knowledge while
+incrementally learning new information, which is caused by task-level domain
+gaps and limited old task datasets. Existing methods based on CNN backbone are
+insufficient to explore the representation of each instance from different
+perspectives, limiting model performance on limited old task datasets and new
+task datasets. Unlike these methods, we propose a Diverse Representations
+Embedding (DRE) framework that first explores a pure transformer for LReID. The
+proposed DRE preserves old knowledge while adapting to new information based on
+instance-level and task-level layout. Concretely, an Adaptive Constraint Module
+(ACM) is proposed to implement integration and push away operations between
+multiple overlapping representations generated by transformer-based backbone,
+obtaining rich and discriminative representations for each instance to improve
+adaptive ability of LReID. Based on the processed diverse representations, we
+propose Knowledge Update (KU) and Knowledge Preservation (KP) strategies at the
+task-level layout by introducing the adjustment model and the learner model. KU
+strategy enhances the adaptive learning ability of learner models for new
+information under the adjustment model prior, and KP strategy preserves old
+knowledge operated by representation-level alignment and logit-level
+supervision in limited old task datasets while guaranteeing the adaptive
+learning information capacity of the LReID model. Compared to state-of-the-art
+methods, our method achieves significantly improved performance in holistic,
+large-scale, and occluded datasets.",cs.CV,"['cs.CV', 'cs.AI']"
+6-DoF Pose Estimation with MultiScale Residual Correlation,Yuelong Li · Yafei Mao · Raja Bala · Sunil Hadap,https://github.com/amzn/mrc-net-6d-pose,https://arxiv.org/abs/2403.08019,,2403.08019.pdf,MRC-Net: 6-DoF Pose Estimation with MultiScale Residual Correlation,"We propose a single-shot approach to determining 6-DoF pose of an object with
+available 3D computer-aided design (CAD) model from a single RGB image. Our
+method, dubbed MRC-Net, comprises two stages. The first performs pose
+classification and renders the 3D object in the classified pose. The second
+stage performs regression to predict fine-grained residual pose within class.
+Connecting the two stages is a novel multi-scale residual correlation (MRC)
+layer that captures high-and-low level correspondences between the input image
+and rendering from first stage. MRC-Net employs a Siamese network with shared
+weights between both stages to learn embeddings for input and rendered images.
+To mitigate ambiguity when predicting discrete pose class labels on symmetric
+objects, we use soft probabilistic labels to define pose class in the first
+stage. We demonstrate state-of-the-art accuracy, outperforming all competing
+RGB-based methods on four challenging BOP benchmark datasets: T-LESS, LM-O,
+YCB-V, and ITODD. Our method is non-iterative and requires no complex
+post-processing.",cs.CV,['cs.CV']
+Minimal Perspective Autocalibration,Andrea Porfiri Dal Cin · Timothy Duff · Luca Magri · Tomas Pajdla, ,https://arxiv.org/abs/2405.05605,,2405.05605.pdf,Minimal Perspective Autocalibration,"We introduce a new family of minimal problems for reconstruction from
+multiple views. Our primary focus is a novel approach to autocalibration, a
+long-standing problem in computer vision. Traditional approaches to this
+problem, such as those based on Kruppa's equations or the modulus constraint,
+rely explicitly on the knowledge of multiple fundamental matrices or a
+projective reconstruction. In contrast, we consider a novel formulation
+involving constraints on image points, the unknown depths of 3D points, and a
+partially specified calibration matrix $K$. For $2$ and $3$ views, we present a
+comprehensive taxonomy of minimal autocalibration problems obtained by relaxing
+some of these constraints. These problems are organized into classes according
+to the number of views and any assumed prior knowledge of $K$. Within each
+class, we determine problems with the fewest -- or a relatively small number of
+-- solutions. From this zoo of problems, we devise three practical solvers.
+Experiments with synthetic and real data and interfacing our solvers with
+COLMAP demonstrate that we achieve superior accuracy compared to
+state-of-the-art calibration methods. The code is available at
+https://github.com/andreadalcin/MinimalPerspectiveAutocalibration",cs.CV,['cs.CV']
+Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification,Jiancheng Zhang · Haijin Zeng · Yongyong Chen · Dengxiu Yu · Yinping Zhao,https://github.com/ZhangJC-2k/SSR,,https://ieeexplore.ieee.org/document/10411766,,,,,nan
+WorDepth: Variational Language Prior for Monocular Depth Estimation,Ziyao Zeng · Hyoungseob Park · Fengyu Yang · Daniel Wang · Stefano Soatto · Dong Lao · Alex Wong, ,https://arxiv.org/abs/2404.03635,,2404.03635.pdf,WorDepth: Variational Language Prior for Monocular Depth Estimation,"Three-dimensional (3D) reconstruction from a single image is an ill-posed
+problem with inherent ambiguities, i.e. scale. Predicting a 3D scene from text
+description(s) is similarly ill-posed, i.e. spatial arrangements of objects
+described. We investigate the question of whether two inherently ambiguous
+modalities can be used in conjunction to produce metric-scaled reconstructions.
+To test this, we focus on monocular depth estimation, the problem of predicting
+a dense depth map from a single image, but with an additional text caption
+describing the scene. To this end, we begin by encoding the text caption as a
+mean and standard deviation; using a variational framework, we learn the
+distribution of the plausible metric reconstructions of 3D scenes corresponding
+to the text captions as a prior. To ""select"" a specific reconstruction or depth
+map, we encode the given image through a conditional sampler that samples from
+the latent space of the variational text encoder, which is then decoded to the
+output depth map. Our approach is trained alternatingly between the text and
+image branches: in one optimization step, we predict the mean and standard
+deviation from the text description and sample from a standard Gaussian, and in
+the other, we sample using a (image) conditional sampler. Once trained, we
+directly predict depth from the encoded text using the conditional sampler. We
+demonstrate our approach on indoor (NYUv2) and outdoor (KITTI) scenarios, where
+we show that language can consistently improve performance in both.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG', 'cs.MM']"
+Hierarchical Patch Diffusion Models for High-Resolution Video Generation,Ivan Skorokhodov · Willi Menapace · Aliaksandr Siarohin · Sergey Tulyakov, ,http://export.arxiv.org/abs/2310.19512,,2310.19512.pdf,VideoCrafter1: Open Diffusion Models for High-Quality Video Generation,"Video generation has increasingly gained interest in both academia and
+industry. Although commercial tools can generate plausible videos, there is a
+limited number of open-source models available for researchers and engineers.
+In this work, we introduce two diffusion models for high-quality video
+generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V
+models synthesize a video based on a given text input, while I2V models
+incorporate an additional image input. Our proposed T2V model can generate
+realistic and cinematic-quality videos with a resolution of $1024 \times 576$,
+outperforming other open-source T2V models in terms of quality. The I2V model
+is designed to produce videos that strictly adhere to the content of the
+provided reference image, preserving its content, structure, and style. This
+model is the first open-source I2V foundation model capable of transforming a
+given image into a video clip while maintaining content preservation
+constraints. We believe that these open-source video generation models will
+contribute significantly to the technological advancements within the
+community.",cs.CV,['cs.CV']
+End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames,Shuming Liu · Chenlin Zhang · Chen Zhao · Bernard Ghanem, ,https://arxiv.org/abs/2311.17241,,2311.17241.pdf,End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames,"Recently, temporal action detection (TAD) has seen significant performance
+improvement with end-to-end training. However, due to the memory bottleneck,
+only models with limited scales and limited data volumes can afford end-to-end
+training, which inevitably restricts TAD performance. In this paper, we reduce
+the memory consumption for end-to-end training, and manage to scale up the TAD
+backbone to 1 billion parameters and the input video to 1,536 frames, leading
+to significant detection performance. The key to our approach lies in our
+proposed temporal-informative adapter (TIA), which is a novel lightweight
+module that reduces training memory. Using TIA, we free the humongous backbone
+from learning to adapt to the TAD task by only updating the parameters in TIA.
+TIA also leads to better TAD representation by temporally aggregating context
+from adjacent frames throughout the backbone. We evaluate our model across four
+representative datasets. Owing to our efficient design, we are able to train
+end-to-end on VideoMAEv2-giant and achieve 75.4% mAP on THUMOS14, being the
+first end-to-end model to outperform the best feature-based methods. Code is
+available at https://github.com/sming256/AdaTAD.",cs.CV,['cs.CV']
+Dual DETRs for Multi-Label Temporal Action Detection,Yuhan Zhu · Guozhen Zhang · Jing Tan · Gangshan Wu · Limin Wang, ,https://arxiv.org/abs/2404.00653,,2404.00653.pdf,Dual DETRs for Multi-Label Temporal Action Detection,"Temporal Action Detection (TAD) aims to identify the action boundaries and
+the corresponding category within untrimmed videos. Inspired by the success of
+DETR in object detection, several methods have adapted the query-based
+framework to the TAD task. However, these approaches primarily followed DETR to
+predict actions at the instance level (i.e., identify each action by its center
+point), leading to sub-optimal boundary localization. To address this issue, we
+propose a new Dual-level query-based TAD framework, namely DualDETR, to detect
+actions from both instance-level and boundary-level. Decoding at different
+levels requires semantics of different granularity, therefore we introduce a
+two-branch decoding structure. This structure builds distinctive decoding
+processes for different levels, facilitating explicit capture of temporal cues
+and semantics at each level. On top of the two-branch design, we present a
+joint query initialization strategy to align queries from both levels.
+Specifically, we leverage encoder proposals to match queries from each level in
+a one-to-one manner. Then, the matched queries are initialized using position
+and content prior from the matched action proposal. The aligned dual-level
+queries can refine the matched proposal with complementary cues during
+subsequent decoding. We evaluate DualDETR on three challenging multi-label TAD
+benchmarks. The experimental results demonstrate the superior performance of
+DualDETR to the existing state-of-the-art methods, achieving a substantial
+improvement under det-mAP and delivering impressive results under seg-mAP.",cs.CV,['cs.CV']
+LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model,Chenjie Cao · Yunuo Cai · Qiaole Dong · Yikai Wang · Yanwei Fu,https://ewrfcas.github.io/LeftRefill/,https://arxiv.org/html/2405.18416v1,,2405.18416v1.pdf,3D StreetUnveiler with Semantic-Aware 2DGS,"Unveiling an empty street from crowded observations captured by in-car
+cameras is crucial for autonomous driving. However, removing all temporary
+static objects, such as stopped vehicles and standing pedestrians, presents a
+significant challenge. Unlike object-centric 3D inpainting, which relies on
+thorough observation in a small scene, street scenes involve long trajectories
+that differ from previous 3D inpainting tasks. The camera-centric moving
+environment of captured videos further complicates the task due to the limited
+degree and time duration of object observation. To address these obstacles, we
+introduce StreetUnveiler to reconstruct an empty street. StreetUnveiler learns
+a 3D representation of the empty street from crowded observations. Our
+representation is based on the hard-label semantic 2D Gaussian Splatting (2DGS)
+for its scalability and ability to identify Gaussians to be removed. We inpaint
+rendered image after removing unwanted Gaussians to provide pseudo-labels and
+subsequently re-optimize the 2DGS. Given its temporal continuous movement, we
+divide the empty street scene into observed, partial-observed, and unobserved
+regions, which we propose to locate through a rendered alpha map. This
+decomposition helps us to minimize the regions that need to be inpainted. To
+enhance the temporal consistency of the inpainting, we introduce a novel
+time-reversal framework to inpaint frames in reverse order and use later frames
+as references for earlier frames to fully utilize the long-trajectory
+observations. Our experiments conducted on the street scene dataset
+successfully reconstructed a 3D representation of the empty street. The mesh
+representation of the empty street can be extracted for further applications.
+Project page and more visualizations can be found at:
+https://streetunveiler.github.io",cs.CV,['cs.CV']
+3DiffTection: 3D Object Detection with Geometry-aware Diffusion Features,Chenfeng Xu · Huan Ling · Sanja Fidler · Or Litany, ,https://arxiv.org/abs/2311.04391,,2311.04391.pdf,3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features,"We present 3DiffTection, a state-of-the-art method for 3D object detection
+from single images, leveraging features from a 3D-aware diffusion model.
+Annotating large-scale image data for 3D detection is resource-intensive and
+time-consuming. Recently, pretrained large image diffusion models have become
+prominent as effective feature extractors for 2D perception tasks. However,
+these features are initially trained on paired text and image data, which are
+not optimized for 3D tasks, and often exhibit a domain gap when applied to the
+target data. Our approach bridges these gaps through two specialized tuning
+strategies: geometric and semantic. For geometric tuning, we fine-tune a
+diffusion model to perform novel view synthesis conditioned on a single image,
+by introducing a novel epipolar warp operator. This task meets two essential
+criteria: the necessity for 3D awareness and reliance solely on posed image
+data, which are readily available (e.g., from videos) and does not require
+manual annotation. For semantic refinement, we further train the model on
+target data with detection supervision. Both tuning phases employ ControlNet to
+preserve the integrity of the original feature capabilities. In the final step,
+we harness these enhanced capabilities to conduct a test-time prediction
+ensemble across multiple virtual viewpoints. Through our methodology, we obtain
+3D-aware features that are tailored for 3D detection and excel in identifying
+cross-view point correspondences. Consequently, our model emerges as a powerful
+3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a
+precedent in single-view 3D detection by 9.43\% in AP3D on the
+Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data
+efficiency and generalization to cross-domain data.",cs.CV,['cs.CV']
+Unsupervised Feature Learning with Emergent Data-Driven Prototypicality,Yunhui Guo · Youren Zhang · Yubei Chen · Stella X. Yu, ,https://arxiv.org/abs/2307.01421,,2307.01421.pdf,Unsupervised Feature Learning with Emergent Data-Driven Prototypicality,"Given an image set without any labels, our goal is to train a model that maps
+each image to a point in a feature space such that, not only proximity
+indicates visual similarity, but where it is located directly encodes how
+prototypical the image is according to the dataset.
+  Our key insight is to perform unsupervised feature learning in hyperbolic
+instead of Euclidean space, where the distance between points still reflect
+image similarity, and yet we gain additional capacity for representing
+prototypicality with the location of the point: The closer it is to the origin,
+the more prototypical it is. The latter property is simply emergent from
+optimizing the usual metric learning objective: The image similar to many
+training instances is best placed at the center of corresponding points in
+Euclidean space, but closer to the origin in hyperbolic space.
+  We propose an unsupervised feature learning algorithm in Hyperbolic space
+with sphere pACKing. HACK first generates uniformly packed particles in the
+Poincar\'e ball of hyperbolic space and then assigns each image uniquely to
+each particle. Images after congealing are regarded more typical of the dataset
+it belongs to. With our feature mapper simply trained to spread out training
+instances in hyperbolic space, we observe that images move closer to the origin
+with congealing, validating our idea of unsupervised prototypicality discovery.
+We demonstrate that our data-driven prototypicality provides an easy and
+superior unsupervised instance selection to reduce sample complexity, increase
+model generalization with atypical instances and robustness with typical ones.",cs.CV,"['cs.CV', 'cs.AI']"
+Visual In-Context Prompting,Feng Li · Qing Jiang · Hao Zhang · Shilong Liu · Huaizhe Xu · Xueyan Zou · Tianhe Ren · Hongyang Li · Lei Zhang · Chunyuan Li · Jianwei Yang · Jianfeng Gao, ,https://arxiv.org/abs/2311.13601,,2311.13601.pdf,Visual In-Context Prompting,"In-context prompting in large language models (LLMs) has become a prevalent
+approach to improve zero-shot capabilities, but this idea is less explored in
+the vision domain. Existing visual prompting methods focus on referring
+segmentation to segment the most relevant object, falling short of addressing
+many generic vision tasks like open-set segmentation and detection. In this
+paper, we introduce a universal visual in-context prompting framework for both
+tasks. In particular, we build on top of an encoder-decoder architecture, and
+develop a versatile prompt encoder to support a variety of prompts like
+strokes, boxes, and points. We further enhance it to take an arbitrary number
+of reference image segments as the context. Our extensive explorations show
+that the proposed visual in-context prompting elicits extraordinary referring
+and generic segmentation capabilities to refer and detect, yielding competitive
+performance to close-set in-domain datasets and showing promising results on
+many open-set segmentation datasets. By joint training on COCO and SA-1B, our
+model achieves $57.7$ PQ on COCO and $23.2$ PQ on ADE20K. Code will be
+available at https://github.com/UX-Decoder/DINOv.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity,Yuhang Chen · Wenke Huang · Mang Ye,https://github.com/yuhangchen0/FedHEAL,https://arxiv.org/abs/2405.16585,,2405.16585.pdf,Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity,"Federated learning (FL) has emerged as a new paradigm for privacy-preserving
+collaborative training. Under domain skew, the current FL approaches are biased
+and face two fairness problems. 1) Parameter Update Conflict: data disparity
+among clients leads to varying parameter importance and inconsistent update
+directions. These two disparities cause important parameters to potentially be
+overwhelmed by unimportant ones of dominant updates. It consequently results in
+significant performance decreases for lower-performing clients. 2) Model
+Aggregation Bias: existing FL approaches introduce unfair weight allocation and
+neglect domain diversity. It leads to biased model convergence objective and
+distinct performance among domains. We discover a pronounced directional update
+consistency in Federated Learning and propose a novel framework to tackle above
+issues. First, leveraging the discovered characteristic, we selectively discard
+unimportant parameter updates to prevent updates from clients with lower
+performance overwhelmed by unimportant parameters, resulting in fairer
+generalization performance. Second, we propose a fair aggregation objective to
+prevent global model bias towards some domains, ensuring that the global model
+continuously aligns with an unbiased model. The proposed method is generic and
+can be combined with other existing FL methods to enhance fairness.
+Comprehensive experiments on Digits and Office-Caltech demonstrate the high
+fairness and performance of our method.",cs.LG,"['cs.LG', 'cs.AI']"
+Reg-PTQ: Regression-specialized Post-training Quantization for Fully Quantized Object Detector,Yifu Ding · Weilun Feng · Chuyan Chen · Jinyang Guo · Xianglong Liu, ,,,,,,,nan
+MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion,Roy Kapon · Guy Tevet · Daniel Cohen-Or · Amit H. Bermano, ,https://arxiv.org/abs/2310.14729,,2310.14729.pdf,MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion,"We introduce Multi-view Ancestral Sampling (MAS), a method for 3D motion
+generation, using 2D diffusion models that were trained on motions obtained
+from in-the-wild videos. As such, MAS opens opportunities to exciting and
+diverse fields of motion previously under-explored as 3D data is scarce and
+hard to collect. MAS works by simultaneously denoising multiple 2D motion
+sequences representing different views of the same 3D motion. It ensures
+consistency across all views at each diffusion step by combining the individual
+generations into a unified 3D sequence, and projecting it back to the original
+views. We demonstrate MAS on 2D pose data acquired from videos depicting
+professional basketball maneuvers, rhythmic gymnastic performances featuring a
+ball apparatus, and horse races. In each of these domains, 3D motion capture is
+arduous, and yet, MAS generates diverse and realistic 3D sequences. Unlike the
+Score Distillation approach, which optimizes each sample by repeatedly applying
+small fixes, our method uses a sampling process that was constructed for the
+diffusion framework. As we demonstrate, MAS avoids common issues such as
+out-of-domain sampling and mode-collapse. https://guytevet.github.io/mas-page/",cs.CV,"['cs.CV', 'cs.GR']"
+PEEKABOO: Interactive Video Generation via Masked-Diffusion,Yash Jain · Anshul Nasery · Vibhav Vineet · Harkirat Behl, ,https://arxiv.org/abs/2312.07509,,2312.07509.pdf,PEEKABOO: Interactive Video Generation via Masked-Diffusion,"Modern video generation models like Sora have achieved remarkable success in
+producing high-quality videos. However, a significant limitation is their
+inability to offer interactive control to users, a feature that promises to
+open up unprecedented applications and creativity. In this work, we introduce
+the first solution to equip diffusion-based video generation models with
+spatio-temporal control. We present Peekaboo, a novel masked attention module,
+which seamlessly integrates with current video generation models offering
+control without the need for additional training or inference overhead. To
+facilitate future research, we also introduce a comprehensive benchmark for
+interactive video generation. This benchmark offers a standardized framework
+for the community to assess the efficacy of emerging interactive video
+generation models. Our extensive qualitative and quantitative assessments
+reveal that Peekaboo achieves up to a 3.8x improvement in mIoU over baseline
+models, all while maintaining the same latency. Code and benchmark are
+available on the webpage.",cs.CV,"['cs.CV', 'cs.LG']"
+Efficiently Assemble Normalization Layers and Regularization for Federated Domain Generalization,Khiem Le · Tuan Long Ho · Cuong Do · Danh Le-Phuoc · KOK SENG WONG, ,https://arxiv.org/abs/2403.15605,,2403.15605.pdf,Efficiently Assemble Normalization Layers and Regularization for Federated Domain Generalization,"Domain shift is a formidable issue in Machine Learning that causes a model to
+suffer from performance degradation when tested on unseen domains. Federated
+Domain Generalization (FedDG) attempts to train a global model using
+collaborative clients in a privacy-preserving manner that can generalize well
+to unseen clients possibly with domain shift. However, most existing FedDG
+methods either cause additional privacy risks of data leakage or induce
+significant costs in client communication and computation, which are major
+concerns in the Federated Learning paradigm. To circumvent these challenges,
+here we introduce a novel architectural method for FedDG, namely gPerXAN, which
+relies on a normalization scheme working with a guiding regularizer. In
+particular, we carefully design Personalized eXplicitly Assembled Normalization
+to enforce client models selectively filtering domain-specific features that
+are biased towards local data while retaining discrimination of those features.
+Then, we incorporate a simple yet effective regularizer to guide these models
+in directly capturing domain-invariant representations that the global model's
+classifier can leverage. Extensive experimental results on two benchmark
+datasets, i.e., PACS and Office-Home, and a real-world medical dataset,
+Camelyon17, indicate that our proposed method outperforms other existing
+methods in addressing this particular problem.",cs.CV,"['cs.CV', 'cs.LG']"
+S$^2$MVTC: a Simple yet Efficient Scalable Multi-View Tensor Clustering,Zhen Long · Qiyuan Wang · Yazhou Ren · Yipeng Liu · Ce Zhu, ,https://arxiv.org/abs/2403.09107,,2403.09107.pdf,S^2MVTC: a Simple yet Efficient Scalable Multi-View Tensor Clustering,"Anchor-based large-scale multi-view clustering has attracted considerable
+attention for its effectiveness in handling massive datasets. However, current
+methods mainly seek the consensus embedding feature for clustering by exploring
+global correlations between anchor graphs or projection matrices.In this paper,
+we propose a simple yet efficient scalable multi-view tensor clustering
+(S^2MVTC) approach, where our focus is on learning correlations of embedding
+features within and across views. Specifically, we first construct the
+embedding feature tensor by stacking the embedding features of different views
+into a tensor and rotating it. Additionally, we build a novel tensor
+low-frequency approximation (TLFA) operator, which incorporates graph
+similarity into embedding feature learning, efficiently achieving smooth
+representation of embedding features within different views. Furthermore,
+consensus constraints are applied to embedding features to ensure inter-view
+semantic consistency. Experimental results on six large-scale multi-view
+datasets demonstrate that S^2MVTC significantly outperforms state-of-the-art
+algorithms in terms of clustering performance and CPU execution time,
+especially when handling massive data. The code of S^2MVTC is publicly
+available at https://github.com/longzhen520/S2MVTC.",cs.LG,"['cs.LG', 'cs.CV']"
+LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content,Qihao Zhao · Yalun Dai · Hao Li · Wei Hu · Fan Zhang · Jun Liu, ,https://arxiv.org/abs/2403.05854,,2403.05854.pdf,LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content,"Long-tail recognition is challenging because it requires the model to learn
+good representations from tail categories and address imbalances across all
+categories. In this paper, we propose a novel generative and fine-tuning
+framework, LTGC, to handle long-tail recognition via leveraging generated
+content. Firstly, inspired by the rich implicit knowledge in large-scale models
+(e.g., large language models, LLMs), LTGC leverages the power of these models
+to parse and reason over the original tail data to produce diverse tail-class
+content. We then propose several novel designs for LTGC to ensure the quality
+of the generated data and to efficiently fine-tune the model using both the
+generated and original data. The visualization demonstrates the effectiveness
+of the generation module in LTGC, which produces accurate and diverse tail
+data. Additionally, the experimental results demonstrate that our LTGC
+outperforms existing state-of-the-art methods on popular long-tailed
+benchmarks.",cs.CV,['cs.CV']
+BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation,Yunhao Ge · Yihe Tang · Jiashu Xu · Cem Gokmen · Chengshu Li · Wensi Ai · Benjamin Martinez · Arman Aydin · Mona Anvari · Ayush Chakravarthy · Hong-Xing Yu · Josiah Wong · Sanjana Srivastava · Sharon Lee · Shengxin Zha · Laurent Itti · Yunzhu Li · Roberto Martín-Martín · Miao Liu · Pengchuan Zhang · Ruohan Zhang · Li Fei-Fei · Jiajun Wu, ,,,,,,,nan
+Relightful Harmonization: Lighting-aware Portrait Background Replacement,Mengwei Ren · Wei Xiong · Jae Shin Yoon · Zhixin Shu · Jianming Zhang · HyunJoon Jung · Guido Gerig · He Zhang, ,https://arxiv.org/abs/2312.06886,,2312.06886.pdf,Relightful Harmonization: Lighting-aware Portrait Background Replacement,"Portrait harmonization aims to composite a subject into a new background,
+adjusting its lighting and color to ensure harmony with the background scene.
+Existing harmonization techniques often only focus on adjusting the global
+color and brightness of the foreground and ignore crucial illumination cues
+from the background such as apparent lighting direction, leading to unrealistic
+compositions. We introduce Relightful Harmonization, a lighting-aware diffusion
+model designed to seamlessly harmonize sophisticated lighting effect for the
+foreground portrait using any background image. Our approach unfolds in three
+stages. First, we introduce a lighting representation module that allows our
+diffusion model to encode lighting information from target image background.
+Second, we introduce an alignment network that aligns lighting features learned
+from image background with lighting features learned from panorama environment
+maps, which is a complete representation for scene illumination. Last, to
+further boost the photorealism of the proposed method, we introduce a novel
+data simulation pipeline that generates synthetic training pairs from a diverse
+range of natural images, which are used to refine the model. Our method
+outperforms existing benchmarks in visual fidelity and lighting coherence,
+showing superior generalization in real-world testing scenarios, highlighting
+its versatility and practicality.",cs.CV,['cs.CV']
+Image Processing GNN: Breaking Rigidity in Super-Resolution,Yuchuan Tian · Hanting Chen · Chao Xu · Yunhe Wang, ,https://arxiv.org/abs/2310.10413,,2310.10413.pdf,Image super-resolution via dynamic network,"Convolutional neural networks (CNNs) depend on deep network architectures to
+extract accurate information for image super-resolution. However, obtained
+information of these CNNs cannot completely express predicted high-quality
+images for complex scenes. In this paper, we present a dynamic network for
+image super-resolution (DSRNet), which contains a residual enhancement block,
+wide enhancement block, feature refinement block and construction block. The
+residual enhancement block is composed of a residual enhanced architecture to
+facilitate hierarchical features for image super-resolution. To enhance
+robustness of obtained super-resolution model for complex scenes, a wide
+enhancement block achieves a dynamic architecture to learn more robust
+information to enhance applicability of an obtained super-resolution model for
+varying scenes. To prevent interference of components in a wide enhancement
+block, a refinement block utilizes a stacked architecture to accurately learn
+obtained features. Also, a residual learning operation is embedded in the
+refinement block to prevent long-term dependency problem. Finally, a
+construction block is responsible for reconstructing high-quality images.
+Designed heterogeneous architecture can not only facilitate richer structural
+information, but also be lightweight, which is suitable for mobile digital
+devices. Experimental results shows that our method is more competitive in
+terms of performance and recovering time of image super-resolution and
+complexity. The code of DSRNet can be obtained at
+https://github.com/hellloxiaotian/DSRNet.",eess.IV,"['eess.IV', 'cs.CV']"
+TexTile: A Differentiable Metric for Texture Tileability,Carlos Rodriguez-Pardo · Dan Casas · Elena Garces · Jorge Lopez-Moreno,https://mslab.es/projects/TexTile/,,,,,,,nan
+GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning,Ye Yuan · Xueting Li · Yangyi Huang · Shalini De Mello · Koki Nagano · Jan Kautz · Umar Iqbal,https://nvlabs.github.io/GAvatar/,https://arxiv.org/abs/2312.11461,,2312.11461.pdf,GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning,"Gaussian splatting has emerged as a powerful 3D representation that harnesses
+the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
+In this paper, we seek to leverage Gaussian splatting to generate realistic
+animatable avatars from textual descriptions, addressing the limitations (e.g.,
+flexibility and efficiency) imposed by mesh or NeRF-based representations.
+However, a naive application of Gaussian splatting cannot generate high-quality
+animatable avatars and suffers from learning instability; it also cannot
+capture fine avatar geometries and often leads to degenerate body parts. To
+tackle these problems, we first propose a primitive-based 3D Gaussian
+representation where Gaussians are defined inside pose-driven primitives to
+facilitate animation. Second, to stabilize and amortize the learning of
+millions of Gaussians, we propose to use neural implicit fields to predict the
+Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries
+and extract detailed meshes, we propose a novel SDF-based implicit mesh
+learning approach for 3D Gaussians that regularizes the underlying geometries
+and extracts highly detailed textured meshes. Our proposed method, GAvatar,
+enables the large-scale generation of diverse animatable avatars using only
+text prompts. GAvatar significantly surpasses existing methods in terms of both
+appearance and geometry quality, and achieves extremely fast rendering (100
+fps) at 1K resolution.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Instance Tracking in 3D Scenes from Egocentric Videos,Yunhan Zhao · Haoyu Ma · Shu Kong · Charless Fowlkes,https://github.com/IT3DEgo/IT3DEgo/,https://arxiv.org/abs/2312.04117,,2312.04117.pdf,Instance Tracking in 3D Scenes from Egocentric Videos,"Egocentric sensors such as AR/VR devices capture human-object interactions
+and offer the potential to provide task-assistance by recalling 3D locations of
+objects of interest in the surrounding environment. This capability requires
+instance tracking in real-world 3D scenes from egocentric videos (IT3DEgo). We
+explore this problem by first introducing a new benchmark dataset, consisting
+of RGB and depth videos, per-frame camera pose, and instance-level annotations
+in both 2D camera and 3D world coordinates. We present an evaluation protocol
+which evaluates tracking performance in 3D coordinates with two settings for
+enrolling instances to track: (1) single-view online enrollment where an
+instance is specified on-the-fly based on the human wearer's interactions. and
+(2) multi-view pre-enrollment where images of an instance to be tracked are
+stored in memory ahead of time. To address IT3DEgo, we first re-purpose methods
+from relevant areas, e.g., single object tracking (SOT) -- running SOT methods
+to track instances in 2D frames and lifting them to 3D using camera pose and
+depth. We also present a simple method that leverages pretrained segmentation
+and detection models to generate proposals from RGB frames and match proposals
+with enrolled instance images. Perhaps surprisingly, our extensive experiments
+show that our method (with no finetuning) significantly outperforms SOT-based
+approaches. We conclude by arguing that the problem of egocentric instance
+tracking is made easier by leveraging camera pose and using a 3D allocentric
+(world) coordinate representation.",cs.CV,['cs.CV']
+ViT-Lens: Towards Omni-modal Representations,Stan Weixian Lei · Yixiao Ge · Kun Yi · Jianfeng Zhang · Difei Gao · Dylan Sun · Yuying Ge · Ying Shan · Mike Zheng Shou, ,https://arxiv.org/abs/2311.16081,,2311.16081.pdf,ViT-Lens: Towards Omni-modal Representations,"Aiming to advance AI agents, large foundation models significantly improve
+reasoning and instruction execution, yet the current focus on vision and
+language neglects the potential of perceiving diverse modalities in open-world
+environments. However, the success of data-driven vision and language models is
+costly or even infeasible to be reproduced for rare modalities. In this paper,
+we present ViT-Lens-2 that facilitates efficient omni-modal representation
+learning by perceiving novel modalities with a pretrained ViT and aligning them
+to a pre-defined space. Specifically, the modality-specific lens is tuned to
+project any-modal signals to an intermediate embedding space, which are then
+processed by a strong ViT with pre-trained visual knowledge. The encoded
+representations are optimized toward aligning with the modal-independent space,
+pre-defined by off-the-shelf foundation models. ViT-Lens-2 provides a unified
+solution for representation learning of increasing modalities with two
+appealing advantages: (i) Unlocking the great potential of pretrained ViTs to
+novel modalities effectively with efficient data regime; (ii) Enabling emergent
+downstream capabilities through modality alignment and shared ViT parameters.
+We tailor ViT-Lens-2 to learn representations for 3D point cloud, depth, audio,
+tactile and EEG, and set new state-of-the-art results across various
+understanding tasks, such as zero-shot classification. By seamlessly
+integrating ViT-Lens-2 into Multimodal Foundation Models, we enable
+Any-modality to Text and Image Generation in a zero-shot manner. Code and
+models are available at https://github.com/TencentARC/ViT-Lens.",cs.CV,"['cs.CV', 'cs.AI']"
+VideoDistill: Language-aware Vision Distillation for Video Question Answering,Bo Zou · Chao Yang · Yu Qiao · Chengbin Quan · Youjian Zhao, ,https://arxiv.org/abs/2404.00973,,2404.00973.pdf,VideoDistill: Language-aware Vision Distillation for Video Question Answering,"Significant advancements in video question answering (VideoQA) have been made
+thanks to thriving large image-language pretraining frameworks. Although these
+image-language models can efficiently represent both video and language
+branches, they typically employ a goal-free vision perception process and do
+not interact vision with language well during the answer generation, thus
+omitting crucial visual cues. In this paper, we are inspired by the human
+recognition and learning pattern and propose VideoDistill, a framework with
+language-aware (i.e., goal-driven) behavior in both vision perception and
+answer generation process. VideoDistill generates answers only from
+question-related visual embeddings and follows a thinking-observing-answering
+approach that closely resembles human behavior, distinguishing it from previous
+research. Specifically, we develop a language-aware gating mechanism to replace
+the standard cross-attention, avoiding language's direct fusion into visual
+representations. We incorporate this mechanism into two key components of the
+entire framework. The first component is a differentiable sparse sampling
+module, which selects frames containing the necessary dynamics and semantics
+relevant to the questions. The second component is a vision refinement module
+that merges existing spatial-temporal attention layers to ensure the extraction
+of multi-grained visual semantics associated with the questions. We conduct
+experimental evaluations on various challenging video question-answering
+benchmarks, and VideoDistill achieves state-of-the-art performance in both
+general and long-form VideoQA datasets. In Addition, we verify that
+VideoDistill can effectively alleviate the utilization of language shortcut
+solutions in the EgoTaskQA dataset.",cs.CV,['cs.CV']
+Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model,Zelin Peng · Zhengqin Xu · Zhilin Zeng · Lingxi Xie · Qi Tian · Wei Shen, ,https://arxiv.org/abs/2311.17112,,2311.17112.pdf,Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model,"Parameter-efficient fine-tuning (PEFT) is an effective methodology to unleash
+the potential of large foundation models in novel scenarios with limited
+training data. In the computer vision community, PEFT has shown effectiveness
+in image classification, but little research has studied its ability for image
+segmentation. Fine-tuning segmentation models usually require a heavier
+adjustment of parameters to align the proper projection directions in the
+parameter space for new scenarios. This raises a challenge to existing PEFT
+algorithms, as they often inject a limited number of individual parameters into
+each block, which prevents substantial adjustment of the projection direction
+of the parameter space due to the limitation of Hidden Markov Chain along
+blocks. In this paper, we equip PEFT with a cross-block orchestration mechanism
+to enable the adaptation of the Segment Anything Model (SAM) to various
+downstream scenarios. We introduce a novel inter-block communication module,
+which integrates a learnable relation matrix to facilitate communication among
+different coefficient sets of each PEFT block's parameter space. Moreover, we
+propose an intra-block enhancement module, which introduces a linear projection
+head whose weights are generated from a hyper-complex layer, further enhancing
+the impact of the adjustment of projection directions on the entire parameter
+space. Extensive experiments on diverse benchmarks demonstrate that our
+proposed approach consistently improves the segmentation performance
+significantly on novel scenarios with only around 1K additional parameters.",cs.CV,['cs.CV']
+Generating Illustrated Instructions,Sachit Menon · Ishan Misra · Rohit Girdhar, ,https://arxiv.org/abs/2312.04552,,2312.04552.pdf,Generating Illustrated Instructions,"We introduce the new task of generating Illustrated Instructions, i.e.,
+visual instructions customized to a user's needs. We identify desiderata unique
+to this task, and formalize it through a suite of automatic and human
+evaluation metrics, designed to measure the validity, consistency, and efficacy
+of the generations. We combine the power of large language models (LLMs)
+together with strong text-to-image generation diffusion models to propose a
+simple approach called StackedDiffusion, which generates such illustrated
+instructions given text as input. The resulting model strongly outperforms
+baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases,
+users even prefer it to human-generated articles. Most notably, it enables
+various new and exciting applications far beyond what static articles on the
+web can provide, such as personalized instructions complete with intermediate
+steps and pictures in response to a user's individual situation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.MM']"
+3D-LFM: Lifting Foundation Model,Mosam Dabhi · László A. Jeni · Simon Lucey, ,https://arxiv.org/abs/2312.11894,,2312.11894.pdf,3D-LFM: Lifting Foundation Model,"The lifting of 3D structure and camera from 2D landmarks is at the
+cornerstone of the entire discipline of computer vision. Traditional methods
+have been confined to specific rigid objects, such as those in
+Perspective-n-Point (PnP) problems, but deep learning has expanded our
+capability to reconstruct a wide range of object classes (e.g. C3DPO and PAUL)
+with resilience to noise, occlusions, and perspective distortions. All these
+techniques, however, have been limited by the fundamental need to establish
+correspondences across the 3D training data -- significantly limiting their
+utility to applications where one has an abundance of ""in-correspondence"" 3D
+data. Our approach harnesses the inherent permutation equivariance of
+transformers to manage varying number of points per 3D data instance,
+withstands occlusions, and generalizes to unseen categories. We demonstrate
+state of the art performance across 2D-3D lifting task benchmarks. Since our
+approach can be trained across such a broad class of structures we refer to it
+simply as a 3D Lifting Foundation Model (3D-LFM) -- the first of its kind.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction,Bo Zou · Chao Yang · Yu Qiao · Chengbin Quan · Youjian Zhao, ,https://arxiv.org/abs/2404.00913v1,,2404.00913v1.pdf,LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction,"Existing methods to fine-tune LLMs, like Adapter, Prefix-tuning, and LoRA,
+which introduce extra modules or additional input sequences to inject new
+skills or knowledge, may compromise the innate abilities of LLMs. In this
+paper, we propose LLaMA-Excitor, a lightweight method that stimulates the LLMs'
+potential to better follow instructions by gradually paying more attention to
+worthwhile information. Specifically, the LLaMA-Excitor does not directly
+change the intermediate hidden state during the self-attention calculation of
+the transformer structure. We designed the Excitor block as a bypass module for
+the similarity score computation in LLMs' self-attention to reconstruct keys
+and change the importance of values by learnable prompts. LLaMA-Excitor ensures
+a self-adaptive allocation of additional attention to input instructions, thus
+effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on
+low-quality instruction-following datasets. Furthermore, we unify the modeling
+of multi-modal tuning and language-only tuning, extending LLaMA-Excitor to a
+powerful visual instruction follower without the need for complex multi-modal
+alignment. Our proposed approach is evaluated in language-only and multi-modal
+tuning experimental scenarios. Notably, LLaMA-Excitor is the only method that
+maintains basic capabilities while achieving a significant improvement (+6%) on
+the MMLU benchmark. In the visual instruction tuning, we achieve a new
+state-of-the-art image captioning performance of 157.5 CIDEr on MSCOCO, and a
+comparable performance (88.39%) on ScienceQA to cutting-edge models with more
+parameters and extensive vision-language pertaining.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation,Dale Decatur · Itai Lang · Kfir Aberman · Rana Hanocka, ,https://arxiv.org/abs/2311.09571,,2311.09571.pdf,3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation,"In this work we develop 3D Paintbrush, a technique for automatically
+texturing local semantic regions on meshes via text descriptions. Our method is
+designed to operate directly on meshes, producing texture maps which seamlessly
+integrate into standard graphics pipelines. We opt to simultaneously produce a
+localization map (to specify the edit region) and a texture map which conforms
+to it. This synergistic approach improves the quality of both the localization
+and the stylization. To enhance the details and resolution of the textured
+area, we leverage multiple stages of a cascaded diffusion model to supervise
+our local editing technique with generative priors learned from images at
+different resolutions. Our technique, referred to as Cascaded Score
+Distillation (CSD), simultaneously distills scores at multiple resolutions in a
+cascaded fashion, enabling control over both the granularity and global
+understanding of the supervision. We demonstrate the effectiveness of 3D
+Paintbrush to locally texture a variety of shapes within different semantic
+regions. Project page: https://threedle.github.io/3d-paintbrush",cs.GR,"['cs.GR', 'cs.CV']"
+Epistemic Uncertainty Quantification For Pre-trained Neural Networks,Hanjing Wang · Qiang Ji, ,https://arxiv.org/abs/2404.10124,,2404.10124.pdf,Epistemic Uncertainty Quantification For Pre-trained Neural Network,"Epistemic uncertainty quantification (UQ) identifies where models lack
+knowledge. Traditional UQ methods, often based on Bayesian neural networks, are
+not suitable for pre-trained non-Bayesian models. Our study addresses
+quantifying epistemic uncertainty for any pre-trained model, which does not
+need the original training data or model modifications and can ensure broad
+applicability regardless of network architectures or training techniques.
+Specifically, we propose a gradient-based approach to assess epistemic
+uncertainty, analyzing the gradients of outputs relative to model parameters,
+and thereby indicating necessary model adjustments to accurately represent the
+inputs. We first explore theoretical guarantees of gradient-based methods for
+epistemic UQ, questioning the view that this uncertainty is only calculable
+through differences between multiple models. We further improve gradient-driven
+UQ by using class-specific weights for integrating gradients and emphasizing
+distinct contributions from neural network layers. Additionally, we enhance UQ
+accuracy by combining gradient and perturbation methods to refine the
+gradients. We evaluate our approach on out-of-distribution detection,
+uncertainty calibration, and active learning, demonstrating its superiority
+over current state-of-the-art UQ methods for pre-trained models.",cs.LG,"['cs.LG', 'cs.CV']"
+Teeth-SEG: An Efficient Instance Segmentation Framework for Orthodontic Treatment based on Anthropic Prior Knowledge,Bo Zou · Shaofeng Wang · Hao Liu · Gaoyue Sun · Yajie Wang · Zuo FeiFei · Chengbin Quan · Youjian Zhao, ,,https://paperswithcode.com/paper/teeth-seg-an-efficient-instance-segmentation,,,,,nan
+Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer,Junyi Wu · Bin Duan · Weitai Kang · Hao Tang · Yan Yan, ,https://arxiv.org/abs/2403.14552,,2403.14552.pdf,Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer,"While Transformers have rapidly gained popularity in various computer vision
+applications, post-hoc explanations of their internal mechanisms remain largely
+unexplored. Vision Transformers extract visual information by representing
+image regions as transformed tokens and integrating them via attention weights.
+However, existing post-hoc explanation methods merely consider these attention
+weights, neglecting crucial information from the transformed tokens, which
+fails to accurately illustrate the rationales behind the models' predictions.
+To incorporate the influence of token transformation into interpretation, we
+propose TokenTM, a novel post-hoc explanation method that utilizes our
+introduced measurement of token transformation effects. Specifically, we
+quantify token transformation effects by measuring changes in token lengths and
+correlations in their directions pre- and post-transformation. Moreover, we
+develop initialization and aggregation rules to integrate both attention
+weights and token transformation effects across all layers, capturing holistic
+token contributions throughout the model. Experimental results on segmentation
+and perturbation tests demonstrate the superiority of our proposed TokenTM
+compared to state-of-the-art Vision Transformer explanation methods.",cs.CV,['cs.CV']
+Global Latent Neural Rendering,Thomas Tanay · Matteo Maggioni, ,https://arxiv.org/abs/2312.08338,,2312.08338.pdf,Global Latent Neural Rendering,"A recent trend among generalizable novel view synthesis methods is to learn a
+rendering operator acting over single camera rays. This approach is promising
+because it removes the need for explicit volumetric rendering, but it
+effectively treats target images as collections of independent pixels. Here, we
+propose to learn a global rendering operator acting over all camera rays
+jointly. We show that the right representation to enable such rendering is a
+5-dimensional plane sweep volume consisting of the projection of the input
+images on a set of planes facing the target camera. Based on this
+understanding, we introduce our Convolutional Global Latent Renderer (ConvGLR),
+an efficient convolutional architecture that performs the rendering operation
+globally in a low-resolution latent space. Experiments on various datasets
+under sparse and generalizable setups show that our approach consistently
+outperforms existing methods by significant margins.",cs.CV,['cs.CV']
+MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers,Yawar Siddiqui · Antonio Alliegro · Alexey Artemov · Tatiana Tommasi · Daniele Sirigatti · Vladislav Rosov · Angela Dai · Matthias Nießner, ,https://arxiv.org/abs/2311.15475,,2311.15475.pdf,MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers,"We introduce MeshGPT, a new approach for generating triangle meshes that
+reflects the compactness typical of artist-created meshes, in contrast to dense
+triangle meshes extracted by iso-surfacing methods from neural fields. Inspired
+by recent advances in powerful large language models, we adopt a sequence-based
+approach to autoregressively generate triangle meshes as sequences of
+triangles. We first learn a vocabulary of latent quantized embeddings, using
+graph convolutions, which inform these embeddings of the local mesh geometry
+and topology. These embeddings are sequenced and decoded into triangles by a
+decoder, ensuring that they can effectively reconstruct the mesh. A transformer
+is then trained on this learned vocabulary to predict the index of the next
+embedding given previous embeddings. Once trained, our model can be
+autoregressively sampled to generate new triangle meshes, directly generating
+compact meshes with sharp edges, more closely imitating the efficient
+triangulation patterns of human-crafted meshes. MeshGPT demonstrates a notable
+improvement over state of the art mesh generation methods, with a 9% increase
+in shape coverage and a 30-point enhancement in FID scores across various
+categories.",cs.CV,"['cs.CV', 'cs.LG']"
+Video Recognition in Portrait Mode,Mingfei Han · Linjie Yang · Xiaojie Jin · Jiashi Feng · Xiaojun Chang · Heng Wang, ,https://arxiv.org/abs/2312.13746v1,,2312.13746v1.pdf,Video Recognition in Portrait Mode,"The creation of new datasets often presents new challenges for video
+recognition and can inspire novel ideas while addressing these challenges.
+While existing datasets mainly comprise landscape mode videos, our paper seeks
+to introduce portrait mode videos to the research community and highlight the
+unique challenges associated with this video format. With the growing
+popularity of smartphones and social media applications, recognizing portrait
+mode videos is becoming increasingly important. To this end, we have developed
+the first dataset dedicated to portrait mode video recognition, namely
+PortraitMode-400. The taxonomy of PortraitMode-400 was constructed in a
+data-driven manner, comprising 400 fine-grained categories, and rigorous
+quality assurance was implemented to ensure the accuracy of human annotations.
+In addition to the new dataset, we conducted a comprehensive analysis of the
+impact of video format (portrait mode versus landscape mode) on recognition
+accuracy and spatial bias due to the different formats. Furthermore, we
+designed extensive experiments to explore key aspects of portrait mode video
+recognition, including the choice of data augmentation, evaluation procedure,
+the importance of temporal information, and the role of audio modality.
+Building on the insights from our experimental results and the introduction of
+PortraitMode-400, our paper aims to inspire further research efforts in this
+emerging research area.",cs.CV,['cs.CV']
+VGGSfM: Visual Geometry Grounded Deep Structure From Motion,Jianyuan Wang · Nikita Karaev · Christian Rupprecht · David Novotny, ,https://arxiv.org/abs/2312.04563,,2312.04563.pdf,Visual Geometry Grounded Deep Structure From Motion,"Structure-from-motion (SfM) is a long-standing problem in the computer vision
+community, which aims to reconstruct the camera poses and 3D structure of a
+scene from a set of unconstrained 2D images. Classical frameworks solve this
+problem in an incremental manner by detecting and matching keypoints,
+registering images, triangulating 3D points, and conducting bundle adjustment.
+Recent research efforts have predominantly revolved around harnessing the power
+of deep learning techniques to enhance specific elements (e.g., keypoint
+matching), but are still based on the original, non-differentiable pipeline.
+Instead, we propose a new deep pipeline VGGSfM, where each component is fully
+differentiable and thus can be trained in an end-to-end manner. To this end, we
+introduce new mechanisms and simplifications. First, we build on recent
+advances in deep 2D point tracking to extract reliable pixel-accurate tracks,
+which eliminates the need for chaining pairwise matches. Furthermore, we
+recover all cameras simultaneously based on the image and track features
+instead of gradually registering cameras. Finally, we optimise the cameras and
+triangulate 3D points via a differentiable bundle adjustment layer. We attain
+state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism,
+and ETH3D.",cs.CV,"['cs.CV', 'cs.RO']"
+Intrinsic Image Diffusion for Indoor Single-view Material Estimation,Peter Kocsis · Vincent Sitzmann · Matthias Nießner,https://peter-kocsis.github.io/IntrinsicImageDiffusion/,https://arxiv.org/abs/2312.12274,,2312.12274.pdf,Intrinsic Image Diffusion for Indoor Single-view Material Estimation,"We present Intrinsic Image Diffusion, a generative model for appearance
+decomposition of indoor scenes. Given a single input view, we sample multiple
+possible material explanations represented as albedo, roughness, and metallic
+maps. Appearance decomposition poses a considerable challenge in computer
+vision due to the inherent ambiguity between lighting and material properties
+and the lack of real datasets. To address this issue, we advocate for a
+probabilistic formulation, where instead of attempting to directly predict the
+true material properties, we employ a conditional generative model to sample
+from the solution space. Furthermore, we show that utilizing the strong learned
+prior of recent diffusion models trained on large-scale real-world images can
+be adapted to material estimation and highly improves the generalization to
+real images. Our method produces significantly sharper, more consistent, and
+more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on
+PSNR and by $45\%$ better FID score on albedo prediction. We demonstrate the
+effectiveness of our approach through experiments on both synthetic and
+real-world datasets.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'I.4.8; I.2.10']"
+An N-Point Linear Solver for Line and Motion Estimation with Event Cameras,Ling Gao · Daniel Gehrig · Hang Su · Davide Scaramuzza · Laurent Kneip,https://mgaoling.github.io/eventail/,https://arxiv.org/abs/2404.00842v1,,2404.00842v1.pdf,An N-Point Linear Solver for Line and Motion Estimation with Event Cameras,"Event cameras respond primarily to edges--formed by strong gradients--and are
+thus particularly well-suited for line-based motion estimation. Recent work has
+shown that events generated by a single line each satisfy a polynomial
+constraint which describes a manifold in the space-time volume. Multiple such
+constraints can be solved simultaneously to recover the partial linear velocity
+and line parameters. In this work, we show that, with a suitable line
+parametrization, this system of constraints is actually linear in the unknowns,
+which allows us to design a novel linear solver. Unlike existing solvers, our
+linear solver (i) is fast and numerically stable since it does not rely on
+expensive root finding, (ii) can solve both minimal and overdetermined systems
+with more than 5 events, and (iii) admits the characterization of all
+degenerate cases and multiple solutions. The found line parameters are
+singularity-free and have a fixed scale, which eliminates the need for
+auxiliary constraints typically encountered in previous work. To recover the
+full linear camera velocity we fuse observations from multiple lines with a
+novel velocity averaging scheme that relies on a geometrically-motivated
+residual, and thus solves the problem more efficiently than previous schemes
+which minimize an algebraic residual. Extensive experiments in synthetic and
+real-world settings demonstrate that our method surpasses the previous work in
+numerical stability, and operates over 600 times faster.",cs.CV,['cs.CV']
+Benchmarking Segmentation Models with Mask-Preserved Attribute Editing,Zijin Yin · Kongming Liang · Bing Li · Zhanyu Ma · Jun Guo, ,https://arxiv.org/abs/2403.01231,,2403.01231.pdf,Benchmarking Segmentation Models with Mask-Preserved Attribute Editing,"When deploying segmentation models in practice, it is critical to evaluate
+their behaviors in varied and complex scenes. Different from the previous
+evaluation paradigms only in consideration of global attribute variations (e.g.
+adverse weather), we investigate both local and global attribute variations for
+robustness evaluation. To achieve this, we construct a mask-preserved attribute
+editing pipeline to edit visual attributes of real images with precise control
+of structural information. Therefore, the original segmentation labels can be
+reused for the edited images. Using our pipeline, we construct a benchmark
+covering both object and image attributes (e.g. color, material, pattern,
+style). We evaluate a broad variety of semantic segmentation models, spanning
+from conventional close-set models to recent open-vocabulary large models on
+their robustness to different types of variations. We find that both local and
+global attribute variations affect segmentation performances, and the
+sensitivity of models diverges across different variation types. We argue that
+local attributes have the same importance as global attributes, and should be
+considered in the robustness evaluation of segmentation models. Code:
+https://github.com/PRIS-CV/Pascal-EA.",cs.CV,['cs.CV']
+How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?,Yuxin Chen · Zongyang Ma · Ziqi Zhang · Zhongang Qi · Chunfeng Yuan · Bing Li · Junfu Pu · Ying Shan · Xiaojuan Qi · Weiming Hu, ,https://arxiv.org/abs/2310.19654,,2310.19654.pdf,MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval,"Due to the success of large-scale visual-language pretraining (VLP) models
+and the widespread use of image-text retrieval in industry areas, it is now
+critically necessary to reduce the model size and streamline their
+mobile-device deployment. Single- and dual-stream model structures are commonly
+used in image-text retrieval with the goal of closing the semantic gap between
+textual and visual modalities. While single-stream models use deep feature
+fusion to achieve more accurate cross-model alignment, dual-stream models are
+better at offline indexing and fast inference.We propose a Multi-teacher
+Cross-modality Alignment Distillation (MCAD) technique to integrate the
+advantages of single- and dual-stream models. By incorporating the fused
+single-stream features into the image and text features of the dual-stream
+model, we formulate new modified teacher similarity distributions and features.
+Then, we conduct both distribution and feature distillation to boost the
+capability of the student dual-stream model, achieving high retrieval
+performance without increasing inference complexity.Extensive experiments
+demonstrate the remarkable performance and high efficiency of MCAD on
+image-text retrieval tasks. Furthermore, we implement a lightweight CLIP model
+on Snapdragon/Dimensity chips with only $\sim$100M running memory and
+$\sim$8.0ms search latency, achieving the mobile-device application of VLP
+models.",cs.CV,"['cs.CV', 'cs.AI']"
+A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals,Jiangnan Tang · Jingya Wang · Kaiyang Ji · Lan Xu · Jingyi Yu · Ye Shi, ,https://arxiv.org/abs/2404.04890,,2404.04890.pdf,A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals,"Estimating full-body human motion via sparse tracking signals from
+head-mounted displays and hand controllers in 3D scenes is crucial to
+applications in AR/VR. One of the biggest challenges to this task is the
+one-to-many mapping from sparse observations to dense full-body motions, which
+endowed inherent ambiguities. To help resolve this ambiguous problem, we
+introduce a new framework to combine rich contextual information provided by
+scenes to benefit full-body motion tracking from sparse observations. To
+estimate plausible human motions given sparse tracking signals and 3D scenes,
+we develop $\text{S}^2$Fusion, a unified framework fusing \underline{S}cene and
+sparse \underline{S}ignals with a conditional dif\underline{Fusion} model.
+$\text{S}^2$Fusion first extracts the spatial-temporal relations residing in
+the sparse signals via a periodic autoencoder, and then produces time-alignment
+feature embedding as additional inputs. Subsequently, by drawing initial noisy
+motion from a pre-trained prior, $\text{S}^2$Fusion utilizes conditional
+diffusion to fuse scene geometry and sparse tracking signals to generate
+full-body scene-aware motions. The sampling procedure of $\text{S}^2$Fusion is
+further guided by a specially designed scene-penetration loss and
+phase-matching loss, which effectively regularizes the motion of the lower body
+even in the absence of any tracking signals, making the generated motion much
+more plausible and coherent. Extensive experimental results have demonstrated
+that our $\text{S}^2$Fusion outperforms the state-of-the-art in terms of
+estimation quality and smoothness.",cs.CV,['cs.CV']
+Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction,Ziyi Yang · Xinyu Gao · Wen Zhou · Shaohui Jiao · Yuqing Zhang · Xiaogang Jin, ,https://arxiv.org/abs/2309.13101,,2309.13101.pdf,Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction,"Implicit neural representation has paved the way for new approaches to
+dynamic scene reconstruction and rendering. Nonetheless, cutting-edge dynamic
+neural rendering methods rely heavily on these implicit representations, which
+frequently struggle to capture the intricate details of objects in the scene.
+Furthermore, implicit methods have difficulty achieving real-time rendering in
+general dynamic scenes, limiting their use in a variety of tasks. To address
+the issues, we propose a deformable 3D Gaussians Splatting method that
+reconstructs scenes using 3D Gaussians and learns them in canonical space with
+a deformation field to model monocular dynamic scenes. We also introduce an
+annealing smoothing training mechanism with no extra overhead, which can
+mitigate the impact of inaccurate poses on the smoothness of time interpolation
+tasks in real-world datasets. Through a differential Gaussian rasterizer, the
+deformable 3D Gaussians not only achieve higher rendering quality but also
+real-time rendering speed. Experiments show that our method outperforms
+existing methods significantly in terms of both rendering quality and speed,
+making it well-suited for tasks such as novel-view synthesis, time
+interpolation, and real-time rendering.",cs.CV,['cs.CV']
+Towards Generalizable Tumor Synthesis,Qi Chen · Xiaoxi Chen · Haorui Song · Alan L. Yuille · Zhiwei Xiong · Chen Wei · Zongwei Zhou, ,https://arxiv.org/abs/2402.19470,,2402.19470.pdf,Towards Generalizable Tumor Synthesis,"Tumor synthesis enables the creation of artificial tumors in medical images,
+facilitating the training of AI models for tumor detection and segmentation.
+However, success in tumor synthesis hinges on creating visually realistic
+tumors that are generalizable across multiple organs and, furthermore, the
+resulting AI models being capable of detecting real tumors in images sourced
+from different domains (e.g., hospitals). This paper made a progressive stride
+toward generalizable tumor synthesis by leveraging a critical observation:
+early-stage tumors (< 2cm) tend to have similar imaging characteristics in
+computed tomography (CT), whether they originate in the liver, pancreas, or
+kidneys. We have ascertained that generative AI models, e.g., Diffusion Models,
+can create realistic tumors generalized to a range of organs even when trained
+on a limited number of tumor examples from only one organ. Moreover, we have
+shown that AI models trained on these synthetic tumors can be generalized to
+detect and segment real tumors from CT volumes, encompassing a broad spectrum
+of patient demographics, imaging protocols, and healthcare facilities.",eess.IV,"['eess.IV', 'cs.CV']"
+Prompt3D: Random Prompt Assisted Weakly-Supervised 3D Object Detection,Xiaohong Zhang · Huisheng Ye · Jingwen Li · Qinyu Tang · Yuanqi Li · Yanwen Guo · Jie Guo,https://huishengye.github.io/prompt3d/,https://arxiv.org/abs/2312.07530,,2312.07530.pdf,Weakly Supervised 3D Object Detection via Multi-Level Visual Guidance,"Weakly supervised 3D object detection aims to learn a 3D detector with lower
+annotation cost, e.g., 2D labels. Unlike prior work which still relies on few
+accurate 3D annotations, we propose a framework to study how to leverage
+constraints between 2D and 3D domains without requiring any 3D labels.
+Specifically, we employ visual data from three perspectives to establish
+connections between 2D and 3D domains. First, we design a feature-level
+constraint to align LiDAR and image features based on object-aware regions.
+Second, the output-level constraint is developed to enforce the overlap between
+2D and projected 3D box estimations. Finally, the training-level constraint is
+utilized by producing accurate and consistent 3D pseudo-labels that align with
+the visual data. We conduct extensive experiments on the KITTI dataset to
+validate the effectiveness of the proposed three constraints. Without using any
+3D labels, our method achieves favorable performance against state-of-the-art
+approaches and is competitive with the method that uses 500-frame 3D
+annotations. Code and models will be made publicly available at
+https://github.com/kuanchihhuang/VG-W3D.",cs.CV,['cs.CV']
+Discriminative Pattern Calibration Mechanism for Source-Free Domain Adaptation,Haifeng Xia · Siyu Xia · Zhengming Ding, ,https://arxiv.org/abs/2405.02954,,2405.02954.pdf,Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Training,"Source-free domain adaptation (SFDA) aims to adapt a source model trained on
+a fully-labeled source domain to a related but unlabeled target domain. While
+the source model is a key avenue for acquiring target pseudolabels, the
+generated pseudolabels may exhibit source bias. In the conventional SFDA
+pipeline, a large data (e.g. ImageNet) pre-trained feature extractor is used to
+initialize the source model at the start of source training, and subsequently
+discarded. Despite having diverse features important for generalization, the
+pre-trained feature extractor can overfit to the source data distribution
+during source training and forget relevant target domain knowledge. Rather than
+discarding this valuable knowledge, we introduce an integrated framework to
+incorporate pre-trained networks into the target adaptation process. The
+proposed framework is flexible and allows us to plug modern pre-trained
+networks into the adaptation process to leverage their stronger representation
+learning capabilities. For adaptation, we propose the Co-learn algorithm to
+improve target pseudolabel quality collaboratively through the source model and
+a pre-trained feature extractor. Building on the recent success of the
+vision-language model CLIP in zero-shot image recognition, we present an
+extension Co-learn++ to further incorporate CLIP's zero-shot classification
+decisions. We evaluate on 3 benchmark datasets and include more challenging
+scenarios such as open-set, partial-set and open-partial SFDA. Experimental
+results demonstrate that our proposed strategy improves adaptation performance
+and can be successfully integrated with existing SFDA methods.",cs.CV,"['cs.CV', 'cs.LG']"
+Reconstruction-free Cascaded Adaptive Compressive Sensing,Chenxi Qiu · Tao Yue · Xuemei Hu, ,https://arxiv.org/abs/2403.17006,,2403.17006.pdf,Invertible Diffusion Models for Compressed Sensing,"While deep neural networks (NN) significantly advance image compressed
+sensing (CS) by improving reconstruction quality, the necessity of training
+current CS NNs from scratch constrains their effectiveness and hampers rapid
+deployment. Although recent methods utilize pre-trained diffusion models for
+image reconstruction, they struggle with slow inference and restricted
+adaptability to CS. To tackle these challenges, this paper proposes Invertible
+Diffusion Models (IDM), a novel efficient, end-to-end diffusion-based CS
+method. IDM repurposes a large-scale diffusion sampling process as a
+reconstruction model, and finetunes it end-to-end to recover original images
+directly from CS measurements, moving beyond the traditional paradigm of
+one-step noise estimation learning. To enable such memory-intensive end-to-end
+finetuning, we propose a novel two-level invertible design to transform both
+(1) the multi-step sampling process and (2) the noise estimation U-Net in each
+step into invertible networks. As a result, most intermediate features are
+cleared during training to reduce up to 93.8% GPU memory. In addition, we
+develop a set of lightweight modules to inject measurements into noise
+estimator to further facilitate reconstruction. Experiments demonstrate that
+IDM outperforms existing state-of-the-art CS networks by up to 2.64dB in PSNR.
+Compared to the recent diffusion model-based approach DDNM, our IDM achieves up
+to 10.09dB PSNR gain and 14.54 times faster inference.",cs.CV,['cs.CV']
+CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention,Mohammad Sadil Khan · Elona Dupont · Sk Aziz Ali · Kseniya Cherenkova · Anis Kacem · Djamila Aouada,https://cvi2.uni.lu/cadsig-net/,https://arxiv.org/abs/2402.17678,,2402.17678.pdf,CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention,"Reverse engineering in the realm of Computer-Aided Design (CAD) has been a
+longstanding aspiration, though not yet entirely realized. Its primary aim is
+to uncover the CAD process behind a physical object given its 3D scan. We
+propose CAD-SIGNet, an end-to-end trainable and auto-regressive architecture to
+recover the design history of a CAD model represented as a sequence of
+sketch-and-extrusion from an input point cloud. Our model learns
+visual-language representations by layer-wise cross-attention between point
+cloud and CAD language embedding. In particular, a new Sketch instance Guided
+Attention (SGA) module is proposed in order to reconstruct the fine-grained
+details of the sketches. Thanks to its auto-regressive nature, CAD-SIGNet not
+only reconstructs a unique full design history of the corresponding CAD model
+given an input point cloud but also provides multiple plausible design choices.
+This allows for an interactive reverse engineering scenario by providing
+designers with multiple next-step choices along with the design process.
+Extensive experiments on publicly available CAD datasets showcase the
+effectiveness of our approach against existing baseline models in two settings,
+namely, full design history recovery and conditional auto-completion from point
+clouds.",cs.CV,['cs.CV']
+SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation,Junyan Ye · Qiyan Luo · Jinhua Yu · Huaping Zhong · Zhimeng Zheng · Conghui He · Weijia Li, ,https://arxiv.org/abs/2404.02638,,2404.02638.pdf,SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation,"This paper aims at achieving fine-grained building attribute segmentation in
+a cross-view scenario, i.e., using satellite and street-view image pairs. The
+main challenge lies in overcoming the significant perspective differences
+between street views and satellite views. In this work, we introduce SG-BEV, a
+novel approach for satellite-guided BEV fusion for cross-view semantic
+segmentation. To overcome the limitations of existing cross-view projection
+methods in capturing the complete building facade features, we innovatively
+incorporate Bird's Eye View (BEV) method to establish a spatially explicit
+mapping of street-view features. Moreover, we fully leverage the advantages of
+multiple perspectives by introducing a novel satellite-guided reprojection
+module, optimizing the uneven feature distribution issues associated with
+traditional BEV methods. Our method demonstrates significant improvements on
+four cross-view datasets collected from multiple cities, including New York,
+San Francisco, and Boston. On average across these datasets, our method
+achieves an increase in mIOU by 10.13% and 5.21% compared with the
+state-of-the-art satellite-based and cross-view methods. The code and datasets
+of this work will be released at https://github.com/yejy53/SG-BEV.",cs.CV,['cs.CV']
+MSU-4S - The Michigan State University Four Seasons Dataset,Daniel Kent · Mohammed Alyaqoub · Xiaohu Lu · Sayed Khatounabadi · Kookjin Sung · Cole Scheller · Alexander Dalat · Xinwei Guo · Asma Bin Thabit · Roberto Muntaner Whitley · Hayder Radha, ,,https://msuspartans.com/news/2024/5/1/womens-basketball-fralick-adds-four-to-womens-basketball-roster.aspx?print=true,,,,,nan
+Retraining-free Model Quantization via One-Shot Weight-Coupling Learning,Chen Tang · Yuan Meng · Jiacheng Jiang · Shuzhao Xie · Rongwei Lu · Xinzhu Ma · Zhi Wang · Wenwu Zhu, ,https://arxiv.org/abs/2401.01543,,2401.01543.pdf,Retraining-free Model Quantization via One-Shot Weight-Coupling Learning,"Quantization is of significance for compressing the over-parameterized deep
+neural models and deploying them on resource-limited devices. Fixed-precision
+quantization suffers from performance drop due to the limited numerical
+representation ability. Conversely, mixed-precision quantization (MPQ) is
+advocated to compress the model effectively by allocating heterogeneous
+bit-width for layers. MPQ is typically organized into a searching-retraining
+two-stage process. Previous works only focus on determining the optimal
+bit-width configuration in the first stage efficiently, while ignoring the
+considerable time costs in the second stage. However, retraining always
+consumes hundreds of GPU-hours on the cutting-edge GPUs, thus hindering
+deployment efficiency significantly. In this paper, we devise a one-shot
+training-searching paradigm for mixed-precision model compression.
+Specifically, in the first stage, all potential bit-width configurations are
+coupled and thus optimized simultaneously within a set of shared weights.
+However, our observations reveal a previously unseen and severe bit-width
+interference phenomenon among highly coupled weights during optimization,
+leading to considerable performance degradation under a high compression ratio.
+To tackle this problem, we first design a bit-width scheduler to dynamically
+freeze the most turbulent bit-width of layers during training, to ensure the
+rest bit-widths converged properly. Then, taking inspiration from information
+theory, we present an information distortion mitigation technique to align the
+behaviour of the bad-performing bit-widths to the well-performing ones.",cs.CV,['cs.CV']
+Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder,Jinseok Kim · Tae-Kyun Kim, ,https://arxiv.org/abs/2403.10255,,2403.10255.pdf,Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder,"Super-resolution (SR) and image generation are important tasks in computer
+vision and are widely adopted in real-world applications. Most existing
+methods, however, generate images only at fixed-scale magnification and suffer
+from over-smoothing and artifacts. Additionally, they do not offer enough
+diversity of output images nor image consistency at different scales. Most
+relevant work applied Implicit Neural Representation (INR) to the denoising
+diffusion model to obtain continuous-resolution yet diverse and high-quality SR
+results. Since this model operates in the image space, the larger the
+resolution of image is produced, the more memory and inference time is
+required, and it also does not maintain scale-specific consistency. We propose
+a novel pipeline that can super-resolve an input image or generate from a
+random noise a novel image at arbitrary scales. The method consists of a
+pretrained auto-encoder, a latent diffusion model, and an implicit neural
+decoder, and their learning strategies. The proposed method adopts diffusion
+processes in a latent space, thus efficient, yet aligned with output image
+space decoded by MLPs at arbitrary scales. More specifically, our
+arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling
+from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in
+series. The latent diffusion process is learnt by the denoising and the
+alignment losses jointly. Errors in output images are backpropagated via the
+fixed decoder, improving the quality of output images. In the extensive
+experiments using multiple public benchmarks on the two tasks i.e. image
+super-resolution and novel image generation at arbitrary scales, the proposed
+method outperforms relevant methods in metrics of image quality, diversity and
+scale consistency. It is significantly better than the relevant prior-art in
+the inference speed and memory usage.",cs.CV,['cs.CV']
+Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation,Huyong Wang · Huisi Wu · Jing Qin, ,,https://bmcmedimaging.biomedcentral.com/articles/10.1186/s12880-023-01121-3,,,,,nan
+PLGSLAM: Progressive Neural Scene Represenation with Local to Global Bundle Adjustment,Tianchen Deng · Guole Shen · Tong Qin · jianyu wang · Wentao Zhao · Jingchuan Wang · Danwei Wang · Weidong Chen, ,https://arxiv.org/abs/2312.09866,,2312.09866.pdf,PLGSLAM: Progressive Neural Scene Represenation with Local to Global Bundle Adjustment,"Neural implicit scene representations have recently shown encouraging results
+in dense visual SLAM. However, existing methods produce low-quality scene
+reconstruction and low-accuracy localization performance when scaling up to
+large indoor scenes and long sequences. These limitations are mainly due to
+their single, global radiance field with finite capacity, which does not adapt
+to large scenarios. Their end-to-end pose networks are also not robust enough
+with the growth of cumulative errors in large scenes. To this end, we introduce
+PLGSLAM, a neural visual SLAM system capable of high-fidelity surface
+reconstruction and robust camera tracking in real-time. To handle large-scale
+indoor scenes, PLGSLAM proposes a progressive scene representation method which
+dynamically allocates new local scene representation trained with frames within
+a local sliding window. This allows us to scale up to larger indoor scenes and
+improves robustness (even under pose drifts). In local scene representation,
+PLGSLAM utilizes tri-planes for local high-frequency features with multi-layer
+perceptron (MLP) networks for the low-frequency feature, achieving smoothness
+and scene completion in unobserved areas. Moreover, we propose local-to-global
+bundle adjustment method with a global keyframe database to address the
+increased pose drifts on long sequences. Experimental results demonstrate that
+PLGSLAM achieves state-of-the-art scene reconstruction results and tracking
+performance across various datasets and scenarios (both in small and
+large-scale indoor environments).",cs.CV,['cs.CV']
+Bayesian Exploration of Pre-trained Models for Low-shot Image Classification,Yibo Miao · Yu lei · Feng Zhou · Zhijie Deng, ,https://arxiv.org/abs/2404.00312,,2404.00312.pdf,Bayesian Exploration of Pre-trained Models for Low-shot Image Classification,"Low-shot image classification is a fundamental task in computer vision, and
+the emergence of large-scale vision-language models such as CLIP has greatly
+advanced the forefront of research in this field. However, most existing
+CLIP-based methods lack the flexibility to effectively incorporate other
+pre-trained models that encompass knowledge distinct from CLIP. To bridge the
+gap, this work proposes a simple and effective probabilistic model ensemble
+framework based on Gaussian processes, which have previously demonstrated
+remarkable efficacy in processing small data. We achieve the integration of
+prior knowledge by specifying the mean function with CLIP and the kernel
+function with an ensemble of deep kernels built upon various pre-trained
+models. By regressing the classification label directly, our framework enables
+analytical inference, straightforward uncertainty quantification, and
+principled hyper-parameter tuning. Through extensive experiments on standard
+benchmarks, we demonstrate that our method consistently outperforms competitive
+ensemble baselines regarding predictive performance. Additionally, we assess
+the robustness of our method and the quality of the yielded uncertainty
+estimates on out-of-distribution datasets. We also illustrate that our method,
+despite relying on label regression, still enjoys superior model calibration
+compared to most deterministic baselines.",cs.CV,"['cs.CV', 'cs.AI']"
+What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models,Letian Zhang · Xiaotong Zhai · Zhongkai Zhao · Yongshuo Zong · Xin Wen · Bingchen Zhao, ,https://arxiv.org/abs/2310.06627,,2310.06627.pdf,What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models,"Counterfactual reasoning, a fundamental aspect of human cognition, involves
+contemplating alternatives to established facts or past events, significantly
+enhancing our abilities in planning and decision-making. In light of the
+advancements in current multi-modal large language models, we explore their
+effectiveness in counterfactual reasoning. To facilitate this investigation, we
+introduce a novel dataset, C-VQA, specifically designed to test the
+counterfactual reasoning capabilities of modern multi-modal large language
+models. This dataset is constructed by infusing original questions with
+counterfactual presuppositions, spanning various types such as numerical and
+boolean queries. It encompasses a mix of real and synthetic data, representing
+a wide range of difficulty levels. Our thorough evaluations of contemporary
+vision-language models using this dataset have revealed substantial performance
+drops, with some models showing up to a 40% decrease, highlighting a
+significant gap between current models and human-like vision reasoning
+capabilities. We hope our dataset will serve as a vital benchmark for
+evaluating the counterfactual reasoning capabilities of models. Code and
+dataset are publicly available at https://bzhao.me/C-VQA/.",cs.CL,"['cs.CL', 'cs.CV', 'cs.LG']"
+Vision-and-Language Navigation via Causal Learning,Liuyi Wang · Zongtao He · Ronghao Dang · mengjiao shen · Chengju Liu · Qijun Chen, ,https://arxiv.org/abs/2404.10241,,2404.10241.pdf,Vision-and-Language Navigation via Causal Learning,"In the pursuit of robust and generalizable environment perception and
+language understanding, the ubiquitous challenge of dataset bias continues to
+plague vision-and-language navigation (VLN) agents, hindering their performance
+in unseen environments. This paper introduces the generalized cross-modal
+causal transformer (GOAT), a pioneering solution rooted in the paradigm of
+causal inference. By delving into both observable and unobservable confounders
+within vision, language, and history, we propose the back-door and front-door
+adjustment causal learning (BACL and FACL) modules to promote unbiased learning
+by comprehensively mitigating potential spurious correlations. Additionally, to
+capture global confounder features, we propose a cross-modal feature pooling
+(CFP) module supervised by contrastive learning, which is also shown to be
+effective in improving cross-modal representations during pre-training.
+Extensive experiments across multiple VLN datasets (R2R, REVERIE, RxR, and
+SOON) underscore the superiority of our proposed method over previous
+state-of-the-art approaches. Code is available at
+https://github.com/CrystalSixone/VLN-GOAT.",cs.CV,"['cs.CV', 'cs.AI']"
+TIM: A Time Interval Machine for Audio-Visual Action Recognition,Jacob Chalk · Jaesung Huh · Evangelos Kazakos · Andrew Zisserman · Dima Damen,https://jacobchalk.github.io/TIM-Project/,https://arxiv.org/abs/2404.05559,,2404.05559.pdf,TIM: A Time Interval Machine for Audio-Visual Action Recognition,"Diverse actions give rise to rich audio-visual signals in long videos. Recent
+works showcase that the two modalities of audio and video exhibit different
+temporal extents of events and distinct labels. We address the interplay
+between the two modalities in long videos by explicitly modelling the temporal
+extents of audio and visual events. We propose the Time Interval Machine (TIM)
+where a modality-specific time interval poses as a query to a transformer
+encoder that ingests a long video input. The encoder then attends to the
+specified interval, as well as the surrounding context in both modalities, in
+order to recognise the ongoing action.
+  We test TIM on three long audio-visual video datasets: EPIC-KITCHENS,
+Perception Test, and AVE, reporting state-of-the-art (SOTA) for recognition. On
+EPIC-KITCHENS, we beat previous SOTA that utilises LLMs and significantly
+larger pre-training by 2.9% top-1 action recognition accuracy. Additionally, we
+show that TIM can be adapted for action detection, using dense multi-scale
+interval queries, outperforming SOTA on EPIC-KITCHENS-100 for most metrics, and
+showing strong performance on the Perception Test. Our ablations show the
+critical role of integrating the two modalities and modelling their time
+intervals in achieving this performance. Code and models at:
+https://github.com/JacobChalk/TIM",cs.CV,['cs.CV']
+Retrieval-Augmented Open-Vocabulary Object Detection,Jooyeon Kim · Eulrang Cho · Sehyung Kim · Hyunwoo J. Kim, ,https://arxiv.org/abs/2404.05687,,2404.05687.pdf,Retrieval-Augmented Open-Vocabulary Object Detection,"Open-vocabulary object detection (OVD) has been studied with Vision-Language
+Models (VLMs) to detect novel objects beyond the pre-trained categories.
+Previous approaches improve the generalization ability to expand the knowledge
+of the detector, using 'positive' pseudo-labels with additional 'class' names,
+e.g., sock, iPod, and alligator. To extend the previous methods in two aspects,
+we propose Retrieval-Augmented Losses and visual Features (RALF). Our method
+retrieves related 'negative' classes and augments loss functions. Also, visual
+features are augmented with 'verbalized concepts' of classes, e.g., worn on the
+feet, handheld music player, and sharp teeth. Specifically, RALF consists of
+two modules: Retrieval Augmented Losses (RAL) and Retrieval-Augmented visual
+Features (RAF). RAL constitutes two losses reflecting the semantic similarity
+with negative vocabularies. In addition, RAF augments visual features with the
+verbalized concepts from a large language model (LLM). Our experiments
+demonstrate the effectiveness of RALF on COCO and LVIS benchmark datasets. We
+achieve improvement up to 3.4 box AP$_{50}^{\text{N}}$ on novel categories of
+the COCO dataset and 3.6 mask AP$_{\text{r}}$ gains on the LVIS dataset. Code
+is available at https://github.com/mlvlab/RALF .",cs.CV,['cs.CV']
+Continual Motion Prediction Learning Framework via Meta-Representation Learning and Optimal Memory Buffer Retention Strategy,Dae Jun Kang · Dongsuk Kum · Sanmin Kim, ,https://arxiv.org/html/2311.11908v3,,2311.11908v3.pdf,Continual Learning: Applications and the Road Forward,"Continual learning is a subfield of machine learning, which aims to allow
+machine learning models to continuously learn on new data, by accumulating
+knowledge without forgetting what was learned in the past. In this work, we
+take a step back, and ask: ""Why should one care about continual learning in the
+first place?"". We set the stage by examining recent continual learning papers
+published at four major machine learning conferences, and show that
+memory-constrained settings dominate the field. Then, we discuss five open
+problems in machine learning, and even though they might seem unrelated to
+continual learning at first sight, we show that continual learning will
+inevitably be part of their solution. These problems are model editing,
+personalization and specialization, on-device learning, faster (re-)training
+and reinforcement learning. Finally, by comparing the desiderata from these
+unsolved problems and the current assumptions in continual learning, we
+highlight and discuss four future directions for continual learning research.
+We hope that this work offers an interesting perspective on the future of
+continual learning, while displaying its potential value and the paths we have
+to pursue in order to make it successful. This work is the result of the many
+discussions the authors had at the Dagstuhl seminar on Deep Continual Learning,
+in March 2023.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Learning Visual Prompt for Gait Recognition,Kang Ma · Ying Fu · Chunshui Cao · Saihui Hou · Yongzhen Huang · Dezhi Zheng, ,https://arxiv.org/abs/2402.19122,,2402.19122.pdf,BigGait: Learning Gait Representation You Want by Large Vision Models,"Gait recognition stands as one of the most pivotal remote identification
+technologies and progressively expands across research and industry
+communities. However, existing gait recognition methods heavily rely on
+task-specific upstream driven by supervised learning to provide explicit gait
+representations like silhouette sequences, which inevitably introduce expensive
+annotation costs and potential error accumulation. Escaping from this trend,
+this work explores effective gait representations based on the all-purpose
+knowledge produced by task-agnostic Large Vision Models (LVMs) and proposes a
+simple yet efficient gait framework, termed BigGait. Specifically, the Gait
+Representation Extractor (GRE) within BigGait draws upon design principles from
+established gait representations, effectively transforming all-purpose
+knowledge into implicit gait representations without requiring third-party
+supervision signals. Experiments on CCPG, CAISA-B* and SUSTech1K indicate that
+BigGait significantly outperforms the previous methods in both within-domain
+and cross-domain tasks in most cases, and provides a more practical paradigm
+for learning the next-generation gait representation. Finally, we delve into
+prospective challenges and promising directions in LVMs-based gait recognition,
+aiming to inspire future work in this emerging topic. The source code is
+available at https://github.com/ShiqiYu/OpenGait.",cs.CV,['cs.CV']
+Zero-Reference Low-Light Enhancement via Physical Quadruple Priors,Wenjing Wang · Huan Yang · Jianlong Fu · Jiaying Liu,https://daooshee.github.io/QuadPrior-Website/,https://arxiv.org/abs/2403.12933,,2403.12933.pdf,Zero-Reference Low-Light Enhancement via Physical Quadruple Priors,"Understanding illumination and reducing the need for supervision pose a
+significant challenge in low-light enhancement. Current approaches are highly
+sensitive to data usage during training and illumination-specific
+hyper-parameters, limiting their ability to handle unseen scenarios. In this
+paper, we propose a new zero-reference low-light enhancement framework
+trainable solely with normal light images. To accomplish this, we devise an
+illumination-invariant prior inspired by the theory of physical light transfer.
+This prior serves as the bridge between normal and low-light images. Then, we
+develop a prior-to-image framework trained without low-light data. During
+testing, this framework is able to restore our illumination-invariant prior
+back to images, automatically achieving low-light enhancement. Within this
+framework, we leverage a pretrained generative diffusion model for model
+ability, introduce a bypass decoder to handle detail distortion, as well as
+offer a lightweight version for practicality. Extensive experiments demonstrate
+our framework's superiority in various scenarios as well as good
+interpretability, robustness, and efficiency. Code is available on our project
+homepage: http://daooshee.github.io/QuadPrior-Website/",cs.CV,['cs.CV']
+Differentiable Information Bottleneck for Deterministic Multi-view Clustering,Xiaoqiang Yan · Zhixiang Jin · Fengshou Han · Yangdong Ye, ,https://arxiv.org/abs/2403.15681,,2403.15681.pdf,Differentiable Information Bottleneck for Deterministic Multi-view Clustering,"In recent several years, the information bottleneck (IB) principle provides
+an information-theoretic framework for deep multi-view clustering (MVC) by
+compressing multi-view observations while preserving the relevant information
+of multiple views. Although existing IB-based deep MVC methods have achieved
+huge success, they rely on variational approximation and distribution
+assumption to estimate the lower bound of mutual information, which is a
+notoriously hard and impractical problem in high-dimensional multi-view spaces.
+In this work, we propose a new differentiable information bottleneck (DIB)
+method, which provides a deterministic and analytical MVC solution by fitting
+the mutual information without the necessity of variational approximation.
+Specifically, we first propose to directly fit the mutual information of
+high-dimensional spaces by leveraging normalized kernel Gram matrix, which does
+not require any auxiliary neural estimator to estimate the lower bound of
+mutual information. Then, based on the new mutual information measurement, a
+deterministic multi-view neural network with analytical gradients is explicitly
+trained to parameterize IB principle, which derives a deterministic compression
+of input variables from different views. Finally, a triplet consistency
+discovery mechanism is devised, which is capable of mining the feature
+consistency, cluster consistency and joint consistency based on the
+deterministic and compact representations. Extensive experimental results show
+the superiority of our DIB method on 6 benchmarks compared with 13
+state-of-the-art baselines.",cs.IT,"['cs.IT', 'cs.LG', 'math.IT']"
+Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction,Zilin Du · Haoxin Li · Xu Guo · Boyang Li, ,https://arxiv.org/abs/2312.03025,,2312.03025.pdf,Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction,"The task of multimodal relation extraction has attracted significant research
+attention, but progress is constrained by the scarcity of available training
+data. One natural thought is to extend existing datasets with cross-modal
+generative models. In this paper, we consider a novel problem setting, where
+only unimodal data, either text or image, are available during training. We aim
+to train a multimodal classifier from synthetic data that perform well on real
+multimodal test data. However, training with synthetic data suffers from two
+obstacles: lack of data diversity and label information loss. To alleviate the
+issues, we propose Mutual Information-aware Multimodal Iterated Relational dAta
+GEneration (MI2RAGE), which applies Chained Cross-modal Generation (CCG) to
+promote diversity in the generated data and exploits a teacher network to
+select valuable training samples with high mutual information with the
+ground-truth labels. Comparing our method to direct training on synthetic data,
+we observed a significant improvement of 24.06% F1 with synthetic text and
+26.42% F1 with synthetic images. Notably, our best model trained on completely
+synthetic images outperforms prior state-of-the-art models trained on real
+multimodal data by a margin of 3.76% in F1. Our codebase will be made available
+upon acceptance.",cs.AI,"['cs.AI', 'cs.CL', 'cs.CV', 'cs.LG']"
+DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data,Hanrong Ye · Dan Xu, ,https://arxiv.org/abs/2403.15389,,2403.15389.pdf,DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data,"Recently, there has been an increased interest in the practical problem of
+learning multiple dense scene understanding tasks from partially annotated
+data, where each training sample is only labeled for a subset of the tasks. The
+missing of task labels in training leads to low-quality and noisy predictions,
+as can be observed from state-of-the-art methods. To tackle this issue, we
+reformulate the partially-labeled multi-task dense prediction as a pixel-level
+denoising problem, and propose a novel multi-task denoising diffusion framework
+coined as DiffusionMTL. It designs a joint diffusion and denoising paradigm to
+model a potential noisy distribution in the task prediction or feature maps and
+generate rectified outputs for different tasks. To exploit multi-task
+consistency in denoising, we further introduce a Multi-Task Conditioning
+strategy, which can implicitly utilize the complementary nature of the tasks to
+help learn the unlabeled tasks, leading to an improvement in the denoising
+performance of the different tasks. Extensive quantitative and qualitative
+experiments demonstrate that the proposed multi-task denoising diffusion model
+can significantly improve multi-task prediction maps, and outperform the
+state-of-the-art methods on three challenging multi-task benchmarks, under two
+different partial-labeling evaluation settings. The code is available at
+https://prismformore.github.io/diffusionmtl/.",cs.CV,"['cs.CV', 'cs.LG']"
+Retrieval-Augmented Embodied Agents,Yichen Zhu · Zhicai Ou · Xiaofeng Mou · Jian Tang, ,https://arxiv.org/abs/2404.11699,,2404.11699.pdf,Retrieval-Augmented Embodied Agents,"Embodied agents operating in complex and uncertain environments face
+considerable challenges. While some advanced agents handle complex manipulation
+tasks with proficiency, their success often hinges on extensive training data
+to develop their capabilities. In contrast, humans typically rely on recalling
+past experiences and analogous situations to solve new problems. Aiming to
+emulate this human approach in robotics, we introduce the Retrieval-Augmented
+Embodied Agent (RAEA). This innovative system equips robots with a form of
+shared memory, significantly enhancing their performance. Our approach
+integrates a policy retriever, allowing robots to access relevant strategies
+from an external policy memory bank based on multi-modal inputs. Additionally,
+a policy generator is employed to assimilate these strategies into the learning
+process, enabling robots to formulate effective responses to tasks. Extensive
+testing of RAEA in both simulated and real-world scenarios demonstrates its
+superior performance over traditional methods, representing a major leap
+forward in robotic technology.",cs.RO,['cs.RO']
+Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models,Shengqu Cai · Duygu Ceylan · Matheus Gadelha · Chun-Hao P. Huang · Tuanfeng Y. Wang · Gordon Wetzstein,https://primecai.github.io/generative_rendering/,https://arxiv.org/abs/2312.01409,,2312.01409.pdf,Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models,"Traditional 3D content creation tools empower users to bring their
+imagination to life by giving them direct control over a scene's geometry,
+appearance, motion, and camera path. Creating computer-generated videos,
+however, is a tedious manual process, which can be automated by emerging
+text-to-video diffusion models. Despite great promise, video diffusion models
+are difficult to control, hindering a user to apply their own creativity rather
+than amplifying it. To address this challenge, we present a novel approach that
+combines the controllability of dynamic 3D meshes with the expressivity and
+editability of emerging diffusion models. For this purpose, our approach takes
+an animated, low-fidelity rendered mesh as input and injects the ground truth
+correspondence information obtained from the dynamic mesh into various stages
+of a pre-trained text-to-image generation model to output high-quality and
+temporally consistent frames. We demonstrate our approach on various examples
+where motion can be obtained by animating rigged assets or changing the camera
+path.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+Not All Classes Stand on Same Embeddings: Calibrating a Semantic Distance with Metric Tensor,Jae Hyeon Park · Gyoomin Lee · Seunggi Park · Sung In Cho, ,,https://stackoverflow.com/questions/76678783/langchains-chroma-vectordb-similarity-search-with-score-and-vectordb-simil,,,,,nan
+OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation,Qidong Huang · Xiaoyi Dong · Pan Zhang · Bin Wang · Conghui He · Jiaqi Wang · Dahua Lin · Weiming Zhang · Nenghai Yu, ,https://arxiv.org/abs/2311.17911,,2311.17911.pdf,OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation,"Hallucination, posed as a pervasive challenge of multi-modal large language
+models (MLLMs), has significantly impeded their real-world usage that demands
+precise judgment. Existing methods mitigate this issue with either training
+with specific designed data or inferencing with external knowledge from other
+sources, incurring inevitable additional costs. In this paper, we present
+OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a
+Retrospection-Allocation strategy, serving as a nearly free lunch to alleviate
+the hallucination issue without additional data, knowledge, or training. Our
+approach begins with an interesting observation that, most hallucinations are
+closely tied to the knowledge aggregation patterns manifested in the
+self-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on a
+few summary tokens, but not all the previous tokens. Such partial over-trust
+inclination results in the neglecting of image tokens and describes the image
+content with hallucination. Based on the observation, OPERA introduces a
+penalty term on the model logits during the beam-search decoding to mitigate
+the over-trust issue, along with a rollback strategy that retrospects the
+presence of summary tokens in the previously generated tokens, and re-allocate
+the token selection if necessary. With extensive experiments, OPERA shows
+significant hallucination-mitigating performance on different MLLMs and
+metrics, proving its effectiveness and generality. Our code is available at:
+https://github.com/shikiw/OPERA.",cs.CV,['cs.CV']
+Combining Frame and GOP Embeddings for Neural Video Representation,Jens Eirik Saethre · Roberto Azevedo · Christopher Schroers, ,https://arxiv.org/abs/2403.15679,,2403.15679.pdf,DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes,"Implicit neural representations for video (NeRV) have recently become a novel
+way for high-quality video representation. However, existing works employ a
+single network to represent the entire video, which implicitly confuse static
+and dynamic information. This leads to an inability to effectively compress the
+redundant static information and lack the explicitly modeling of global
+temporal-coherent dynamic details. To solve above problems, we propose DS-NeRV,
+which decomposes videos into sparse learnable static codes and dynamic codes
+without the need for explicit optical flow or residual supervision. By setting
+different sampling rates for two codes and applying weighted sum and
+interpolation sampling methods, DS-NeRV efficiently utilizes redundant static
+information while maintaining high-frequency details. Additionally, we design a
+cross-channel attention-based (CCA) fusion module to efficiently fuse these two
+codes for frame decoding. Our approach achieves a high quality reconstruction
+of 31.2 PSNR with only 0.35M parameters thanks to separate static and dynamic
+codes representation and outperforms existing NeRV methods in many downstream
+tasks. Our project website is at https://haoyan14.github.io/DS-NeRV.",cs.CV,"['cs.CV', 'cs.MM']"
+FlowIE：Efficient Image Enhancement via Rectified Flow,Yixuan Zhu · Wenliang Zhao · Ao Li · Yansong Tang · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2405.14677,,2405.14677.pdf,RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance,"Customizing diffusion models to generate identity-preserving images from
+user-provided reference images is an intriguing new problem. The prevalent
+approaches typically require training on extensive domain-specific images to
+achieve identity preservation, which lacks flexibility across different use
+cases. To address this issue, we exploit classifier guidance, a training-free
+technique that steers diffusion models using an existing classifier, for
+personalized image generation. Our study shows that based on a recent rectified
+flow framework, the major limitation of vanilla classifier guidance in
+requiring a special classifier can be resolved with a simple fixed-point
+solution, allowing flexible personalization with off-the-shelf image
+discriminators. Moreover, its solving procedure proves to be stable when
+anchored to a reference flow trajectory, with a convergence guarantee. The
+derived method is implemented on rectified flow with different off-the-shelf
+image discriminators, delivering advantageous personalization results for human
+faces, live subjects, and certain objects. Code is available at
+https://github.com/feifeiobama/RectifID.",cs.CV,"['cs.CV', 'cs.LG']"
+CPR-Coach: Recognizing Composite Error Actions based on Single-class Training,Shunli Wang · Shuaibing Wang · Dingkang Yang · Mingcheng Li · Haopeng Kuang · Xiao Zhao · Liuzhen Su · Peng Zhai · Lihua Zhang, ,https://arxiv.org/abs/2309.11718,,2309.11718.pdf,CPR-Coach: Recognizing Composite Error Actions based on Single-class Training,"The fine-grained medical action analysis task has received considerable
+attention from pattern recognition communities recently, but it faces the
+problems of data and algorithm shortage. Cardiopulmonary Resuscitation (CPR) is
+an essential skill in emergency treatment. Currently, the assessment of CPR
+skills mainly depends on dummies and trainers, leading to high training costs
+and low efficiency. For the first time, this paper constructs a vision-based
+system to complete error action recognition and skill assessment in CPR.
+Specifically, we define 13 types of single-error actions and 74 types of
+composite error actions during external cardiac compression and then develop a
+video dataset named CPR-Coach. By taking the CPR-Coach as a benchmark, this
+paper thoroughly investigates and compares the performance of existing action
+recognition models based on different data modalities. To solve the unavoidable
+Single-class Training & Multi-class Testing problem, we propose a
+humancognition-inspired framework named ImagineNet to improve the model's
+multierror recognition performance under restricted supervision. Extensive
+experiments verify the effectiveness of the framework. We hope this work could
+advance research toward fine-grained medical action analysis and skill
+assessment. The CPR-Coach dataset and the code of ImagineNet are publicly
+available on Github.",cs.CV,"['cs.CV', 'I.5.4']"
+Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion,Zixian Gao · Xun Jiang · Xing Xu · Fumin Shen · Yujie Li · Heng Tao Shen, ,https://arxiv.org/abs/2307.16121,,2307.16121.pdf,Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving,"Multi-modal fusion has shown initial promising results for object detection
+of autonomous driving perception. However, many existing fusion schemes do not
+consider the quality of each fusion input and may suffer from adverse
+conditions on one or more sensors. While predictive uncertainty has been
+applied to characterize single-modal object detection performance at run time,
+incorporating uncertainties into the multi-modal fusion still lacks effective
+solutions due primarily to the uncertainty's cross-modal incomparability and
+distinct sensitivities to various adverse conditions. To fill this gap, this
+paper proposes Uncertainty-Encoded Mixture-of-Experts (UMoE) that explicitly
+incorporates single-modal uncertainties into LiDAR-camera fusion. UMoE uses
+individual expert network to process each sensor's detection result together
+with encoded uncertainty. Then, the expert networks' outputs are analyzed by a
+gating network to determine the fusion weights. The proposed UMoE module can be
+integrated into any proposal fusion pipeline. Evaluation shows that UMoE
+achieves a maximum of 10.67%, 3.17%, and 5.40% performance gain compared with
+the state-of-the-art proposal-level multi-modal object detectors under extreme
+weather, adversarial, and blinding attack scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+Unsupervised Deep Unrolling Networks for Phase Unwrapping,Zhile Chen · Yuhui Quan · Hui Ji, ,,https://ieeexplore.ieee.org/document/10520881,,,,,nan
+Transductive Zero-Shot $\&$ Few-Shot CLIP,Ségolène Martin · Yunshi HUANG · Fereshteh Shakeri · Jean-Christophe Pesquet · Ismail Ben Ayed, ,https://arxiv.org/abs/2405.18437,,2405.18437.pdf,Transductive Zero-Shot and Few-Shot CLIP,"Transductive inference has been widely investigated in few-shot image
+classification, but completely overlooked in the recent, fast growing
+literature on adapting vision-langage models like CLIP. This paper addresses
+the transductive zero-shot and few-shot CLIP classification challenge, in which
+inference is performed jointly across a mini-batch of unlabeled query samples,
+rather than treating each instance independently. We initially construct
+informative vision-text probability features, leading to a classification
+problem on the unit simplex set. Inspired by Expectation-Maximization (EM), our
+optimization-based classification objective models the data probability
+distribution for each class using a Dirichlet law. The minimization problem is
+then tackled with a novel block Majorization-Minimization algorithm, which
+simultaneously estimates the distribution parameters and class assignments.
+Extensive numerical experiments on 11 datasets underscore the benefits and
+efficacy of our batch inference approach.On zero-shot tasks with test batches
+of 75 samples, our approach yields near 20% improvement in ImageNet accuracy
+over CLIP's zero-shot performance. Additionally, we outperform state-of-the-art
+methods in the few-shot setting. The code is available at:
+https://github.com/SegoleneMartin/transductive-CLIP.",cs.CV,"['cs.CV', 'cs.AI']"
+Just Add $\pi$! Pose Induced Video Transformers for Understanding Activities of Daily Living,Dominick Reilly · Srijan Das, ,https://arxiv.org/abs/2311.18840,,2311.18840.pdf,Just Add $π$! Pose Induced Video Transformers for Understanding Activities of Daily Living,"Video transformers have become the de facto standard for human action
+recognition, yet their exclusive reliance on the RGB modality still limits
+their adoption in certain domains. One such domain is Activities of Daily
+Living (ADL), where RGB alone is not sufficient to distinguish between visually
+similar actions, or actions observed from multiple viewpoints. To facilitate
+the adoption of video transformers for ADL, we hypothesize that the
+augmentation of RGB with human pose information, known for its sensitivity to
+fine-grained motion and multiple viewpoints, is essential. Consequently, we
+introduce the first Pose Induced Video Transformer: PI-ViT (or $\pi$-ViT), a
+novel approach that augments the RGB representations learned by video
+transformers with 2D and 3D pose information. The key elements of $\pi$-ViT are
+two plug-in modules, 2D Skeleton Induction Module and 3D Skeleton Induction
+Module, that are responsible for inducing 2D and 3D pose information into the
+RGB representations. These modules operate by performing pose-aware auxiliary
+tasks, a design choice that allows $\pi$-ViT to discard the modules during
+inference. Notably, $\pi$-ViT achieves the state-of-the-art performance on
+three prominent ADL datasets, encompassing both real-world and large-scale
+RGB-D datasets, without requiring poses or additional computational overhead at
+inference.",cs.CV,['cs.CV']
+Specularity Factorization for Low Light Enhancement,Saurabh Saini · P. J. Narayanan, ,https://arxiv.org/abs/2404.01998,,2404.01998.pdf,Specularity Factorization for Low-Light Enhancement,"We present a new additive image factorization technique that treats images to
+be composed of multiple latent specular components which can be simply
+estimated recursively by modulating the sparsity during decomposition. Our
+model-driven {\em RSFNet} estimates these factors by unrolling the optimization
+into network layers requiring only a few scalars to be learned. The resultant
+factors are interpretable by design and can be fused for different image
+enhancement tasks via a network or combined directly by the user in a
+controllable fashion. Based on RSFNet, we detail a zero-reference Low Light
+Enhancement (LLE) application trained without paired or unpaired supervision.
+Our system improves the state-of-the-art performance on standard benchmarks and
+achieves better generalization on multiple other datasets. We also integrate
+our factors with other task specific fusion networks for applications like
+deraining, deblurring and dehazing with negligible overhead thereby
+highlighting the multi-domain and multi-task generalizability of our proposed
+RSFNet. The code and data is released for reproducibility on the project
+homepage.",cs.CV,"['cs.CV', 'cs.LG']"
+Draw Step by Step: Reconstructing CAD Construction Sequences from Point Clouds via Multimodal Diffusion.,Weijian Ma · Shuaiqi Chen · Yunzhong Lou · Xueyang Li · Xiangdong Zhou, ,https://arxiv.org/abs/2405.15188,,2405.15188.pdf,PS-CAD: Local Geometry Guidance via Prompting and Selection for CAD Reconstruction,"Reverse engineering CAD models from raw geometry is a classic but challenging
+research problem. In particular, reconstructing the CAD modeling sequence from
+point clouds provides great interpretability and convenience for editing. To
+improve upon this problem, we introduce geometric guidance into the
+reconstruction network. Our proposed model, PS-CAD, reconstructs the CAD
+modeling sequence one step at a time. At each step, we provide two forms of
+geometric guidance. First, we provide the geometry of surfaces where the
+current reconstruction differs from the complete model as a point cloud. This
+helps the framework to focus on regions that still need work. Second, we use
+geometric analysis to extract a set of planar prompts, that correspond to
+candidate surfaces where a CAD extrusion step could be started. Our framework
+has three major components. Geometric guidance computation extracts the two
+types of geometric guidance. Single-step reconstruction computes a single
+candidate CAD modeling step for each provided prompt. Single-step selection
+selects among the candidate CAD modeling steps. The process continues until the
+reconstruction is completed. Our quantitative results show a significant
+improvement across all metrics. For example, on the dataset DeepCAD, PS-CAD
+improves upon the best published SOTA method by reducing the geometry errors
+(CD and HD) by 10%, and the structural error (ECD metric) by about 15%.",cs.CV,['cs.CV']
+Logarithmic Lenses: Exploring Log RGB Data for Image Classification,Bruce Maxwell · Bruce Maxwell · Sumegha Singhania · Avnish Patel · Rahul Kumar · Heather Fryling · Sihan Li · Haonan Sun · Ping He · Zewen Li, ,,https://medium.com/@adjileyeb/unlocking-visual-insights-applying-the-logit-lens-to-image-data-with-vision-transformers-b99cb70dd704,,,,,nan
+D$^4$M: Dataset Distillation via Disentangled Diffusion Model,Duo Su · Junjie Hou · Weizhi Gao · Yingjie Tian · Bowen Tang, ,https://arxiv.org/abs/2403.03881,,2403.03881.pdf,Latent Dataset Distillation with Diffusion Models,"The efficacy of machine learning has traditionally relied on the availability
+of increasingly larger datasets. However, large datasets pose storage
+challenges and contain non-influential samples, which could be ignored during
+training without impacting the final accuracy of the model. In response to
+these limitations, the concept of distilling the information on a dataset into
+a condensed set of (synthetic) samples, namely a distilled dataset, emerged.
+One crucial aspect is the selected architecture (usually ConvNet) for linking
+the original and synthetic datasets. However, the final accuracy is lower if
+the employed model architecture differs from the model used during
+distillation. Another challenge is the generation of high-resolution images,
+e.g., 128x128 and higher. In this paper, we propose Latent Dataset Distillation
+with Diffusion Models (LD3M) that combine diffusion in latent space with
+dataset distillation to tackle both challenges. LD3M incorporates a novel
+diffusion process tailored for dataset distillation, which improves the
+gradient norms for learning synthetic images. By adjusting the number of
+diffusion steps, LD3M also offers a straightforward way of controlling the
+trade-off between speed and accuracy. We evaluate our approach in several
+ImageNet subsets and for high-resolution images (128x128 and 256x256). As a
+result, LD3M consistently outperforms state-of-the-art distillation techniques
+by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Probing the 3D Awareness of Visual Foundation Models,Mohamed El Banani · Amit Raj · Kevis-kokitsi Maninis · Abhishek Kar · Yuanzhen Li · Michael Rubinstein · Deqing Sun · Leonidas Guibas · Justin Johnson · Varun Jampani, ,https://arxiv.org/abs/2404.08636,,2404.08636.pdf,Probing the 3D Awareness of Visual Foundation Models,"Recent advances in large-scale pretraining have yielded visual foundation
+models with strong capabilities. Not only can recent models generalize to
+arbitrary images for their training task, their intermediate representations
+are useful for other visual tasks such as detection and segmentation. Given
+that such models can classify, delineate, and localize objects in 2D, we ask
+whether they also represent their 3D structure? In this work, we analyze the 3D
+awareness of visual foundation models. We posit that 3D awareness implies that
+representations (1) encode the 3D structure of the scene and (2) consistently
+represent the surface across views. We conduct a series of experiments using
+task-specific probes and zero-shot inference procedures on frozen features. Our
+experiments reveal several limitations of the current models. Our code and
+analysis can be found at https://github.com/mbanani/probe3d.",cs.CV,['cs.CV']
+HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models,Nataniel Ruiz · Yuanzhen Li · Varun Jampani · Wei Wei · Tingbo Hou · Yael Pritch · Neal Wadhwa · Michael Rubinstein · Kfir Aberman, ,https://arxiv.org/abs/2307.06949,,2307.06949.pdf,HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models,"Personalization has emerged as a prominent aspect within the field of
+generative AI, enabling the synthesis of individuals in diverse contexts and
+styles, while retaining high-fidelity to their identities. However, the process
+of personalization presents inherent challenges in terms of time and memory
+requirements. Fine-tuning each personalized model needs considerable GPU time
+investment, and storing a personalized model per subject can be demanding in
+terms of storage capacity. To overcome these challenges, we propose
+HyperDreamBooth-a hypernetwork capable of efficiently generating a small set of
+personalized weights from a single image of a person. By composing these
+weights into the diffusion model, coupled with fast finetuning, HyperDreamBooth
+can generate a person's face in various contexts and styles, with high subject
+details while also preserving the model's crucial knowledge of diverse styles
+and semantic modifications. Our method achieves personalization on faces in
+roughly 20 seconds, 25x faster than DreamBooth and 125x faster than Textual
+Inversion, using as few as one reference image, with the same quality and style
+diversity as DreamBooth. Also our method yields a model that is 10000x smaller
+than a normal DreamBooth model. Project page: https://hyperdreambooth.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams,Liao Wang · Kaixin Yao · Chengcheng Guo · Zhirui Zhang · Qiang Hu · Jingyi Yu · Lan Xu · Minye Wu, ,https://arxiv.org/abs/2312.01407,,2312.01407.pdf,VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams,"Neural Radiance Fields (NeRFs) excel in photorealistically rendering static
+scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous
+devices remains challenging, due to data storage and computational constraints.
+In this paper, we introduce VideoRF, the first approach to enable real-time
+streaming and rendering of dynamic radiance fields on mobile platforms. At the
+core is a serialized 2D feature image stream representing the 4D radiance field
+all in one. We introduce a tailored training scheme directly applied to this 2D
+domain to impose the temporal and spatial redundancy of the feature image
+stream. By leveraging the redundancy, we show that the feature image stream can
+be efficiently compressed by 2D video codecs, which allows us to exploit video
+hardware accelerators to achieve real-time decoding. On the other hand, based
+on the feature image stream, we propose a novel rendering pipeline for VideoRF,
+which has specialized space mappings to query radiance properties efficiently.
+Paired with a deferred shading model, VideoRF has the capability of real-time
+rendering on mobile devices thanks to its efficiency. We have developed a
+real-time interactive player that enables online streaming and rendering of
+dynamic scenes, offering a seamless and immersive free-viewpoint experience
+across a range of devices, from desktops to mobile phones.",cs.CV,['cs.CV']
+GLaMM: Pixel Grounding Large Multimodal Model,Hanoona Rasheed · Muhammad Maaz · Sahal Shaji Mullappilly · Abdelrahman Shaker · Salman Khan · Hisham Cholakkal · Rao Anwer · Eric P. Xing · Ming-Hsuan Yang · Fahad Shahbaz Khan, ,https://arxiv.org/abs/2311.03356v1,,2311.03356v1.pdf,GLaMM: Pixel Grounding Large Multimodal Model,"Large Multimodal Models (LMMs) extend Large Language Models to the vision
+domain. Initial efforts towards LMMs used holistic images and text prompts to
+generate ungrounded textual responses. Very recently, region-level LMMs have
+been used to generate visually grounded responses. However, they are limited to
+only referring a single object category at a time, require users to specify the
+regions in inputs, or cannot offer dense pixel-wise object grounding. In this
+work, we present Grounding LMM (GLaMM), the first model that can generate
+natural language responses seamlessly intertwined with corresponding object
+segmentation masks. GLaMM not only grounds objects appearing in the
+conversations but is flexible enough to accept both textual and optional visual
+prompts (region of interest) as input. This empowers users to interact with the
+model at various levels of granularity, both in textual and visual domains. Due
+to the lack of standard benchmarks for the novel setting of generating visually
+grounded detailed conversations, we introduce a comprehensive evaluation
+protocol with our curated grounded conversations. Our proposed Grounded
+Conversation Generation (GCG) task requires densely grounded concepts in
+natural scenes at a large-scale. To this end, we propose a densely annotated
+Grounding-anything Dataset (GranD) using our proposed automated annotation
+pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
+regions available with segmentation masks. Besides GCG, GLaMM also performs
+effectively on several downstream tasks e.g., referring expression
+segmentation, image and region-level captioning and vision-language
+conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.",cs.CV,"['cs.CV', 'cs.AI']"
+pix2gestalt: Amodal Segmentation by Synthesizing Wholes,Ege Ozguroglu · Ruoshi Liu · Dídac Surís · Dian Chen · Achal Dave · Pavel Tokmakov · Carl Vondrick, ,https://arxiv.org/abs/2401.14398,,2401.14398.pdf,pix2gestalt: Amodal Segmentation by Synthesizing Wholes,"We introduce pix2gestalt, a framework for zero-shot amodal segmentation,
+which learns to estimate the shape and appearance of whole objects that are
+only partially visible behind occlusions. By capitalizing on large-scale
+diffusion models and transferring their representations to this task, we learn
+a conditional diffusion model for reconstructing whole objects in challenging
+zero-shot cases, including examples that break natural and physical priors,
+such as art. As training data, we use a synthetically curated dataset
+containing occluded objects paired with their whole counterparts. Experiments
+show that our approach outperforms supervised baselines on established
+benchmarks. Our model can furthermore be used to significantly improve the
+performance of existing object recognition and 3D reconstruction methods in the
+presence of occlusions.",cs.CV,"['cs.CV', 'cs.LG']"
+LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimation,Xuecan Wang · Shibang Xiao · Xiaohui Liang, ,https://arxiv.org/abs/2404.03925,,2404.03925.pdf,LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimation,"We present a lightweight solution for estimating spatially-coherent indoor
+lighting from a single RGB image. Previous methods for estimating illumination
+using volumetric representations have overlooked the sparse distribution of
+light sources in space, necessitating substantial memory and computational
+resources for achieving high-quality results. We introduce a unified, voxel
+octree-based illumination estimation framework to produce 3D spatially-coherent
+lighting. Additionally, a differentiable voxel octree cone tracing rendering
+layer is proposed to eliminate regular volumetric representation throughout the
+entire process and ensure the retention of features across different frequency
+domains. This reduction significantly decreases spatial usage and required
+floating-point operations without substantially compromising precision.
+Experimental results demonstrate that our approach achieves high-quality
+coherent estimation with minimal cost compared to previous methods.",cs.CV,['cs.CV']
+3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis,Zhicheng Lu · xiang guo · Le Hui · Tianrui Chen · Min Yang · Xiao Tang · feng zhu · Yuchao Dai, ,https://arxiv.org/abs/2404.06270,,2404.06270.pdf,3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis,"In this paper, we propose a 3D geometry-aware deformable Gaussian Splatting
+method for dynamic view synthesis. Existing neural radiance fields (NeRF) based
+solutions learn the deformation in an implicit manner, which cannot incorporate
+3D scene geometry. Therefore, the learned deformation is not necessarily
+geometrically coherent, which results in unsatisfactory dynamic view synthesis
+and 3D dynamic reconstruction. Recently, 3D Gaussian Splatting provides a new
+representation of the 3D scene, building upon which the 3D geometry could be
+exploited in learning the complex 3D deformation. Specifically, the scenes are
+represented as a collection of 3D Gaussian, where each 3D Gaussian is optimized
+to move and rotate over time to model the deformation. To enforce the 3D scene
+geometry constraint during deformation, we explicitly extract 3D geometry
+features and integrate them in learning the 3D deformation. In this way, our
+solution achieves 3D geometry-aware deformation modeling, which enables
+improved dynamic view synthesis and 3D dynamic reconstruction. Extensive
+experimental results on both synthetic and real datasets prove the superiority
+of our solution, which achieves new state-of-the-art performance.
+  The project is available at https://npucvr.github.io/GaGS/",cs.CV,['cs.CV']
+PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection,Qihang Ma · Zhizhong Zhang · Xin Tan · Yanyun Qu · Chengwei Chen · Yuan Xie · Lizhuang Ma, ,https://arxiv.org/abs/2404.05231,,2404.05231.pdf,PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection,"The vision-language model has brought great improvement to few-shot
+industrial anomaly detection, which usually needs to design of hundreds of
+prompts through prompt engineering. For automated scenarios, we first use
+conventional prompt learning with many-class paradigm as the baseline to
+automatically learn prompts but found that it can not work well in one-class
+anomaly detection. To address the above problem, this paper proposes a
+one-class prompt learning method for few-shot anomaly detection, termed
+PromptAD. First, we propose semantic concatenation which can transpose normal
+prompts into anomaly prompts by concatenating normal prompts with anomaly
+suffixes, thus constructing a large number of negative samples used to guide
+prompt learning in one-class setting. Furthermore, to mitigate the training
+challenge caused by the absence of anomaly images, we introduce the concept of
+explicit anomaly margin, which is used to explicitly control the margin between
+normal prompt features and anomaly prompt features through a hyper-parameter.
+For image-level/pixel-level anomaly detection, PromptAD achieves first place in
+11/12 few-shot settings on MVTec and VisA.",cs.CV,['cs.CV']
+Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation,Dong Lao · Congli Wang · Alex Wong · Stefano Soatto, ,,https://www.semanticscholar.org/paper/Diffeomorphic-Template-Registration-for-Atmospheric-Lao-Wang/d03a9da146a21840a76c6a42b1a1572736fe5a14/figure/2,,,,,nan
+From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers,Swaminathan Gurumurthy · Karnik Ram · Bingqing Chen · Zachary Manchester · Zico Kolter, ,https://arxiv.org/abs/2307.08873,,2307.08873.pdf,An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient,"Restricting the variance of a policy's return is a popular choice in
+risk-averse Reinforcement Learning (RL) due to its clear mathematical
+definition and easy interpretability. Traditional methods directly restrict the
+total return variance. Recent methods restrict the per-step reward variance as
+a proxy. We thoroughly examine the limitations of these variance-based methods,
+such as sensitivity to numerical scale and hindering of policy learning, and
+propose to use an alternative risk measure, Gini deviation, as a substitute. We
+study various properties of this new risk measure and derive a policy gradient
+algorithm to minimize it. Empirical evaluation in domains where risk-aversion
+can be clearly defined, shows that our algorithm can mitigate the limitations
+of variance-based risk measures and achieves high return with low risk in terms
+of variance and Gini deviation when others fail to learn a reasonable policy.",cs.LG,"['cs.LG', 'cs.AI']"
+HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D,Sangmin Woo · byeongjun park · Hyojun Go · Jin-Young Kim · Changick Kim, ,,https://github.com/byeongjun-park/HarmonyView,,,,,nan
+Learning SO(3)-Invariant Semantic Correspondence via Local Shape Transform,Chunghyun Park · Seungwook Kim · Jaesik Park · Minsu Cho, ,https://arxiv.org/abs/2404.11156,,2404.11156.pdf,Learning SO(3)-Invariant Semantic Correspondence via Local Shape Transform,"Establishing accurate 3D correspondences between shapes stands as a pivotal
+challenge with profound implications for computer vision and robotics. However,
+existing self-supervised methods for this problem assume perfect input shape
+alignment, restricting their real-world applicability. In this work, we
+introduce a novel self-supervised Rotation-Invariant 3D correspondence learner
+with Local Shape Transform, dubbed RIST, that learns to establish dense
+correspondences between shapes even under challenging intra-class variations
+and arbitrary orientations. Specifically, RIST learns to dynamically formulate
+an SO(3)-invariant local shape transform for each point, which maps the
+SO(3)-equivariant global shape descriptor of the input shape to a local shape
+descriptor. These local shape descriptors are provided as inputs to our decoder
+to facilitate point cloud self- and cross-reconstruction. Our proposed
+self-supervised training pipeline encourages semantically corresponding points
+from different shapes to be mapped to similar local shape descriptors, enabling
+RIST to establish dense point-wise correspondences. RIST demonstrates
+state-of-the-art performances on 3D part label transfer and semantic keypoint
+transfer given arbitrarily rotated point cloud pairs, outperforming existing
+methods by significant margins.",cs.CV,['cs.CV']
+BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation,Jiahao Lu · Jiacheng Deng · Tianzhu Zhang, ,https://arxiv.org/abs/2403.15019,,2403.15019.pdf,BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation,"3D instance segmentation (3DIS) is a crucial task, but point-level
+annotations are tedious in fully supervised settings. Thus, using bounding
+boxes (bboxes) as annotations has shown great potential. The current mainstream
+approach is a two-step process, involving the generation of pseudo-labels from
+box annotations and the training of a 3DIS network with the pseudo-labels.
+However, due to the presence of intersections among bboxes, not every point has
+a determined instance label, especially in overlapping areas. To generate
+higher quality pseudo-labels and achieve more precise weakly supervised 3DIS
+results, we propose the Box-Supervised Simulation-assisted Mean Teacher for 3D
+Instance Segmentation (BSNet), which devises a novel pseudo-labeler called
+Simulation-assisted Transformer. The labeler consists of two main components.
+The first is Simulation-assisted Mean Teacher, which introduces Mean Teacher
+for the first time in this task and constructs simulated samples to assist the
+labeler in acquiring prior knowledge about overlapping areas. To better model
+local-global structure, we also propose Local-Global Aware Attention as the
+decoder for teacher and student labelers. Extensive experiments conducted on
+the ScanNetV2 and S3DIS datasets verify the superiority of our designs. Code is
+available at
+\href{https://github.com/peoplelu/BSNet}{https://github.com/peoplelu/BSNet}.",cs.CV,['cs.CV']
+Motion Diversification Networks,Hee Jae Kim · Eshed Ohn-Bar, ,,https://www.kdramastars.com/articles/131362/20230922/moving-actor-stuns-viewers-unrecognizable-transformation-villain.htm,,,,,nan
+PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks,Marina Neseem · Conor McCullough · Randy Hsin · Chas Leichner · Shan Li · In Suk Chong · Andrew Howard · Lukasz Lew · Sherief Reda · Ville-Mikko Rautio · Daniele Moro, ,https://arxiv.org/abs/2404.00103,,2404.00103.pdf,PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks,"Low-precision quantization is recognized for its efficacy in neural network
+optimization. Our analysis reveals that non-quantized elementwise operations
+which are prevalent in layers such as parameterized activation functions, batch
+normalization, and quantization scaling dominate the inference cost of
+low-precision models. These non-quantized elementwise operations are commonly
+overlooked in SOTA efficiency metrics such as Arithmetic Computation Effort
+(ACE). In this paper, we propose ACEv2 - an extended version of ACE which
+offers a better alignment with the inference cost of quantized models and their
+energy consumption on ML hardware. Moreover, we introduce PikeLPN, a model that
+addresses these efficiency issues by applying quantization to both elementwise
+operations and multiply-accumulate operations. In particular, we present a
+novel quantization technique for batch normalization layers named QuantNorm
+which allows for quantizing the batch normalization parameters without
+compromising the model performance. Additionally, we propose applying Double
+Quantization where the quantization scaling parameters are quantized.
+Furthermore, we recognize and resolve the issue of distribution mismatch in
+Separable Convolution layers by introducing Distribution-Heterogeneous
+Quantization which enables quantizing them to low-precision. PikeLPN achieves
+Pareto-optimality in efficiency-accuracy trade-off with up to 3X efficiency
+improvement compared to SOTA low-precision models.",cs.LG,"['cs.LG', 'cs.CV']"
+Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed,Yifan Wang · Xingyi He · Sida Peng · Dongli Tan · Xiaowei Zhou, ,https://arxiv.org/abs/2403.04765,,2403.04765.pdf,Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed,"We present a novel method for efficiently producing semi-dense matches across
+images. Previous detector-free matcher LoFTR has shown remarkable matching
+capability in handling large-viewpoint change and texture-poor scenarios but
+suffers from low efficiency. We revisit its design choices and derive multiple
+improvements for both efficiency and accuracy. One key observation is that
+performing the transformer over the entire feature map is redundant due to
+shared local information, therefore we propose an aggregated attention
+mechanism with adaptive token selection for efficiency. Furthermore, we find
+spatial variance exists in LoFTR's fine correlation module, which is adverse to
+matching accuracy. A novel two-stage correlation layer is proposed to achieve
+accurate subpixel correspondences for accuracy improvement. Our efficiency
+optimized model is $\sim 2.5\times$ faster than LoFTR which can even surpass
+state-of-the-art efficient sparse matching pipeline SuperPoint + LightGlue.
+Moreover, extensive experiments show that our method can achieve higher
+accuracy compared with competitive semi-dense matchers, with considerable
+efficiency benefits. This opens up exciting prospects for large-scale or
+latency-sensitive applications such as image retrieval and 3D reconstruction.
+Project page: https://zju3dv.github.io/efficientloftr.",cs.CV,['cs.CV']
+"Uncovering What, Why and How:  A Comprehensive Benchmark for Causation Understanding of Video Anomaly",Hang Du · Sicheng Zhang · Binzhu Xie · Guoshun Nan · Jiayang Zhang · Junrui Xu · Hangyu Liu · Sicong Leng · Jiangming Liu · Hehe Fan · Dajiu Huang · Jing Feng · Linli Chen · Can Zhang · Xuhuan Li · Hao Zhang · Jianhang Chen · Qimei Cui · Xiaofeng Tao, ,https://arxiv.org/abs/2405.00181,,2405.00181.pdf,"Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly","Video anomaly understanding (VAU) aims to automatically comprehend unusual
+occurrences in videos, thereby enabling various applications such as traffic
+surveillance and industrial manufacturing. While existing VAU benchmarks
+primarily concentrate on anomaly detection and localization, our focus is on
+more practicality, prompting us to raise the following crucial questions: ""what
+anomaly occurred?"", ""why did it happen?"", and ""how severe is this abnormal
+event?"". In pursuit of these answers, we present a comprehensive benchmark for
+Causation Understanding of Video Anomaly (CUVA). Specifically, each instance of
+the proposed benchmark involves three sets of human annotations to indicate the
+""what"", ""why"" and ""how"" of an anomaly, including 1) anomaly type, start and end
+times, and event descriptions, 2) natural language explanations for the cause
+of an anomaly, and 3) free text reflecting the effect of the abnormality. In
+addition, we also introduce MMEval, a novel evaluation metric designed to
+better align with human preferences for CUVA, facilitating the measurement of
+existing LLMs in comprehending the underlying cause and corresponding effect of
+video anomalies. Finally, we propose a novel prompt-based method that can serve
+as a baseline approach for the challenging CUVA. We conduct extensive
+experiments to show the superiority of our evaluation metric and the
+prompt-based approach. Our code and dataset are available at
+https://github.com/fesvhtr/CUVA.",cs.CV,"['cs.CV', 'cs.AI']"
+GART: Gaussian Articulated Template Models,Jiahui Lei · Yufu Wang · Georgios Pavlakos · Lingjie Liu · Kostas Daniilidis, ,https://arxiv.org/abs/2311.16099,,2311.16099.pdf,GART: Gaussian Articulated Template Models,"We introduce Gaussian Articulated Template Model GART, an explicit,
+efficient, and expressive representation for non-rigid articulated subject
+capturing and rendering from monocular videos. GART utilizes a mixture of
+moving 3D Gaussians to explicitly approximate a deformable subject's geometry
+and appearance. It takes advantage of a categorical template model prior (SMPL,
+SMAL, etc.) with learnable forward skinning while further generalizing to more
+complex non-rigid deformations with novel latent bones. GART can be
+reconstructed via differentiable rendering from monocular videos in seconds or
+minutes and rendered in novel poses faster than 150fps.",cs.CV,"['cs.CV', 'cs.GR']"
+Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering,Tao Lu · Mulin Yu · Linning Xu · Yuanbo Xiangli · Limin Wang · Dahua Lin · Bo Dai, ,https://arxiv.org/abs/2312.00109,,2312.00109.pdf,Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering,"Neural rendering methods have significantly advanced photo-realistic 3D scene
+rendering in various academic and industrial applications. The recent 3D
+Gaussian Splatting method has achieved the state-of-the-art rendering quality
+and speed combining the benefits of both primitive-based representations and
+volumetric representations. However, it often leads to heavily redundant
+Gaussians that try to fit every training view, neglecting the underlying scene
+geometry. Consequently, the resulting model becomes less robust to significant
+view changes, texture-less area and lighting effects. We introduce Scaffold-GS,
+which uses anchor points to distribute local 3D Gaussians, and predicts their
+attributes on-the-fly based on viewing direction and distance within the view
+frustum. Anchor growing and pruning strategies are developed based on the
+importance of neural Gaussians to reliably improve the scene coverage. We show
+that our method effectively reduces redundant Gaussians while delivering
+high-quality rendering. We also demonstrates an enhanced capability to
+accommodate scenes with varying levels-of-detail and view-dependent
+observations, without sacrificing the rendering speed.",cs.CV,['cs.CV']
+DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF,Jie Long Lee · Chen Li · Gim Hee Lee, ,https://arxiv.org/abs/2404.00874,,2404.00874.pdf,DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF,"We present DiSR-NeRF, a diffusion-guided framework for view-consistent
+super-resolution (SR) NeRF. Unlike prior works, we circumvent the requirement
+for high-resolution (HR) reference images by leveraging existing powerful 2D
+super-resolution models. Nonetheless, independent SR 2D images are often
+inconsistent across different views. We thus propose Iterative 3D
+Synchronization (I3DS) to mitigate the inconsistency problem via the inherent
+multi-view consistency property of NeRF. Specifically, our I3DS alternates
+between upscaling low-resolution (LR) rendered images with diffusion models,
+and updating the underlying 3D representation with standard NeRF training. We
+further introduce Renoised Score Distillation (RSD), a novel score-distillation
+objective for 2D image resolution. Our RSD combines features from ancestral
+sampling and Score Distillation Sampling (SDS) to generate sharp images that
+are also LR-consistent. Qualitative and quantitative results on both synthetic
+and real-world datasets demonstrate that our DiSR-NeRF can achieve better
+results on NeRF super-resolution compared with existing works. Code and video
+results available at the project website.",cs.CV,['cs.CV']
+SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction,Yuanhui Huang · Wenzhao Zheng · Borui Zhang · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2311.12754,,2311.12754.pdf,SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction,"3D occupancy prediction is an important task for the robustness of
+vision-centric autonomous driving, which aims to predict whether each point is
+occupied in the surrounding 3D space. Existing methods usually require 3D
+occupancy labels to produce meaningful results. However, it is very laborious
+to annotate the occupancy status of each voxel. In this paper, we propose
+SelfOcc to explore a self-supervised way to learn 3D occupancy using only video
+sequences. We first transform the images into the 3D space (e.g., bird's eye
+view) to obtain 3D representation of the scene. We directly impose constraints
+on the 3D representations by treating them as signed distance fields. We can
+then render 2D images of previous and future frames as self-supervision signals
+to learn the 3D representations. We propose an MVS-embedded strategy to
+directly optimize the SDF-induced weights with multiple depth proposals. Our
+SelfOcc outperforms the previous best method SceneRF by 58.7% using a single
+frame as input on SemanticKITTI and is the first self-supervised work that
+produces reasonable 3D occupancy for surround cameras on nuScenes. SelfOcc
+produces high-quality depth and achieves state-of-the-art results on novel
+depth synthesis, monocular depth estimation, and surround-view depth estimation
+on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code:
+https://github.com/huang-yh/SelfOcc.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Seeing the Unseen: Visual Common Sense for Semantic Placement,Ram Ramrakhya · Aniruddha Kembhavi · Dhruv Batra · Zsolt Kira · Kuo-Hao Zeng · Luca Weihs, ,https://arxiv.org/abs/2401.07770,,2401.07770.pdf,Seeing the Unseen: Visual Common Sense for Semantic Placement,"Computer vision tasks typically involve describing what is present in an
+image (e.g. classification, detection, segmentation, and captioning). We study
+a visual common sense task that requires understanding what is not present.
+Specifically, given an image (e.g. of a living room) and name of an object
+(""cushion""), a vision system is asked to predict semantically-meaningful
+regions (masks or bounding boxes) in the image where that object could be
+placed or is likely be placed by humans (e.g. on the sofa). We call this task:
+Semantic Placement (SP) and believe that such common-sense visual understanding
+is critical for assitive robots (tidying a house), and AR devices
+(automatically rendering an object in the user's space). Studying the invisible
+is hard. Datasets for image description are typically constructed by curating
+relevant images and asking humans to annotate the contents of the image;
+neither of those two steps are straightforward for objects not present in the
+image. We overcome this challenge by operating in the opposite direction: we
+start with an image of an object in context from web, and then remove that
+object from the image via inpainting. This automated pipeline converts
+unstructured web data into a dataset comprising pairs of images with/without
+the object. Using this, we collect a novel dataset, with ${\sim}1.3$M images
+across $9$ object categories, and train a SP prediction model called CLIP-UNet.
+CLIP-UNet outperforms existing VLMs and baselines that combine semantic priors
+with object detectors on real-world and simulated images. In our user studies,
+we find that the SP masks predicted by CLIP-UNet are favored $43.7\%$ and
+$31.3\%$ times when comparing against the $4$ SP baselines on real and
+simulated images. In addition, we demonstrate leveraging SP mask predictions
+from CLIP-UNet enables downstream applications like building tidying robots in
+indoor environments.",cs.CV,['cs.CV']
+Non-autoregressive Sequence-to-Sequence Vision-Language Models,Kunyu Shi · Qi Dong · Luis Goncalves · Zhuowen Tu · Stefano Soatto, ,https://arxiv.org/abs/2403.02249,,2403.02249.pdf,Non-autoregressive Sequence-to-Sequence Vision-Language Models,"Sequence-to-sequence vision-language models are showing promise, but their
+applicability is limited by their inference latency due to their autoregressive
+way of generating predictions. We propose a parallel decoding
+sequence-to-sequence vision-language model, trained with a Query-CTC loss, that
+marginalizes over multiple inference paths in the decoder. This allows us to
+model the joint distribution of tokens, rather than restricting to conditional
+distribution as in an autoregressive model. The resulting model, NARVL,
+achieves performance on-par with its state-of-the-art autoregressive
+counterpart, but is faster at inference time, reducing from the linear
+complexity associated with the sequential generation of tokens to a paradigm of
+constant time joint inference.",cs.CV,"['cs.CV', 'cs.AI']"
+Deep Video Inverse Tone Mapping Based on Temporal Clues,Yuyao Ye · Ning Zhang · Yang Zhao · Hongbin Cao · Ronggang Wang, ,,https://dl.acm.org/doi/10.1145/3648570,,,,,nan
+L2B: Learning to Bootstrap Robust Models for Combating Label Noise,Yuyin Zhou · Xianhang li · Fengze Liu · Qingyue Wei · Xuxi Chen · Lequan Yu · Cihang Xie · Matthew P. Lungren · Lei Xing, ,,https://link.springer.com/chapter/10.1007/978-3-031-43415-0_1,,,,,nan
+Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation,Sangyun Shin · Kaichen Zhou · Madhu Vankadari · Andrew Markham · Niki Trigoni, ,https://arxiv.org/abs/2312.11269,,2312.11269.pdf,Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation,"Coarse-to-fine 3D instance segmentation methods show weak performances
+compared to recent Grouping-based, Kernel-based and Transformer-based methods.
+We argue that this is due to two limitations: 1) Instance size overestimation
+by axis-aligned bounding box(AABB) 2) False negative error accumulation from
+inaccurate box to the refinement phase. In this work, we introduce Spherical
+Mask, a novel coarse-to-fine approach based on spherical representation,
+overcoming those two limitations with several benefits. Specifically, our
+coarse detection estimates each instance with a 3D polygon using a center and
+radial distance predictions, which avoids excessive size estimation of AABB. To
+cut the error propagation in the existing coarse-to-fine approaches, we
+virtually migrate points based on the polygon, allowing all foreground points,
+including false negatives, to be refined. During inference, the proposal and
+point migration modules run in parallel and are assembled to form binary masks
+of instances. We also introduce two margin-based losses for the point migration
+to enforce corrections for the false positives/negatives and cohesion of
+foreground points, significantly improving the performance. Experimental
+results from three datasets, such as ScanNetV2, S3DIS, and STPLS3D, show that
+our proposed method outperforms existing works, demonstrating the effectiveness
+of the new instance representation with spherical coordinates.",cs.CV,"['cs.CV', 'cs.LG']"
+DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model,Zhenghao Pan · Haijin Zeng · Jiezhang Cao · Kai Zhang · Yongyong Chen,https://github.com/PAN083/DiffSCI,https://arxiv.org/abs/2311.11417,,2311.11417.pdf,DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model,"This paper endeavors to advance the precision of snapshot compressive imaging
+(SCI) reconstruction for multispectral image (MSI). To achieve this, we
+integrate the advantageous attributes of established SCI techniques and an
+image generative model, propose a novel structured zero-shot diffusion model,
+dubbed DiffSCI. DiffSCI leverages the structural insights from the deep prior
+and optimization-based methodologies, complemented by the generative
+capabilities offered by the contemporary denoising diffusion model.
+Specifically, firstly, we employ a pre-trained diffusion model, which has been
+trained on a substantial corpus of RGB images, as the generative denoiser
+within the Plug-and-Play framework for the first time. This integration allows
+for the successful completion of SCI reconstruction, especially in the case
+that current methods struggle to address effectively. Secondly, we
+systematically account for spectral band correlations and introduce a robust
+methodology to mitigate wavelength mismatch, thus enabling seamless adaptation
+of the RGB diffusion model to MSIs. Thirdly, an accelerated algorithm is
+implemented to expedite the resolution of the data subproblem. This
+augmentation not only accelerates the convergence rate but also elevates the
+quality of the reconstruction process. We present extensive testing to show
+that DiffSCI exhibits discernible performance enhancements over prevailing
+self-supervised and zero-shot approaches, surpassing even supervised
+transformer counterparts across both simulated and real datasets. Our code will
+be available.",cs.CV,['cs.CV']
+$\mathsf{LQMFormer}$:~Language-aware Query Mask Transformer for Referring Image Segmentation,Nisarg Shah · Vibashan VS · Vishal M. Patel, ,https://arxiv.org/abs/2312.12198,,,Mask Grounding for Referring Image Segmentation,"Referring Image Segmentation (RIS) is a challenging task that requires an
+algorithm to segment objects referred by free-form language expressions.
+Despite significant progress in recent years, most state-of-the-art (SOTA)
+methods still suffer from considerable language-image modality gap at the pixel
+and word level. These methods generally 1) rely on sentence-level language
+features for language-image alignment and 2) lack explicit training supervision
+for fine-grained visual grounding. Consequently, they exhibit weak object-level
+correspondence between visual and language features. Without well-grounded
+features, prior methods struggle to understand complex expressions that require
+strong reasoning over relationships among multiple objects, especially when
+dealing with rarely used or ambiguous clauses. To tackle this challenge, we
+introduce a novel Mask Grounding auxiliary task that significantly improves
+visual grounding within language features, by explicitly teaching the model to
+learn fine-grained correspondence between masked textual tokens and their
+matching visual objects. Mask Grounding can be directly used on prior RIS
+methods and consistently bring improvements. Furthermore, to holistically
+address the modality gap, we also design a cross-modal alignment loss and an
+accompanying alignment module. These additions work synergistically with Mask
+Grounding. With all these techniques, our comprehensive approach culminates in
+MagNet (Mask-grounded Network), an architecture that significantly outperforms
+prior arts on three key benchmarks (RefCOCO, RefCOCO+ and G-Ref), demonstrating
+our method's effectiveness in addressing current limitations of RIS algorithms.
+Our code and pre-trained weights will be released.",cs.CV,['cs.CV']
+CLIP as RNN:  Segment Countless Visual Concepts without Training Endeavor,Shuyang Sun · Runjia Li · Philip H.S. Torr · Xiuye Gu · Siyang Li, ,https://arxiv.org/abs/2312.07661,,2312.07661.pdf,CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor,"Existing open-vocabulary image segmentation methods require a fine-tuning
+step on mask labels and/or image-text datasets. Mask labels are
+labor-intensive, which limits the number of categories in segmentation
+datasets. Consequently, the vocabulary capacity of pre-trained VLMs is severely
+reduced after fine-tuning. However, without fine-tuning, VLMs trained under
+weak image-text supervision tend to make suboptimal mask predictions. To
+alleviate these issues, we introduce a novel recurrent framework that
+progressively filters out irrelevant texts and enhances mask quality without
+training efforts. The recurrent unit is a two-stage segmenter built upon a
+frozen VLM. Thus, our model retains the VLM's broad vocabulary space and equips
+it with segmentation ability. Experiments show that our method outperforms not
+only the training-free counterparts, but also those fine-tuned with millions of
+data samples, and sets the new state-of-the-art records for both zero-shot
+semantic and referring segmentation. Concretely, we improve the current record
+by 28.8, 16.0, and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG', 'cs.MM']"
+Improving Generalization via Meta-Learning on Hard Samples,Nishant Jain · Arun Suggala · Pradeep Shenoy, ,https://arxiv.org/abs/2403.12236,,2403.12236.pdf,Improving Generalization via Meta-Learning on Hard Samples,"Learned reweighting (LRW) approaches to supervised learning use an
+optimization criterion to assign weights for training instances, in order to
+maximize performance on a representative validation dataset. We pose and
+formalize the problem of optimized selection of the validation set used in LRW
+training, to improve classifier generalization. In particular, we show that
+using hard-to-classify instances in the validation set has both a theoretical
+connection to, and strong empirical evidence of generalization. We provide an
+efficient algorithm for training this meta-optimized model, as well as a simple
+train-twice heuristic for careful comparative study. We demonstrate that LRW
+with easy validation data performs consistently worse than LRW with hard
+validation data, establishing the validity of our meta-optimization problem.
+Our proposed algorithm outperforms a wide range of baselines on a range of
+datasets and domain shift challenges (Imagenet-1K, CIFAR-100, Clothing-1M,
+CAMELYON, WILDS, etc.), with ~1% gains using VIT-B on Imagenet. We also show
+that using naturally hard examples for validation (Imagenet-R / Imagenet-A) in
+LRW training for Imagenet improves performance on both clean and naturally hard
+test instances by 1-2%. Secondary analyses show that using hard validation data
+in an LRW framework improves margins on test data, hinting at the mechanism
+underlying our empirical gains. We believe this work opens up new research
+directions for the meta-optimization of meta-learning in a supervised learning
+context.",cs.LG,"['cs.LG', 'cs.CV']"
+PICTURE: PhotorealistIC virtual Try-on from UnconstRained dEsigns,Shuliang Ning · Duomin Wang · Yipeng Qin · Zirong Jin · Baoyuan Wang · Xiaoguang Han, ,https://arxiv.org/abs/2312.04534,,2312.04534.pdf,PICTURE: PhotorealistIC virtual Try-on from UnconstRained dEsigns,"In this paper, we propose a novel virtual try-on from unconstrained designs
+(ucVTON) task to enable photorealistic synthesis of personalized composite
+clothing on input human images. Unlike prior arts constrained by specific input
+types, our method allows flexible specification of style (text or image) and
+texture (full garment, cropped sections, or texture patches) conditions. To
+address the entanglement challenge when using full garment images as
+conditions, we develop a two-stage pipeline with explicit disentanglement of
+style and texture. In the first stage, we generate a human parsing map
+reflecting the desired style conditioned on the input. In the second stage, we
+composite textures onto the parsing map areas based on the texture input. To
+represent complex and non-stationary textures that have never been achieved in
+previous fashion editing works, we first propose extracting hierarchical and
+balanced CLIP features and applying position encoding in VTON. Experiments
+demonstrate superior synthesis quality and personalization enabled by our
+method. The flexible control over style and texture mixing brings virtual
+try-on to a new level of user experience for online shopping and fashion
+design.",cs.CV,['cs.CV']
+KPConvX: Modernizing Kernel Point Convolution with Kernel Attention,Hugues Thomas · Yao-Hung Hubert Tsai · Timothy Barfoot · Jian Zhang, ,https://arxiv.org/abs/2405.13194,,2405.13194.pdf,KPConvX: Modernizing Kernel Point Convolution with Kernel Attention,"In the field of deep point cloud understanding, KPConv is a unique
+architecture that uses kernel points to locate convolutional weights in space,
+instead of relying on Multi-Layer Perceptron (MLP) encodings. While it
+initially achieved success, it has since been surpassed by recent MLP networks
+that employ updated designs and training strategies. Building upon the kernel
+point principle, we present two novel designs: KPConvD (depthwise KPConv), a
+lighter design that enables the use of deeper architectures, and KPConvX, an
+innovative design that scales the depthwise convolutional weights of KPConvD
+with kernel attention values. Using KPConvX with a modern architecture and
+training strategy, we are able to outperform current state-of-the-art
+approaches on the ScanObjectNN, Scannetv2, and S3DIS datasets. We validate our
+design choices through ablation studies and release our code and models.",cs.CV,['cs.CV']
+FedAS: Bridging Inconsistency in Personalized Federated Learning,Xiyuan Yang · Wenke Huang · Mang Ye,https://github.com/xiyuanyang45/FedAS,,https://dl.acm.org/doi/10.5555/3666122.3669282,,,,,nan
+DeIl: Direct and Inverse CLIP for Open-World Few-Shot Learning,Shuai Shao · Yu Bai · Yan WANG · Bao-di Liu · Yicong Zhou, ,,https://www.semanticscholar.org/paper/Collaborative-Consortium-of-Foundation-Models-for-Shao-Bai/90668de8b1c5dcb0471444e3177dc28e20fce5d4,,,,,nan
+"Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding",Wujian Peng · Sicheng Xie · Zuyao You · Shiyi Lan · Zuxuan Wu,https://github.com/wjpoom/SPEC,https://arxiv.org/abs/2312.00081,,2312.00081.pdf,"Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding","Vision language models (VLM) have demonstrated remarkable performance across
+various downstream tasks. However, understanding fine-grained visual-linguistic
+concepts, such as attributes and inter-object relationships, remains a
+significant challenge. While several benchmarks aim to evaluate VLMs in finer
+granularity, their primary focus remains on the linguistic aspect, neglecting
+the visual dimension. Here, we highlight the importance of evaluating VLMs from
+both a textual and visual perspective. We introduce a progressive pipeline to
+synthesize images that vary in a specific attribute while ensuring consistency
+in all other aspects. Utilizing this data engine, we carefully design a
+benchmark, SPEC, to diagnose the comprehension of object size, position,
+existence, and count. Subsequently, we conduct a thorough evaluation of four
+leading VLMs on SPEC. Surprisingly, their performance is close to random guess,
+revealing significant limitations. With this in mind, we propose a simple yet
+effective approach to optimize VLMs in fine-grained understanding, achieving
+significant improvements on SPEC without compromising the zero-shot
+performance. Results on two additional fine-grained benchmarks also show
+consistent improvements, further validating the transferability of our
+approach. Code and data are available at https://github.com/wjpoom/SPEC.",cs.CV,['cs.CV']
+CPR: Retrieval Augmented Generation for Copyright Protection,Aditya Golatkar · Alessandro Achille · Luca Zancato · Yu-Xiang Wang · Ashwin Swaminathan · Stefano Soatto · Stefano Soatto, ,https://arxiv.org/abs/2403.18920,,2403.18920.pdf,CPR: Retrieval Augmented Generation for Copyright Protection,"Retrieval Augmented Generation (RAG) is emerging as a flexible and robust
+technique to adapt models to private users data without training, to handle
+credit attribution, and to allow efficient machine unlearning at scale.
+However, RAG techniques for image generation may lead to parts of the retrieved
+samples being copied in the model's output. To reduce risks of leaking private
+information contained in the retrieved set, we introduce Copy-Protected
+generation with Retrieval (CPR), a new method for RAG with strong copyright
+protection guarantees in a mixed-private setting for diffusion models.CPR
+allows to condition the output of diffusion models on a set of retrieved
+images, while also guaranteeing that unique identifiable information about
+those example is not exposed in the generated outputs. In particular, it does
+so by sampling from a mixture of public (safe) distribution and private (user)
+distribution by merging their diffusion scores at inference. We prove that CPR
+satisfies Near Access Freeness (NAF) which bounds the amount of information an
+attacker may be able to extract from the generated images. We provide two
+algorithms for copyright protection, CPR-KL and CPR-Choose. Unlike previously
+proposed rejection-sampling-based NAF methods, our methods enable efficient
+copyright-protected sampling with a single run of backward diffusion. We show
+that our method can be applied to any pre-trained conditional diffusion model,
+such as Stable Diffusion or unCLIP. In particular, we empirically show that
+applying CPR on top of unCLIP improves quality and text-to-image alignment of
+the generated results (81.4 to 83.17 on TIFA benchmark), while enabling credit
+attribution, copy-right protection, and deterministic, constant time,
+unlearning.",cs.CR,"['cs.CR', 'cs.AI', 'cs.CV']"
+FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models,Shivangi Aneja · Justus Thies · Angela Dai · Matthias Nießner, ,https://arxiv.org/abs/2312.08459,,2312.08459.pdf,FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models,"We introduce FaceTalk, a novel generative approach designed for synthesizing
+high-fidelity 3D motion sequences of talking human heads from input audio
+signal. To capture the expressive, detailed nature of human heads, including
+hair, ears, and finer-scale eye movements, we propose to couple speech signal
+with the latent space of neural parametric head models to create high-fidelity,
+temporally coherent motion sequences. We propose a new latent diffusion model
+for this task, operating in the expression space of neural parametric head
+models, to synthesize audio-driven realistic head sequences. In the absence of
+a dataset with corresponding NPHM expressions to audio, we optimize for these
+correspondences to produce a dataset of temporally-optimized NPHM expressions
+fit to audio-video recordings of people talking. To the best of our knowledge,
+this is the first work to propose a generative approach for realistic and
+high-quality motion synthesis of volumetric human heads, representing a
+significant advancement in the field of audio-driven 3D animation. Notably, our
+approach stands out in its ability to generate plausible motion sequences that
+can produce high-fidelity head animation coupled with the NPHM shape space. Our
+experimental results substantiate the effectiveness of FaceTalk, consistently
+achieving superior and visually natural motion, encompassing diverse facial
+expressions and styles, outperforming existing methods by 75% in perceptual
+user study evaluation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.SD', 'eess.AS']"
+Binding Touch to Everything: Learning Unified Multimodal Tactile Representations,Fengyu Yang · Chao Feng · Ziyang Chen · Hyoungseob Park · Daniel Wang · Yiming Dou · Ziyao Zeng · xien chen · Suchisrit Gangopadhyay · Andrew Owens · Alex Wong, ,https://arxiv.org/abs/2401.18084,,2401.18084.pdf,Binding Touch to Everything: Learning Unified Multimodal Tactile Representations,"The ability to associate touch with other modalities has huge implications
+for humans and computational systems. However, multimodal learning with touch
+remains challenging due to the expensive data collection process and
+non-standardized sensor outputs. We introduce UniTouch, a unified tactile model
+for vision-based touch sensors connected to multiple modalities, including
+vision, language, and sound. We achieve this by aligning our UniTouch
+embeddings to pretrained image embeddings already associated with a variety of
+other modalities. We further propose learnable sensor-specific tokens, allowing
+the model to learn from a set of heterogeneous tactile sensors, all at the same
+time. UniTouch is capable of conducting various touch sensing tasks in the
+zero-shot setting, from robot grasping prediction to touch image question
+answering. To the best of our knowledge, UniTouch is the first to demonstrate
+such capabilities. Project page: https://cfeng16.github.io/UniTouch/",cs.CV,"['cs.CV', 'cs.RO']"
+Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles,Vanessa Sklyarova · Egor Zakharov · Otmar Hilliges · Michael J. Black · Justus Thies,https://haar.is.tue.mpg.de/,https://arxiv.org/abs/2312.11666,,2312.11666.pdf,HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles,"We present HAAR, a new strand-based generative model for 3D human hairstyles.
+Specifically, based on textual inputs, HAAR produces 3D hairstyles that could
+be used as production-level assets in modern computer graphics engines. Current
+AI-based generative models take advantage of powerful 2D priors to reconstruct
+3D content in the form of point clouds, meshes, or volumetric functions.
+However, by using the 2D priors, they are intrinsically limited to only
+recovering the visual parts. Highly occluded hair structures can not be
+reconstructed with those methods, and they only model the ''outer shell'',
+which is not ready to be used in physics-based rendering or simulation
+pipelines. In contrast, we propose a first text-guided generative method that
+uses 3D hair strands as an underlying representation. Leveraging 2D visual
+question-answering (VQA) systems, we automatically annotate synthetic hair
+models that are generated from a small set of artist-created hairstyles. This
+allows us to train a latent diffusion model that operates in a common hairstyle
+UV space. In qualitative and quantitative studies, we demonstrate the
+capabilities of the proposed model and compare it to existing hairstyle
+generation approaches.",cs.CV,"['cs.CV', 'cs.GR']"
+Sieve: Multimodal Dataset Pruning using Image-Captioning Models,Anas Mahmoud · Mostafa Elhoushi · Amro Abbas · Yu Yang · Newsha Ardalani · Hugh Leather · Ari Morcos, ,https://arxiv.org/abs/2310.02110,,2310.02110.pdf,Sieve: Multimodal Dataset Pruning Using Image Captioning Models,"Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy
+web-crawled datasets. This underscores the critical need for dataset pruning,
+as the quality of these datasets is strongly correlated with the performance of
+VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train
+models using highly-aligned samples is one of the most successful methods for
+pruning. We argue that this approach suffers from multiple limitations
+including: false positives and negatives due to CLIP's pretraining on noisy
+labels. We propose a pruning signal, Sieve, that employs synthetic captions
+generated by image-captioning models pretrained on small, diverse, and
+well-aligned image-text pairs to evaluate the alignment of noisy image-text
+pairs. To bridge the gap between the limited diversity of generated captions
+and the high diversity of alternative text (alt-text), we estimate the semantic
+textual similarity in the embedding space of a language model pretrained on
+unlabeled text corpus. Using DataComp, a multimodal dataset filtering
+benchmark, when evaluating on 38 downstream tasks, our pruning approach,
+surpasses CLIPScore by 2.6\% and 1.7\% on medium and large scale respectively.
+In addition, on retrieval tasks, Sieve leads to a significant improvement of
+2.7% and 4.5% on medium and large scale respectively.",cs.CV,['cs.CV']
+Streaming Dense Video Captioning,Xingyi Zhou · Anurag Arnab · Shyamal Buch · Shen Yan · Austin Myers · Xuehan Xiong · Arsha Nagrani · Cordelia Schmid, ,https://arxiv.org/abs/2404.01297,,2404.01297.pdf,Streaming Dense Video Captioning,"An ideal model for dense video captioning -- predicting captions localized
+temporally in a video -- should be able to handle long input videos, predict
+rich, detailed textual descriptions, and be able to produce outputs before
+processing the entire video. Current state-of-the-art models, however, process
+a fixed number of downsampled frames, and make a single full prediction after
+seeing the whole video. We propose a streaming dense video captioning model
+that consists of two novel components: First, we propose a new memory module,
+based on clustering incoming tokens, which can handle arbitrarily long videos
+as the memory is of a fixed size. Second, we develop a streaming decoding
+algorithm that enables our model to make predictions before the entire video
+has been processed. Our model achieves this streaming ability, and
+significantly improves the state-of-the-art on three dense video captioning
+benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
+https://github.com/google-research/scenic.",cs.CV,['cs.CV']
+DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes,Hao Yan · Zhihui Ke · Xiaobo Zhou · Tie Qiu · Xidong Shi · DaDong Jiang,https://haoyan14.github.io/DS-NeRV/,https://arxiv.org/abs/2403.15679,,,DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes,"Implicit neural representations for video (NeRV) have recently become a novel
+way for high-quality video representation. However, existing works employ a
+single network to represent the entire video, which implicitly confuse static
+and dynamic information. This leads to an inability to effectively compress the
+redundant static information and lack the explicitly modeling of global
+temporal-coherent dynamic details. To solve above problems, we propose DS-NeRV,
+which decomposes videos into sparse learnable static codes and dynamic codes
+without the need for explicit optical flow or residual supervision. By setting
+different sampling rates for two codes and applying weighted sum and
+interpolation sampling methods, DS-NeRV efficiently utilizes redundant static
+information while maintaining high-frequency details. Additionally, we design a
+cross-channel attention-based (CCA) fusion module to efficiently fuse these two
+codes for frame decoding. Our approach achieves a high quality reconstruction
+of 31.2 PSNR with only 0.35M parameters thanks to separate static and dynamic
+codes representation and outperforms existing NeRV methods in many downstream
+tasks. Our project website is at https://haoyan14.github.io/DS-NeRV.",cs.CV,"['cs.CV', 'cs.MM']"
+SANeRF-HQ: Segment Anything for NeRF in High Quality,Yichen Liu · Benran Hu · Chi-Keung Tang · Yu-Wing Tai, ,https://arxiv.org/abs/2312.01531,,2312.01531.pdf,SANeRF-HQ: Segment Anything for NeRF in High Quality,"Recently, the Segment Anything Model (SAM) has showcased remarkable
+capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields) has
+gained popularity as a method for various 3D problems beyond novel view
+synthesis. Though there exist initial attempts to incorporate these two methods
+into 3D segmentation, they face the challenge of accurately and consistently
+segmenting objects in complex scenarios. In this paper, we introduce the
+Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve high-quality
+3D segmentation of any target object in a given scene. SANeRF-HQ utilizes SAM
+for open-world object segmentation guided by user-supplied prompts, while
+leveraging NeRF to aggregate information from different viewpoints. To overcome
+the aforementioned challenges, we employ density field and RGB similarity to
+enhance the accuracy of segmentation boundary during the aggregation.
+Emphasizing on segmentation accuracy, we evaluate our method on multiple NeRF
+datasets where high-quality ground-truths are available or manually annotated.
+SANeRF-HQ shows a significant quality improvement over state-of-the-art methods
+in NeRF object segmentation, provides higher flexibility for object
+localization, and enables more consistent object segmentation across multiple
+views. Results and code are available at the project site:
+https://lyclyc52.github.io/SANeRF-HQ/.",cs.CV,['cs.CV']
+\emph{RealCustom}: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization,Mengqi Huang · Zhendong Mao · Mingcong Liu · Qian HE · Yongdong Zhang,https://corleone-huang.github.io/realcustom/,https://arxiv.org/abs/2403.00483,,2403.00483.pdf,RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization,"Text-to-image customization, which aims to synthesize text-driven images for
+the given subjects, has recently revolutionized content creation. Existing
+works follow the pseudo-word paradigm, i.e., represent the given subjects as
+pseudo-words and then compose them with the given text. However, the inherent
+entangled influence scope of pseudo-words with the given text results in a
+dual-optimum paradox, i.e., the similarity of the given subjects and the
+controllability of the given text could not be optimal simultaneously. We
+present RealCustom that, for the first time, disentangles similarity from
+controllability by precisely limiting subject influence to relevant parts only,
+achieved by gradually narrowing real text word from its general connotation to
+the specific subject and using its cross-attention to distinguish relevance.
+Specifically, RealCustom introduces a novel ""train-inference"" decoupled
+framework: (1) during training, RealCustom learns general alignment between
+visual conditions to original textual conditions by a novel adaptive scoring
+module to adaptively modulate influence quantity; (2) during inference, a novel
+adaptive mask guidance strategy is proposed to iteratively update the influence
+scope and influence quantity of the given subjects to gradually narrow the
+generation of the real text word. Comprehensive experiments demonstrate the
+superior real-time customization ability of RealCustom in the open domain,
+achieving both unprecedented similarity of the given subjects and
+controllability of the given text for the first time. The project page is
+https://corleone-huang.github.io/realcustom/.",cs.CV,['cs.CV']
+Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification,Sravanti Addepalli · Ashish Asokan · Lakshay Sharma · R. Venkatesh Babu, ,https://arxiv.org/abs/2310.08255,,2310.08255.pdf,Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification,"Vision-Language Models (VLMs) such as CLIP are trained on large amounts of
+image-text pairs, resulting in remarkable generalization across several data
+distributions. However, in several cases, their expensive training and data
+collection/curation costs do not justify the end application. This motivates a
+vendor-client paradigm, where a vendor trains a large-scale VLM and grants only
+input-output access to clients on a pay-per-query basis in a black-box setting.
+The client aims to minimize inference cost by distilling the VLM to a student
+model using the limited available task-specific data, and further deploying
+this student model in the downstream application. While naive distillation
+largely improves the In-Domain (ID) accuracy of the student, it fails to
+transfer the superior out-of-distribution (OOD) generalization of the VLM
+teacher using the limited available labeled images. To mitigate this, we
+propose Vision-Language to Vision - Align, Distill, Predict (VL2V-ADiP), which
+first aligns the vision and language modalities of the teacher model with the
+vision modality of a pre-trained student model, and further distills the
+aligned VLM representations to the student. This maximally retains the
+pre-trained features of the student, while also incorporating the rich
+representations of the VLM image encoder and the superior generalization of the
+text embeddings. The proposed approach achieves state-of-the-art results on the
+standard Domain Generalization benchmarks in a black-box teacher setting as
+well as a white-box setting where the weights of the VLM are accessible.",cs.CV,['cs.CV']
+TransLoc4D: Transformer-based 4D Radar Place Recognition,Guohao Peng · Heshan Li · Yangyang Zhao · Jun Zhang · Zhenyu Wu · Pengyu Zheng · Danwei Wang, ,https://arxiv.org/abs/2401.13082,,2401.13082.pdf,PlaceFormer: Transformer-based Visual Place Recognition using Multi-Scale Patch Selection and Fusion,"Visual place recognition is a challenging task in the field of computer
+vision, and autonomous robotics and vehicles, which aims to identify a location
+or a place from visual inputs. Contemporary methods in visual place recognition
+employ convolutional neural networks and utilize every region within the image
+for the place recognition task. However, the presence of dynamic and
+distracting elements in the image may impact the effectiveness of the place
+recognition process. Therefore, it is meaningful to focus on task-relevant
+regions of the image for improved recognition. In this paper, we present
+PlaceFormer, a novel transformer-based approach for visual place recognition.
+PlaceFormer employs patch tokens from the transformer to create global image
+descriptors, which are then used for image retrieval. To re-rank the retrieved
+images, PlaceFormer merges the patch tokens from the transformer to form
+multi-scale patches. Utilizing the transformer's self-attention mechanism, it
+selects patches that correspond to task-relevant areas in an image. These
+selected patches undergo geometric verification, generating similarity scores
+across different patch sizes. Subsequently, spatial scores from each patch size
+are fused to produce a final similarity score. This score is then used to
+re-rank the images initially retrieved using global image descriptors.
+Extensive experiments on benchmark datasets demonstrate that PlaceFormer
+outperforms several state-of-the-art methods in terms of accuracy and
+computational efficiency, requiring less time and memory.",cs.CV,"['cs.CV', 'cs.RO']"
+Domain Gap Embeddings for Generative Dataset Augmentation,Yinong Wang · Younjoon Chung · Chen Henry Wu · Fernando De la Torre, ,https://arxiv.org/abs/2312.05387,,2312.05387.pdf,Cross Domain Generative Augmentation: Domain Generalization with Latent Diffusion Models,"Despite the huge effort in developing novel regularizers for Domain
+Generalization (DG), adding simple data augmentation to the vanilla ERM which
+is a practical implementation of the Vicinal Risk Minimization principle (VRM)
+\citep{chapelle2000vicinal} outperforms or stays competitive with many of the
+proposed regularizers. The VRM reduces the estimation error in ERM by replacing
+the point-wise kernel estimates with a more precise estimation of true data
+distribution that reduces the gap between data points \textbf{within each
+domain}. However, in the DG setting, the estimation error of true data
+distribution by ERM is mainly caused by the distribution shift \textbf{between
+domains} which cannot be fully addressed by simple data augmentation techniques
+within each domain. Inspired by this limitation of VRM, we propose a novel data
+augmentation named Cross Domain Generative Augmentation (CDGA) that replaces
+the pointwise kernel estimates in ERM with new density estimates in the
+\textbf{vicinity of domain pairs} so that the gap between domains is further
+reduced. To this end, CDGA, which is built upon latent diffusion models (LDM),
+generates synthetic images to fill the gap between all domains and as a result,
+reduces the non-iidness. We show that CDGA outperforms SOTA DG methods under
+the Domainbed benchmark. To explain the effectiveness of CDGA, we generate more
+than 5 Million synthetic images and perform extensive ablation studies
+including data scaling laws, distribution visualization, domain shift
+quantification, adversarial robustness, and loss landscape analysis.",cs.LG,['cs.LG']
+Detours for Navigating Instructional Videos,Kumar Ashutosh · Zihui Xue · Tushar Nagarajan · Kristen Grauman, ,https://arxiv.org/abs/2401.01823,,2401.01823.pdf,Detours for Navigating Instructional Videos,"We introduce the video detours problem for navigating instructional videos.
+Given a source video and a natural language query asking to alter the how-to
+video's current path of execution in a certain way, the goal is to find a
+related ''detour video'' that satisfies the requested alteration. To address
+this challenge, we propose VidDetours, a novel video-language approach that
+learns to retrieve the targeted temporal segments from a large repository of
+how-to's using video-and-text conditioned queries. Furthermore, we devise a
+language-based pipeline that exploits how-to video narration text to create
+weakly supervised training data. We demonstrate our idea applied to the domain
+of how-to cooking videos, where a user can detour from their current recipe to
+find steps with alternate ingredients, tools, and techniques. Validating on a
+ground truth annotated dataset of 16K samples, we show our model's significant
+improvements over best available methods for video retrieval and question
+answering, with recall rates exceeding the state of the art by 35%.",cs.CV,['cs.CV']
+Iterated Learning Improves Compositionality in Large Vision-Language Models,Chenhao Zheng · Jieyu Zhang · Aniruddha Kembhavi · Ranjay Krishna, ,https://arxiv.org/abs/2404.02145,,2404.02145.pdf,Iterated Learning Improves Compositionality in Large Vision-Language Models,"A fundamental characteristic common to both human vision and natural language
+is their compositional nature. Yet, despite the performance gains contributed
+by large vision and language pretraining, recent investigations find that
+most-if not all-our state-of-the-art vision-language models struggle at
+compositionality. They are unable to distinguish between images of "" a girl in
+white facing a man in black"" and ""a girl in black facing a man in white"".
+Moreover, prior work suggests that compositionality doesn't arise with scale:
+larger model sizes or training data don't help. This paper develops a new
+iterated training algorithm that incentivizes compositionality. We draw on
+decades of cognitive science research that identifies cultural transmission-the
+need to teach a new generation-as a necessary inductive prior that incentivizes
+humans to develop compositional languages. Specifically, we reframe
+vision-language contrastive learning as the Lewis Signaling Game between a
+vision agent and a language agent, and operationalize cultural transmission by
+iteratively resetting one of the agent's weights during training. After every
+iteration, this training paradigm induces representations that become ""easier
+to learn"", a property of compositional languages: e.g. our model trained on
+CC3M and CC12M improves standard CLIP by 4.7%, 4.0% respectfully in the
+SugarCrepe benchmark.",cs.CV,['cs.CV']
+Contrastive Mean-Shift Learning for Generalized Category Discovery,Sua Choi · Dahyun Kang · Minsu Cho, ,https://arxiv.org/abs/2404.09451,,2404.09451.pdf,Contrastive Mean-Shift Learning for Generalized Category Discovery,"We address the problem of generalized category discovery (GCD) that aims to
+partition a partially labeled collection of images; only a small part of the
+collection is labeled and the total number of target classes is unknown. To
+address this generalized image clustering problem, we revisit the mean-shift
+algorithm, i.e., a classic, powerful technique for mode seeking, and
+incorporate it into a contrastive learning framework. The proposed method,
+dubbed Contrastive Mean-Shift (CMS) learning, trains an image encoder to
+produce representations with better clustering properties by an iterative
+process of mean shift and contrastive update. Experiments demonstrate that our
+method, both in settings with and without the total number of clusters being
+known, achieves state-of-the-art performance on six public GCD benchmarks
+without bells and whistles.",cs.CV,['cs.CV']
+Volumetric Environment Representation for Vision-Language Navigation,Liu · Wenguan Wang · Yi Yang, ,https://arxiv.org/abs/2403.14158v1,,2403.14158v1.pdf,Volumetric Environment Representation for Vision-Language Navigation,"Vision-language navigation (VLN) requires an agent to navigate through an 3D
+environment based on visual observations and natural language instructions. It
+is clear that the pivotal factor for successful navigation lies in the
+comprehensive scene understanding. Previous VLN agents employ monocular
+frameworks to extract 2D features of perspective views directly. Though
+straightforward, they struggle for capturing 3D geometry and semantics, leading
+to a partial and incomplete environment representation. To achieve a
+comprehensive 3D representation with fine-grained details, we introduce a
+Volumetric Environment Representation (VER), which voxelizes the physical world
+into structured 3D cells. For each cell, VER aggregates multi-view 2D features
+into such a unified 3D space via 2D-3D sampling. Through coarse-to-fine feature
+extraction and multi-task learning for VER, our agent predicts 3D occupancy, 3D
+room layout, and 3D bounding boxes jointly. Based on online collected VERs, our
+agent performs volume state estimation and builds episodic memory for
+predicting the next step. Experimental results show our environment
+representations from multi-task learning lead to evident performance gains on
+VLN. Our model achieves state-of-the-art performance across VLN benchmarks
+(R2R, REVERIE, and R4R).",cs.CV,['cs.CV']
+DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data,Qihao Liu · Yi Zhang · Song Bai · Adam Kortylewski · Alan L. Yuille, ,https://arxiv.org/abs/2405.14832,,2405.14832.pdf,Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer,"Generating high-quality 3D assets from text and images has long been
+challenging, primarily due to the absence of scalable 3D representations
+capable of capturing intricate geometry distributions. In this work, we
+introduce Direct3D, a native 3D generative model scalable to in-the-wild input
+images, without requiring a multiview diffusion model or SDS optimization. Our
+approach comprises two primary components: a Direct 3D Variational Auto-Encoder
+(D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently
+encodes high-resolution 3D shapes into a compact and continuous latent triplane
+space. Notably, our method directly supervises the decoded geometry using a
+semi-continuous surface sampling strategy, diverging from previous methods
+relying on rendered images as supervision signals. D3D-DiT models the
+distribution of encoded 3D latents and is specifically designed to fuse
+positional information from the three feature maps of the triplane latent,
+enabling a native 3D generative model scalable to large-scale 3D datasets.
+Additionally, we introduce an innovative image-to-3D generation pipeline
+incorporating semantic and pixel-level image conditions, allowing the model to
+produce 3D shapes consistent with the provided conditional image input.
+Extensive experiments demonstrate the superiority of our large-scale
+pre-trained Direct3D over previous image-to-3D approaches, achieving
+significantly better generation quality and generalization ability, thus
+establishing a new state-of-the-art for 3D content creation. Project page:
+https://nju-3dv.github.io/projects/Direct3D/.",cs.CV,['cs.CV']
+One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models,Lin Li · Haoyan Guan · Jianing Qiu · Michael Spratling,https://github.com/TreeLLi/APT,https://arxiv.org/abs/2403.01849,,2403.01849.pdf,One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models,"Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having
+remarkable generalization ability, are highly vulnerable to adversarial
+examples. This work studies the adversarial robustness of VLMs from the novel
+perspective of the text prompt instead of the extensively studied model weights
+(frozen in this work). We first show that the effectiveness of both adversarial
+attack and defense are sensitive to the used text prompt. Inspired by this, we
+propose a method to improve resilience to adversarial attacks by learning a
+robust text prompt for VLMs. The proposed method, named Adversarial Prompt
+Tuning (APT), is effective while being both computationally and data efficient.
+Extensive experiments are conducted across 15 datasets and 4 data sparsity
+schemes (from 1-shot to full training data settings) to show APT's superiority
+over hand-engineered prompts and other state-of-the-art adaption methods. APT
+demonstrated excellent abilities in terms of the in-distribution performance
+and the generalization under input distribution shift and across datasets.
+Surprisingly, by simply adding one learned word to the prompts, APT can
+significantly boost the accuracy and robustness (epsilon=4/255) over the
+hand-engineered prompts by +13% and +8.5% on average respectively. The
+improvement further increases, in our most effective setting, to +26.4% for
+accuracy and +16.7% for robustness. Code is available at
+https://github.com/TreeLLi/APT.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Progressive Divide-and-Conquer via Subsampling Decomposition for Accelerated MRI,Chong Wang · Lanqing Guo · Yufei Wang · Hao Cheng · Yi Yu · Bihan Wen,https://github.com/ChongWang1024/PDAC,https://arxiv.org/abs/2403.10064,,2403.10064.pdf,Progressive Divide-and-Conquer via Subsampling Decomposition for Accelerated MRI,"Deep unfolding networks (DUN) have emerged as a popular iterative framework
+for accelerated magnetic resonance imaging (MRI) reconstruction. However,
+conventional DUN aims to reconstruct all the missing information within the
+entire null space in each iteration. Thus it could be challenging when dealing
+with highly ill-posed degradation, usually leading to unsatisfactory
+reconstruction. In this work, we propose a Progressive Divide-And-Conquer
+(PDAC) strategy, aiming to break down the subsampling process in the actual
+severe degradation and thus perform reconstruction sequentially. Starting from
+decomposing the original maximum-a-posteriori problem of accelerated MRI, we
+present a rigorous derivation of the proposed PDAC framework, which could be
+further unfolded into an end-to-end trainable network. Specifically, each
+iterative stage in PDAC focuses on recovering a distinct moderate degradation
+according to the decomposition. Furthermore, as part of the PDAC iteration,
+such decomposition is adaptively learned as an auxiliary task through a
+degradation predictor which provides an estimation of the decomposed sampling
+mask. Following this prediction, the sampling mask is further integrated via a
+severity conditioning module to ensure awareness of the degradation severity at
+each stage. Extensive experiments demonstrate that our proposed method achieves
+superior performance on the publicly available fastMRI and Stanford2D FSE
+datasets in both multi-coil and single-coil settings.",eess.IV,"['eess.IV', 'cs.CV']"
+Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World,Wen Yin · Jian Lou · Pan Zhou · Yulai Xie · Dan Feng · Yuhua Sun · Tailai Zhang · Lichao Sun, ,http://export.arxiv.org/abs/2404.19417,,2404.19417.pdf,Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World,"Backdoor attacks have been well-studied in visible light object detection
+(VLOD) in recent years. However, VLOD can not effectively work in dark and
+temperature-sensitive scenarios. Instead, thermal infrared object detection
+(TIOD) is the most accessible and practical in such environments. In this
+paper, our team is the first to investigate the security vulnerabilities
+associated with TIOD in the context of backdoor attacks, spanning both the
+digital and physical realms. We introduce two novel types of backdoor attacks
+on TIOD, each offering unique capabilities: Object-affecting Attack and
+Range-affecting Attack. We conduct a comprehensive analysis of key factors
+influencing trigger design, which include temperature, size, material, and
+concealment. These factors, especially temperature, significantly impact the
+efficacy of backdoor attacks on TIOD. A thorough understanding of these factors
+will serve as a foundation for designing physical triggers and temperature
+controlling experiments. Our study includes extensive experiments conducted in
+both digital and physical environments. In the digital realm, we evaluate our
+approach using benchmark datasets for TIOD, achieving an Attack Success Rate
+(ASR) of up to 98.21%. In the physical realm, we test our approach in two
+real-world settings: a traffic intersection and a parking lot, using a thermal
+infrared camera. Here, we attain an ASR of up to 98.38%.",cs.CV,['cs.CV']
+Diffusion Model Alignment Using Direct Preference Optimization,Bram Wallace · Meihua Dang · Rafael Rafailov · Linqi Zhou · Aaron Lou · Senthil Purushwalkam · Stefano Ermon · Caiming Xiong · Shafiq Joty · Nikhil Naik, ,https://arxiv.org/abs/2311.12908,,2311.12908.pdf,Diffusion Model Alignment Using Direct Preference Optimization,"Large language models (LLMs) are fine-tuned using human comparison data with
+Reinforcement Learning from Human Feedback (RLHF) methods to make them better
+aligned with users' preferences. In contrast to LLMs, human preference learning
+has not been widely explored in text-to-image diffusion models; the best
+existing approach is to fine-tune a pretrained model using carefully curated
+high quality images and captions to improve visual appeal and text alignment.
+We propose Diffusion-DPO, a method to align diffusion models to human
+preferences by directly optimizing on human comparison data. Diffusion-DPO is
+adapted from the recently developed Direct Preference Optimization (DPO), a
+simpler alternative to RLHF which directly optimizes a policy that best
+satisfies human preferences under a classification objective. We re-formulate
+DPO to account for a diffusion model notion of likelihood, utilizing the
+evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
+dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
+of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
+Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
+SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
+model in human evaluation, improving visual appeal and prompt alignment. We
+also develop a variant that uses AI feedback and has comparable performance to
+training on human preferences, opening the door for scaling of diffusion model
+alignment methods.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+LAMP: Learn A Motion Pattern for Few-Shot Video Generation,Rui-Qi Wu · Liangyu Chen · Tong Yang · Chun-Le Guo · Chongyi Li · Xiangyu Zhang,https://rq-wu.github.io/projects/LAMP/index.html,https://arxiv.org/abs/2310.10769,,2310.10769.pdf,LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation,"With the impressive progress in diffusion-based text-to-image generation,
+extending such powerful generative ability to text-to-video raises enormous
+attention. Existing methods either require large-scale text-video pairs and a
+large number of training resources or learn motions that are precisely aligned
+with template videos. It is non-trivial to balance a trade-off between the
+degree of generation freedom and the resource costs for video generation. In
+our study, we present a few-shot-based tuning framework, LAMP, which enables
+text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos
+on a single GPU. Specifically, we design a first-frame-conditioned pipeline
+that uses an off-the-shelf text-to-image model for content generation so that
+our tuned video diffusion model mainly focuses on motion learning. The
+well-developed text-to-image techniques can provide visually pleasing and
+diverse content as generation conditions, which highly improves video quality
+and generation freedom. To capture the features of temporal dimension, we
+expand the pretrained 2D convolution layers of the T2I model to our novel
+temporal-spatial motion learning layers and modify the attention blocks to the
+temporal level. Additionally, we develop an effective inference trick,
+shared-noise sampling, which can improve the stability of videos with
+computational costs. Our method can also be flexibly applied to other tasks,
+e.g. real-world image animation and video editing. Extensive experiments
+demonstrate that LAMP can effectively learn the motion pattern on limited data
+and generate high-quality videos. The code and models are available at
+https://rq-wu.github.io/projects/LAMP.",cs.CV,['cs.CV']
+Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation,Xiao Lin · Wenfei Yang · Yuan Gao · Tianzhu Zhang, ,https://arxiv.org/abs/2403.19527,,2403.19527.pdf,Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation,"Category-level 6D object pose estimation aims to estimate the rotation,
+translation and size of unseen instances within specific categories. In this
+area, dense correspondence-based methods have achieved leading performance.
+However, they do not explicitly consider the local and global geometric
+information of different instances, resulting in poor generalization ability to
+unseen instances with significant shape variations. To deal with this problem,
+we propose a novel Instance-Adaptive and Geometric-Aware Keypoint Learning
+method for category-level 6D object pose estimation (AG-Pose), which includes
+two key designs: (1) The first design is an Instance-Adaptive Keypoint
+Detection module, which can adaptively detect a set of sparse keypoints for
+various instances to represent their geometric structures. (2) The second
+design is a Geometric-Aware Feature Aggregation module, which can efficiently
+integrate the local and global geometric information into keypoint features.
+These two modules can work together to establish robust keypoint-level
+correspondences for unseen instances, thus enhancing the generalization ability
+of the model.Experimental results on CAMERA25 and REAL275 datasets show that
+the proposed AG-Pose outperforms state-of-the-art methods by a large margin
+without category-specific shape priors.",cs.CV,['cs.CV']
+RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback,Tianyu Yu · Yuan Yao · Haoye Zhang · Taiwen He · Yifeng Han · Ganqu Cui · Jinyi Hu · Zhiyuan Liu · Hai-Tao Zheng · Maosong Sun,https://github.com/RLHF-V/RLHF-V,https://arxiv.org/abs/2312.00849,,2312.00849.pdf,RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback,"Multimodal Large Language Models (MLLMs) have recently demonstrated
+impressive capabilities in multimodal understanding, reasoning, and
+interaction. However, existing MLLMs prevalently suffer from serious
+hallucination problems, generating text that is not factually grounded in
+associated images. The problem makes existing MLLMs untrustworthy and thus
+impractical in real-world (especially high-stakes) applications. To address the
+challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
+alignment from fine-grained correctional human feedback. Specifically, RLHF-V
+collects human preference in the form of segment-level corrections on
+hallucinations, and performs dense direct preference optimization over the
+human feedback. Comprehensive experiments on five benchmarks in both automatic
+and human evaluation show that, RLHF-V can enable substantially more
+trustworthy MLLM behaviors with promising data and computation efficiency.
+Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
+hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
+LLaVA-RLHF trained on 10k annotated data. The final model achieves
+state-of-the-art performance in trustworthiness among open-source MLLMs, and
+shows better robustness than GPT-4V in preventing hallucinations aroused from
+over-generalization. We open-source our code, model, and data at
+https://github.com/RLHF-V/RLHF-V.",cs.CL,"['cs.CL', 'cs.CV']"
+"WWW: A Unified Framework for Explaining What, Where and Why of Neural Networks by Interpretation of Neuron Concept",Yong Hyun Ahn · Hyeon Bae Kim · Seong Tae Kim, ,https://arxiv.org/abs/2402.18956,,2402.18956.pdf,"WWW: A Unified Framework for Explaining What, Where and Why of Neural Networks by Interpretation of Neuron Concepts","Recent advancements in neural networks have showcased their remarkable
+capabilities across various domains. Despite these successes, the ""black box""
+problem still remains. Addressing this, we propose a novel framework, WWW, that
+offers the 'what', 'where', and 'why' of the neural network decisions in
+human-understandable terms. Specifically, WWW utilizes adaptive selection for
+concept discovery, employing adaptive cosine similarity and thresholding
+techniques to effectively explain 'what'. To address the 'where' and 'why', we
+proposed a novel combination of neuron activation maps (NAMs) with Shapley
+values, generating localized concept maps and heatmaps for individual inputs.
+Furthermore, WWW introduces a method for predicting uncertainty, leveraging
+heatmap similarities to estimate 'how' reliable the prediction is. Experimental
+evaluations of WWW demonstrate superior performance in both quantitative and
+qualitative metrics, outperforming existing methods in interpretability. WWW
+provides a unified solution for explaining 'what', 'where', and 'why',
+introducing a method for localized explanations from global interpretations and
+offering a plug-and-play solution adaptable to various architectures.",cs.CV,['cs.CV']
+Towards Variable and Coordinated Holistic Co-Speech Motion Generation,Yifei Liu · Qiong Cao · Yandong Wen · Huaiguang Jiang · Changxing Ding, ,https://arxiv.org/abs/2404.00368,,2404.00368.pdf,Towards Variable and Coordinated Holistic Co-Speech Motion Generation,"This paper addresses the problem of generating lifelike holistic co-speech
+motions for 3D avatars, focusing on two key aspects: variability and
+coordination. Variability allows the avatar to exhibit a wide range of motions
+even with similar speech content, while coordination ensures a harmonious
+alignment among facial expressions, hand gestures, and body poses. We aim to
+achieve both with ProbTalk, a unified probabilistic framework designed to
+jointly model facial, hand, and body movements in speech. ProbTalk builds on
+the variational autoencoder (VAE) architecture and incorporates three core
+designs. First, we introduce product quantization (PQ) to the VAE, which
+enriches the representation of complex holistic motion. Second, we devise a
+novel non-autoregressive model that embeds 2D positional encoding into the
+product-quantized representation, thereby preserving essential structure
+information of the PQ codes. Last, we employ a secondary stage to refine the
+preliminary prediction, further sharpening the high-frequency details. Coupling
+these three designs enables ProbTalk to generate natural and diverse holistic
+co-speech motions, outperforming several state-of-the-art methods in
+qualitative and quantitative evaluations, particularly in terms of realism. Our
+code and model will be released for research purposes at
+https://feifeifeiliu.github.io/probtalk/.",cs.CV,['cs.CV']
+MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images,Junwen Huang · Hao Yu · Kuan-Ting Yu · Nassir Navab · Slobodan Ilic · Benjamin Busam, ,https://arxiv.org/abs/2403.01517,,2403.01517.pdf,MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images,"Recent learning methods for object pose estimation require resource-intensive
+training for each individual object instance or category, hampering their
+scalability in real applications when confronted with previously unseen
+objects. In this paper, we propose MatchU, a Fuse-Describe-Match strategy for
+6D pose estimation from RGB-D images. MatchU is a generic approach that fuses
+2D texture and 3D geometric cues for 6D pose prediction of unseen objects. We
+rely on learning geometric 3D descriptors that are rotation-invariant by
+design. By encoding pose-agnostic geometry, the learned descriptors naturally
+generalize to unseen objects and capture symmetries. To tackle ambiguous
+associations using 3D geometry only, we fuse additional RGB information into
+our descriptor. This is achieved through a novel attention-based mechanism that
+fuses cross-modal information, together with a matching loss that leverages the
+latent space learned from RGB data to guide the descriptor learning process.
+Extensive experiments reveal the generalizability of both the RGB-D fusion
+strategy as well as the descriptor efficacy. Benefiting from the novel designs,
+MatchU surpasses all existing methods by a significant margin in terms of both
+accuracy and speed, even without the requirement of expensive re-training or
+rendering.",cs.CV,['cs.CV']
+SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining,Chull Hwan Song · Taebaek Hwang · Jooyoung Yoon · Shunghyun Choi · Yeong Hyeon Gu, ,https://arxiv.org/abs/2404.01156,,2404.01156.pdf,SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining,"Vision-language models (VLMs) have made significant strides in cross-modal
+understanding through large-scale paired datasets. However, in fashion domain,
+datasets often exhibit a disparity between the information conveyed in image
+and text. This issue stems from datasets containing multiple images of a single
+fashion item all paired with one text, leading to cases where some textual
+details are not visible in individual images. This mismatch, particularly when
+non-co-occurring elements are masked, undermines the training of conventional
+VLM objectives like Masked Language Modeling and Masked Image Modeling, thereby
+hindering the model's ability to accurately align fine-grained visual and
+textual features. Addressing this problem, we propose Synchronized attentional
+Masking (SyncMask), which generate masks that pinpoint the image patches and
+word tokens where the information co-occur in both image and text. This
+synchronization is accomplished by harnessing cross-attentional features
+obtained from a momentum model, ensuring a precise alignment between the two
+modalities. Additionally, we enhance grouped batch sampling with semi-hard
+negatives, effectively mitigating false negative issues in Image-Text Matching
+and Image-Text Contrastive learning objectives within fashion datasets. Our
+experiments demonstrate the effectiveness of the proposed approach,
+outperforming existing methods in three downstream tasks.",cs.CV,"['cs.CV', 'cs.AI']"
+Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception,Junwen He · Yifan Wang · Lijun Wang · Huchuan Lu · Bin Luo · Jun-Yan He · Jin-Peng Lan · Xuansong Xie, ,https://arxiv.org/abs/2403.02969,,2403.02969.pdf,Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception,"Multimodal Large Language Model (MLLMs) leverages Large Language Models as a
+cognitive framework for diverse visual-language tasks. Recent efforts have been
+made to equip MLLMs with visual perceiving and grounding capabilities. However,
+there still remains a gap in providing fine-grained pixel-level perceptions and
+extending interactions beyond text-specific inputs. In this work, we propose
+{\bf{AnyRef}}, a general MLLM model that can generate pixel-wise object
+perceptions and natural language descriptions from multi-modality references,
+such as texts, boxes, images, or audio. This innovation empowers users with
+greater flexibility to engage with the model beyond textual and regional
+prompts, without modality-specific designs. Through our proposed refocusing
+mechanism, the generated grounding output is guided to better focus on the
+referenced object, implicitly incorporating additional pixel-level supervision.
+This simple modification utilizes attention scores generated during the
+inference of LLM, eliminating the need for extra computations while exhibiting
+performance enhancements in both grounding masks and referring expressions.
+With only publicly available training data, our model achieves state-of-the-art
+results across multiple benchmarks, including diverse modality referring
+segmentation and region-level referring expression generation.",cs.CV,['cs.CV']
+Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes,Haobin Duan · Miao Wang · Yanxun Li · Yong-Liang Yang, ,https://arxiv.org/abs/2311.15637,,2311.15637.pdf,Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes,"We present Neural 3D Strokes, a novel technique to generate stylized images
+of a 3D scene at arbitrary novel views from multi-view 2D images. Different
+from existing methods which apply stylization to trained neural radiance fields
+at the voxel level, our approach draws inspiration from image-to-painting
+methods, simulating the progressive painting process of human artwork with
+vector strokes. We develop a palette of stylized 3D strokes from basic
+primitives and splines, and consider the 3D scene stylization task as a
+multi-view reconstruction process based on these 3D stroke primitives. Instead
+of directly searching for the parameters of these 3D strokes, which would be
+too costly, we introduce a differentiable renderer that allows optimizing
+stroke parameters using gradient descent, and propose a training scheme to
+alleviate the vanishing gradient issue. The extensive evaluation demonstrates
+that our approach effectively synthesizes 3D scenes with significant geometric
+and aesthetic stylization while maintaining a consistent appearance across
+different views. Our method can be further integrated with style loss and
+image-text contrastive models to extend its applications, including color
+transfer and text-driven 3D scene drawing. Results and code are available at
+http://buaavrcg.github.io/Neural3DStrokes.",cs.CV,"['cs.CV', 'cs.GR']"
+A Theory of Joint Light and Heat Transport for Lambertian Scenes,Mani Ramanagopal · Sriram Narayanan · Aswin C. Sankaranarayanan · Srinivasa G. Narasimhan, ,,https://dl.acm.org/doi/10.1145/3596711.3596745,,,,,nan
+Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences,Seungwook Kim · Kejie Li · Xueqing Deng · Yichun Shi · Minsu Cho · Peng Wang, ,https://arxiv.org/abs/2404.10603,,2404.10603.pdf,Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences,"Leveraging multi-view diffusion models as priors for 3D optimization have
+alleviated the problem of 3D consistency, e.g., the Janus face problem or the
+content drift problem, in zero-shot text-to-3D models. However, the 3D
+geometric fidelity of the output remains an unresolved issue; albeit the
+rendered 2D views are realistic, the underlying geometry may contain errors
+such as unreasonable concavities. In this work, we propose CorrespondentDream,
+an effective method to leverage annotation-free, cross-view correspondences
+yielded from the diffusion U-Net to provide additional 3D prior to the NeRF
+optimization process. We find that these correspondences are strongly
+consistent with human perception, and by adopting it in our loss design, we are
+able to produce NeRF models with geometries that are more coherent with common
+sense, e.g., more smoothed object surface, yielding higher 3D fidelity. We
+demonstrate the efficacy of our approach through various comparative
+qualitative results and a solid user study.",cs.CV,['cs.CV']
+Generative Region-Language Pretraining for Open-Ended Object Detection,Chuang Lin · Yi Jiang · Lizhen Qu · Zehuan Yuan · Jianfei Cai, ,https://arxiv.org/abs/2403.10191,,2403.10191.pdf,Generative Region-Language Pretraining for Open-Ended Object Detection,"In recent research, significant attention has been devoted to the
+open-vocabulary object detection task, aiming to generalize beyond the limited
+number of classes labeled during training and detect objects described by
+arbitrary category names at inference. Compared with conventional object
+detection, open vocabulary object detection largely extends the object
+detection categories. However, it relies on calculating the similarity between
+image regions and a set of arbitrary category names with a pretrained
+vision-and-language model. This implies that, despite its open-set nature, the
+task still needs the predefined object categories during the inference stage.
+This raises the question: What if we do not have exact knowledge of object
+categories during inference? In this paper, we call such a new setting as
+generative open-ended object detection, which is a more general and practical
+problem. To address it, we formulate object detection as a generative problem
+and propose a simple framework named GenerateU, which can detect dense objects
+and generate their names in a free-form way. Particularly, we employ Deformable
+DETR as a region proposal generator with a language model translating visual
+regions to object names. To assess the free-form object detection task, we
+introduce an evaluation method designed to quantitatively measure the
+performance of generative outcomes. Extensive experiments demonstrate strong
+zero-shot detection performance of our GenerateU. For example, on the LVIS
+dataset, our GenerateU achieves comparable results to the open-vocabulary
+object detection method GLIP, even though the category names are not seen by
+GenerateU during inference. Code is available at: https://
+github.com/FoundationVision/GenerateU .",cs.CV,['cs.CV']
+GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding,Chengyao Wang · Li Jiang · Xiaoyang Wu · Zhuotao Tian · Bohao Peng · Hengshuang Zhao · Jiaya Jia,https://github.com/dvlab-research/GroupContrast,https://arxiv.org/abs/2403.09639,,2403.09639.pdf,GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding,"Self-supervised 3D representation learning aims to learn effective
+representations from large-scale unlabeled point clouds. Most existing
+approaches adopt point discrimination as the pretext task, which assigns
+matched points in two distinct views as positive pairs and unmatched points as
+negative pairs. However, this approach often results in semantically identical
+points having dissimilar representations, leading to a high number of false
+negatives and introducing a ""semantic conflict"" problem. To address this issue,
+we propose GroupContrast, a novel approach that combines segment grouping and
+semantic-aware contrastive learning. Segment grouping partitions points into
+semantically meaningful regions, which enhances semantic coherence and provides
+semantic guidance for the subsequent contrastive representation learning.
+Semantic-aware contrastive learning augments the semantic information extracted
+from segment grouping and helps to alleviate the issue of ""semantic conflict"".
+We conducted extensive experiments on multiple 3D scene understanding tasks.
+The results demonstrate that GroupContrast learns semantically meaningful
+representations and achieves promising transfer learning performance.",cs.CV,['cs.CV']
+Improved Visual Grounding through Self-Consistent Explanations,Ruozhen He · Paola Cascante-Bonilla · Ziyan Yang · Alex Berg · Vicente Ordonez,https://catherine-r-he.github.io/SelfEQ/,https://arxiv.org/abs/2312.04554v1,,2312.04554v1.pdf,Improved Visual Grounding through Self-Consistent Explanations,"Vision-and-language models trained to match images with text can be combined
+with visual explanation methods to point to the locations of specific objects
+in an image. Our work shows that the localization --""grounding""-- abilities of
+these models can be further improved by finetuning for self-consistent visual
+explanations. We propose a strategy for augmenting existing text-image datasets
+with paraphrases using a large language model, and SelfEQ, a weakly-supervised
+strategy on visual explanation maps for paraphrases that encourages
+self-consistency. Specifically, for an input textual phrase, we attempt to
+generate a paraphrase and finetune the model so that the phrase and paraphrase
+map to the same region in the image. We posit that this both expands the
+vocabulary that the model is able to handle, and improves the quality of the
+object locations highlighted by gradient-based visual explanation methods (e.g.
+GradCAM). We demonstrate that SelfEQ improves performance on Flickr30k,
+ReferIt, and RefCOCO+ over a strong baseline method and several prior works.
+Particularly, comparing to other methods that do not use any type of box
+annotations, we obtain 84.07% on Flickr30k (an absolute improvement of 4.69%),
+67.40% on ReferIt (an absolute improvement of 7.68%), and 75.10%, 55.49% on
+RefCOCO+ test sets A and B respectively (an absolute improvement of 3.74% on
+average).",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution,Shangchen Zhou · Peiqing Yang · Jianyi Wang · Yihang Luo · Chen Change Loy, ,https://arxiv.org/abs/2312.06640,,2312.06640.pdf,Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution,"Text-based diffusion models have exhibited remarkable success in generation
+and editing, showing great promise for enhancing visual content with their
+generative prior. However, applying these models to video super-resolution
+remains challenging due to the high demands for output fidelity and temporal
+consistency, which is complicated by the inherent randomness in diffusion
+models. Our study introduces Upscale-A-Video, a text-guided latent diffusion
+framework for video upscaling. This framework ensures temporal coherence
+through two key mechanisms: locally, it integrates temporal layers into U-Net
+and VAE-Decoder, maintaining consistency within short sequences; globally,
+without training, a flow-guided recurrent latent propagation module is
+introduced to enhance overall video stability by propagating and fusing latent
+across the entire sequences. Thanks to the diffusion paradigm, our model also
+offers greater flexibility by allowing text prompts to guide texture creation
+and adjustable noise levels to balance restoration and generation, enabling a
+trade-off between fidelity and quality. Extensive experiments show that
+Upscale-A-Video surpasses existing methods in both synthetic and real-world
+benchmarks, as well as in AI-generated videos, showcasing impressive visual
+realism and temporal consistency.",cs.CV,['cs.CV']
+Image Neural Field Diffusion Models,Yinbo Chen · Oliver Wang · Richard Zhang · Eli Shechtman · Xiaolong Wang · Michaël Gharbi, ,https://arxiv.org/abs/2310.08337,,2310.08337.pdf,Neural Diffusion Models,"Diffusion models have shown remarkable performance on many generative tasks.
+Despite recent success, most diffusion models are restricted in that they only
+allow linear transformation of the data distribution. In contrast, broader
+family of transformations can potentially help train generative distributions
+more efficiently, simplifying the reverse process and closing the gap between
+the true negative log-likelihood and the variational approximation. In this
+paper, we present Neural Diffusion Models (NDMs), a generalization of
+conventional diffusion models that enables defining and learning time-dependent
+non-linear transformations of data. We show how to optimise NDMs using a
+variational bound in a simulation-free setting. Moreover, we derive a
+time-continuous formulation of NDMs, which allows fast and reliable inference
+using off-the-shelf numerical ODE and SDE solvers. Finally, we demonstrate the
+utility of NDMs with learnable transformations through experiments on standard
+image generation benchmarks, including CIFAR-10, downsampled versions of
+ImageNet and CelebA-HQ. NDMs outperform conventional diffusion models in terms
+of likelihood and produce high-quality samples.",cs.LG,"['cs.LG', 'stat.ML']"
+ViTamin: Designing Scalable Vision Models in the Vision-Language Era,Jieneng Chen · Qihang Yu · Xiaohui Shen · Alan L. Yuille · Liang-Chieh Chen, ,https://arxiv.org/abs/2404.02132,,2404.02132.pdf,ViTamin: Designing Scalable Vision Models in the Vision-Language Era,"Recent breakthroughs in vision-language models (VLMs) start a new page in the
+vision community. The VLMs provide stronger and more generalizable feature
+embeddings compared to those from ImageNet-pretrained models, thanks to the
+training on the large-scale Internet image-text pairs. However, despite the
+amazing achievement from the VLMs, vanilla Vision Transformers (ViTs) remain
+the default choice for the image encoder. Although pure transformer proves its
+effectiveness in the text encoding area, it remains questionable whether it is
+also the case for image encoding, especially considering that various types of
+networks are proposed on the ImageNet benchmark, which, unfortunately, are
+rarely studied in VLMs. Due to small data/model scale, the original conclusions
+of model design on ImageNet can be limited and biased. In this paper, we aim at
+building an evaluation protocol of vision models in the vision-language era
+under the contrastive language-image pretraining (CLIP) framework. We provide a
+comprehensive way to benchmark different vision models, covering their
+zero-shot performance and scalability in both model and training data sizes. To
+this end, we introduce ViTamin, a new vision models tailored for VLMs.
+ViTamin-L significantly outperforms ViT-L by 2.0% ImageNet zero-shot accuracy,
+when using the same publicly available DataComp-1B dataset and the same
+OpenCLIP training scheme. ViTamin-L presents promising results on 60 diverse
+benchmarks, including classification, retrieval, open-vocabulary detection and
+segmentation, and large multi-modal models. When further scaling up the model
+size, our ViTamin-XL with only 436M parameters attains 82.9% ImageNet zero-shot
+accuracy, surpassing 82.0% achieved by EVA-E that has ten times more parameters
+(4.4B).",cs.CV,['cs.CV']
+Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery,Mubashir Noman · Muzammal Naseer · Hisham Cholakkal · Rao Anwer · Salman Khan · Fahad Shahbaz Khan, ,https://web3.arxiv.org/abs/2403.05419,,2403.05419.pdf,Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery,"Recent advances in unsupervised learning have demonstrated the ability of
+large vision models to achieve promising results on downstream tasks by
+pre-training on large amount of unlabelled data. Such pre-training techniques
+have also been explored recently in the remote sensing domain due to the
+availability of large amount of unlabelled data. Different from standard
+natural image datasets, remote sensing data is acquired from various sensor
+technologies and exhibit diverse range of scale variations as well as
+modalities. Existing satellite image pre-training methods either ignore the
+scale information present in the remote sensing imagery or restrict themselves
+to use only a single type of data modality. In this paper, we re-visit
+transformers pre-training and leverage multi-scale information that is
+effectively utilized with multiple modalities. Our proposed approach, named
+SatMAE++, performs multi-scale pre-training and utilizes convolution based
+upsampling blocks to reconstruct the image at higher scales making it
+extensible to include more scales. Compared to existing works, the proposed
+SatMAE++ with multi-scale pre-training is equally effective for both optical as
+well as multi-spectral imagery. Extensive experiments on six datasets reveal
+the merits of proposed contributions, leading to state-of-the-art performance
+on all datasets. SatMAE++ achieves mean average precision (mAP) gain of 2.5\%
+for multi-label classification task on BigEarthNet dataset. Our code and
+pre-trained models are available at \url{https://github.com/techmn/satmae_pp}.",cs.CV,['cs.CV']
+RCL: Reliable Continual Learning for Unified Failure Detection,Fei Zhu · Zhen Cheng · Xu-Yao Zhang · Cheng-Lin Liu · Zhaoxiang Zhang, ,https://arxiv.org/abs/2403.02886,,2403.02886.pdf,Revisiting Confidence Estimation: Towards Reliable Failure Prediction,"Reliable confidence estimation is a challenging yet fundamental requirement
+in many risk-sensitive applications. However, modern deep neural networks are
+often overconfident for their incorrect predictions, i.e., misclassified
+samples from known classes, and out-of-distribution (OOD) samples from unknown
+classes. In recent years, many confidence calibration and OOD detection methods
+have been developed. In this paper, we find a general, widely existing but
+actually-neglected phenomenon that most confidence estimation methods are
+harmful for detecting misclassification errors. We investigate this problem and
+reveal that popular calibration and OOD detection methods often lead to worse
+confidence separation between correctly classified and misclassified examples,
+making it difficult to decide whether to trust a prediction or not. Finally, we
+propose to enlarge the confidence gap by finding flat minima, which yields
+state-of-the-art failure prediction performance under various settings
+including balanced, long-tailed, and covariate-shift classification scenarios.
+Our study not only provides a strong baseline for reliable confidence
+estimation but also acts as a bridge between understanding calibration, OOD
+detection, and failure prediction. The code is available at
+\url{https://github.com/Impression2805/FMFP}.",cs.CV,"['cs.CV', 'cs.LG']"
+Boosting Neural Representations for Videos with a Conditional Decoder,XINJIE ZHANG · Ren Yang · Dailan He · Xingtong Ge · Tongda Xu · Yan Wang · Hongwei Qin · Jun Zhang,https://github.com/Xinjie-Q/Boosting-NeRV,https://arxiv.org/abs/2402.18152,,2402.18152.pdf,Boosting Neural Representations for Videos with a Conditional Decoder,"Implicit neural representations (INRs) have emerged as a promising approach
+for video storage and processing, showing remarkable versatility across various
+video tasks. However, existing methods often fail to fully leverage their
+representation capabilities, primarily due to inadequate alignment of
+intermediate features during target frame decoding. This paper introduces a
+universal boosting framework for current implicit video representation
+approaches. Specifically, we utilize a conditional decoder with a
+temporal-aware affine transform module, which uses the frame index as a prior
+condition to effectively align intermediate features with target frames.
+Besides, we introduce a sinusoidal NeRV-like block to generate diverse
+intermediate features and achieve a more balanced parameter distribution,
+thereby enhancing the model's capacity. With a high-frequency
+information-preserving reconstruction loss, our approach successfully boosts
+multiple baseline INRs in the reconstruction quality and convergence speed for
+video regression, and exhibits superior inpainting and interpolation results.
+Further, we integrate a consistent entropy minimization technique and develop
+video codecs based on these boosted INRs. Experiments on the UVG dataset
+confirm that our enhanced codecs significantly outperform baseline INRs and
+offer competitive rate-distortion performance compared to traditional and
+learning-based codecs. Code is available at
+https://github.com/Xinjie-Q/Boosting-NeRV.",eess.IV,"['eess.IV', 'cs.AI', 'cs.CV']"
+Uncertainty Visualization via Low-Dimensional Posterior Projections,Omer Yair · Tomer Michaeli · Elias Nehme, ,https://arxiv.org/abs/2312.07804,,2312.07804.pdf,Uncertainty Visualization via Low-Dimensional Posterior Projections,"In ill-posed inverse problems, it is commonly desirable to obtain insight
+into the full spectrum of plausible solutions, rather than extracting only a
+single reconstruction. Information about the plausible solutions and their
+likelihoods is encoded in the posterior distribution. However, for
+high-dimensional data, this distribution is challenging to visualize. In this
+work, we introduce a new approach for estimating and visualizing posteriors by
+employing energy-based models (EBMs) over low-dimensional subspaces.
+Specifically, we train a conditional EBM that receives an input measurement and
+a set of directions that span some low-dimensional subspace of solutions, and
+outputs the probability density function of the posterior within that space. We
+demonstrate the effectiveness of our method across a diverse range of datasets
+and image restoration problems, showcasing its strength in uncertainty
+quantification and visualization. As we show, our method outperforms a baseline
+that projects samples from a diffusion-based posterior sampler, while being
+orders of magnitude faster. Furthermore, it is more accurate than a baseline
+that assumes a Gaussian posterior.",cs.CV,['cs.CV']
+ElasticDiffusion: Training-free Arbitrary Size Image Generation,Moayed Haji Ali · Guha Balakrishnan · Vicente Ordonez, ,https://arxiv.org/abs/2311.18822,,2311.18822.pdf,ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation,"Diffusion models have revolutionized image generation in recent years, yet
+they are still limited to a few sizes and aspect ratios. We propose
+ElasticDiffusion, a novel training-free decoding method that enables pretrained
+text-to-image diffusion models to generate images with various sizes.
+ElasticDiffusion attempts to decouple the generation trajectory of a pretrained
+model into local and global signals. The local signal controls low-level pixel
+information and can be estimated on local patches, while the global signal is
+used to maintain overall structural consistency and is estimated with a
+reference image. We test our method on CelebA-HQ (faces) and LAION-COCO
+(objects/indoor/outdoor scenes). Our experiments and qualitative results show
+superior image coherence quality across aspect ratios compared to
+MultiDiffusion and the standard decoding strategy of Stable Diffusion. Project
+page: https://elasticdiffusion.github.io/",cs.CV,['cs.CV']
+Exploiting Diffusion Prior for Generalizable Dense Prediction,Hsin-Ying Lee · Hung-Yu Tseng · Hsin-Ying Lee · Ming-Hsuan Yang,https://shinying.github.io/dmp,https://arxiv.org/abs/2311.18832,,2311.18832.pdf,Exploiting Diffusion Prior for Generalizable Dense Prediction,"Contents generated by recent advanced Text-to-Image (T2I) diffusion models
+are sometimes too imaginative for existing off-the-shelf dense predictors to
+estimate due to the immitigable domain gap. We introduce DMP, a pipeline
+utilizing pre-trained T2I models as a prior for dense prediction tasks. To
+address the misalignment between deterministic prediction tasks and stochastic
+T2I models, we reformulate the diffusion process through a sequence of
+interpolations, establishing a deterministic mapping between input RGB images
+and output prediction distributions. To preserve generalizability, we use
+low-rank adaptation to fine-tune pre-trained models. Extensive experiments
+across five tasks, including 3D property estimation, semantic segmentation, and
+intrinsic image decomposition, showcase the efficacy of the proposed method.
+Despite limited-domain training data, the approach yields faithful estimations
+for arbitrary images, surpassing existing state-of-the-art algorithms.",cs.CV,['cs.CV']
+Make-It-Vivid: Dressing Your Animatable Biped Cartoon Characters from Text,Junshu Tang · Yanhong Zeng · Ke Fan · Xuheng Wang · Bo Dai · Kai Chen · Lizhuang Ma, ,https://arxiv.org/abs/2403.16897,,2403.16897.pdf,Make-It-Vivid: Dressing Your Animatable Biped Cartoon Characters from Text,"Creating and animating 3D biped cartoon characters is crucial and valuable in
+various applications. Compared with geometry, the diverse texture design plays
+an important role in making 3D biped cartoon characters vivid and charming.
+Therefore, we focus on automatic texture design for cartoon characters based on
+input instructions. This is challenging for domain-specific requirements and a
+lack of high-quality data. To address this challenge, we propose Make-It-Vivid,
+the first attempt to enable high-quality texture generation from text in UV
+space. We prepare a detailed text-texture paired data for 3D characters by
+using vision-question-answering agents. Then we customize a pretrained
+text-to-image model to generate texture map with template structure while
+preserving the natural 2D image knowledge. Furthermore, to enhance fine-grained
+details, we propose a novel adversarial learning scheme to shorten the domain
+gap between original dataset and realistic texture domain. Extensive
+experiments show that our approach outperforms current texture generation
+methods, resulting in efficient character texturing and faithful generation
+with prompts. Besides, we showcase various applications such as out of domain
+generation and texture stylization. We also provide an efficient generation
+system for automatic text-guided textured character generation and animation.",cs.CV,['cs.CV']
+Eclipse: Disambiguating Illumination and Materials using Unintended Shadows,Dor Verbin · Ben Mildenhall · Peter Hedman · Jonathan T. Barron · Todd Zickler · Pratul P. Srinivasan, ,,https://www.youtube.com/watch?v=amQLGyza3EU,,,,,nan
+Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection,Zhiyuan Yan · Yuhao Luo · Siwei Lyu · Qingshan Liu · Baoyuan Wu, ,https://arxiv.org/abs/2311.11278v1,,2311.11278v1.pdf,Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection,"Deepfake detection faces a critical generalization hurdle, with performance
+deteriorating when there is a mismatch between the distributions of training
+and testing data. A broadly received explanation is the tendency of these
+detectors to be overfitted to forgery-specific artifacts, rather than learning
+features that are widely applicable across various forgeries. To address this
+issue, we propose a simple yet effective detector called LSDA
+(\underline{L}atent \underline{S}pace \underline{D}ata
+\underline{A}ugmentation), which is based on a heuristic idea: representations
+with a wider variety of forgeries should be able to learn a more generalizable
+decision boundary, thereby mitigating the overfitting of method-specific
+features (see Figure. 1). Following this idea, we propose to enlarge the
+forgery space by constructing and simulating variations within and across
+forgery features in the latent space. This approach encompasses the acquisition
+of enriched, domain-specific features and the facilitation of smoother
+transitions between different forgery types, effectively bridging domain gaps.
+Our approach culminates in refining a binary classifier that leverages the
+distilled knowledge from the enhanced features, striving for a generalizable
+deepfake detector. Comprehensive experiments show that our proposed method is
+surprisingly effective and transcends state-of-the-art detectors across several
+widely used benchmarks.",cs.CV,['cs.CV']
+Revisiting Sampson Approximations for Geometric Estimation Problems,Felix Rydell · Angelica Torres · Viktor Larsson, ,https://arxiv.org/abs/2401.07114,,2401.07114.pdf,Revisiting Sampson Approximations for Geometric Estimation Problems,"Many problems in computer vision can be formulated as geometric estimation
+problems, i.e. given a collection of measurements (e.g. point correspondences)
+we wish to fit a model (e.g. an essential matrix) that agrees with our
+observations. This necessitates some measure of how much an observation
+``agrees"" with a given model. A natural choice is to consider the smallest
+perturbation that makes the observation exactly satisfy the constraints.
+However, for many problems, this metric is expensive or otherwise intractable
+to compute. The so-called Sampson error approximates this geometric error
+through a linearization scheme. For epipolar geometry, the Sampson error is a
+popular choice and in practice known to yield very tight approximations of the
+corresponding geometric residual (the reprojection error).
+  In this paper we revisit the Sampson approximation and provide new
+theoretical insights as to why and when this approximation works, as well as
+provide explicit bounds on the tightness under some mild assumptions. Our
+theoretical results are validated in several experiments on real data and in
+the context of different geometric estimation tasks.",cs.CV,"['cs.CV', 'math.AG', '68T45 (Primary), 14Q99 (Secondary), 68W30']"
+Pick-or-Mix: Dynamic Channel Sampling for ConvNets,Ashish Kumar · Daneul Kim · Jaesik Park · Laxmidhar Behera, ,,https://openreview.net/forum?id=Howb7fXB4V,,,,,nan
+FreePoint: Unsupervised Point Cloud Instance Segmentation,Zhikai Zhang · Jian Ding · Li Jiang · Dengxin Dai · Gui-Song Xia, ,,https://medium.com/forestree/reviewing-unsupervised-semantic-segmentation-methods-for-point-cloud-a50a508f7f88,,,,,nan
+Mind marginal non-crack regions: Clustering-inspired representation learning for crack segmentation,zhuangzhuang chen · Zhuonan Lai · Jie Chen · Jianqiang Li, ,https://arxiv.org/html/2403.03063v1,,2403.03063v1.pdf,CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections,"Routine visual inspections of concrete structures are imperative for
+upholding the safety and integrity of critical infrastructure. Such visual
+inspections sometimes happen under low-light conditions, e.g., checking for
+bridge health. Crack segmentation under such conditions is challenging due to
+the poor contrast between cracks and their surroundings. However, most deep
+learning methods are designed for well-illuminated crack images and hence their
+performance drops dramatically in low-light scenes. In addition, conventional
+approaches require many annotated low-light crack images which is
+time-consuming. In this paper, we address these challenges by proposing
+CrackNex, a framework that utilizes reflectance information based on Retinex
+Theory to help the model learn a unified illumination-invariant representation.
+Furthermore, we utilize few-shot segmentation to solve the inefficient training
+data problem. In CrackNex, both a support prototype and a reflectance prototype
+are extracted from the support set. Then, a prototype fusion module is designed
+to integrate the features from both prototypes. CrackNex outperforms the SOTA
+methods on multiple datasets. Additionally, we present the first benchmark
+dataset, LCSD, for low-light crack segmentation. LCSD consists of 102
+well-illuminated crack images and 41 low-light crack images. The dataset and
+code are available at https://github.com/zy1296/CrackNex.",cs.CV,['cs.CV']
+MV-Adapter: Exploring Parameter Efficient Learning for Video Text Retrieval,bowen zhang · Xiaojie Jin · Weibo Gong · Kai Xu · Xueqing Deng · Peng Wang · Zhao Zhang · Xiaohui Shen · Jiashi Feng, ,https://arxiv.org/abs/2405.19465,,2405.19465.pdf,RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter,"Text-Video Retrieval (TVR) aims to align relevant video content with natural
+language queries. To date, most state-of-the-art TVR methods learn
+image-to-video transfer learning based on large-scale pre-trained
+visionlanguage models (e.g., CLIP). However, fully fine-tuning these
+pre-trained models for TVR incurs prohibitively expensive computation costs. To
+this end, we propose to conduct efficient text-video Retrieval with a
+sparse-andcorrelated AdaPter (RAP), i.e., fine-tuning the pre-trained model
+with a few parameterized layers. To accommodate the text-video scenario, we
+equip our RAP with two indispensable characteristics: temporal sparsity and
+correlation. Specifically, we propose a low-rank modulation module to refine
+the per-image features from the frozen CLIP backbone, which accentuates salient
+frames within the video features while alleviating temporal redundancy.
+Besides, we introduce an asynchronous self-attention mechanism that first
+selects the top responsive visual patches and augments the correlation modeling
+between them with learnable temporal and patch offsets. Extensive experiments
+on four TVR datasets demonstrate that RAP achieves superior or comparable
+performance compared to the fully fine-tuned counterpart and other
+parameter-efficient fine-tuning methods.",cs.CV,['cs.CV']
+Few-shot Learner Parameterization by Diffusion Time-steps,Zhongqi Yue · Pan Zhou · Richang Hong · Hanwang Zhang · Qianru Sun, ,https://arxiv.org/abs/2403.02649,,2403.02649.pdf,Few-shot Learner Parameterization by Diffusion Time-steps,"Even when using large multi-modal foundation models, few-shot learning is
+still challenging -- if there is no proper inductive bias, it is nearly
+impossible to keep the nuanced class attributes while removing the visually
+prominent attributes that spuriously correlate with class labels. To this end,
+we find an inductive bias that the time-steps of a Diffusion Model (DM) can
+isolate the nuanced class attributes, i.e., as the forward diffusion adds noise
+to an image at each time-step, nuanced attributes are usually lost at an
+earlier time-step than the spurious attributes that are visually prominent.
+Building on this, we propose Time-step Few-shot (TiF) learner. We train
+class-specific low-rank adapters for a text-conditioned DM to make up for the
+lost attributes, such that images can be accurately reconstructed from their
+noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt
+are essentially a parameterization of only the nuanced class attributes. For a
+test image, we can use the parameterization to only extract the nuanced class
+attributes for classification. TiF learner significantly outperforms OpenCLIP
+and its adapters on a variety of fine-grained and customized few-shot learning
+tasks. Codes are in https://github.com/yue-zhongqi/tif.",cs.CV,['cs.CV']
+"SPIN: Simultaneous Perception, Interaction and Navigation",Shagun Uppal · Ananye Agarwal · Haoyu Xiong · Kenneth Shaw · Deepak Pathak, ,https://arxiv.org/abs/2405.07991,,2405.07991.pdf,"SPIN: Simultaneous Perception, Interaction and Navigation","While there has been remarkable progress recently in the fields of
+manipulation and locomotion, mobile manipulation remains a long-standing
+challenge. Compared to locomotion or static manipulation, a mobile system must
+make a diverse range of long-horizon tasks feasible in unstructured and dynamic
+environments. While the applications are broad and interesting, there are a
+plethora of challenges in developing these systems such as coordination between
+the base and arm, reliance on onboard perception for perceiving and interacting
+with the environment, and most importantly, simultaneously integrating all
+these parts together. Prior works approach the problem using disentangled
+modular skills for mobility and manipulation that are trivially tied together.
+This causes several limitations such as compounding errors, delays in
+decision-making, and no whole-body coordination. In this work, we present a
+reactive mobile manipulation framework that uses an active visual system to
+consciously perceive and react to its environment. Similar to how humans
+leverage whole-body and hand-eye coordination, we develop a mobile manipulator
+that exploits its ability to move and see, more specifically -- to move in
+order to see and to see in order to move. This allows it to not only move
+around and interact with its environment but also, choose ""when"" to perceive
+""what"" using an active visual system. We observe that such an agent learns to
+navigate around complex cluttered scenarios while displaying agile whole-body
+coordination using only ego-vision without needing to create environment maps.
+Results visualizations and videos at https://spin-robot.github.io/",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV', 'cs.LG', 'cs.SY', 'eess.SY']"
+Pose-Guided Self-Training with Two-Stage Clustering for Unsupervised Landmark Discovery,Siddharth Tourani · Ahmed Alwheibi · Arif Mahmood · Muhammad Haris Khan, ,https://arxiv.org/abs/2403.16194,,2403.16194.pdf,Pose-Guided Self-Training with Two-Stage Clustering for Unsupervised Landmark Discovery,"Unsupervised landmarks discovery (ULD) for an object category is a
+challenging computer vision problem. In pursuit of developing a robust ULD
+framework, we explore the potential of a recent paradigm of self-supervised
+learning algorithms, known as diffusion models. Some recent works have shown
+that these models implicitly contain important correspondence cues. Towards
+harnessing the potential of diffusion models for the ULD task, we make the
+following core contributions. First, we propose a ZeroShot ULD baseline based
+on simple clustering of random pixel locations with nearest neighbour matching.
+It delivers better results than existing ULD methods. Second, motivated by the
+ZeroShot performance, we develop a ULD algorithm based on diffusion features
+using self-training and clustering which also outperforms prior methods by
+notable margins. Third, we introduce a new proxy task based on generating
+latent pose codes and also propose a two-stage clustering mechanism to
+facilitate effective pseudo-labeling, resulting in a significant performance
+improvement. Overall, our approach consistently outperforms state-of-the-art
+methods on four challenging benchmarks AFLW, MAFL, CatHeads and LS3D by
+significant margins.",cs.CV,['cs.CV']
+Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency,Yuqi Zhang · Han Luo · Yinjie Lei, ,https://arxiv.org/abs/2311.15383,,2311.15383.pdf,Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding,"3D Visual Grounding (3DVG) aims at localizing 3D object based on textual
+descriptions. Conventional supervised methods for 3DVG often necessitate
+extensive annotations and a predefined vocabulary, which can be restrictive. To
+address this issue, we propose a novel visual programming approach for
+zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language
+models (LLMs). Our approach begins with a unique dialog-based method, engaging
+with LLMs to establish a foundational understanding of zero-shot 3DVG. Building
+on this, we design a visual program that consists of three types of modules,
+i.e., view-independent, view-dependent, and functional modules. These modules,
+specifically tailored for 3D scenarios, work collaboratively to perform complex
+reasoning and inference. Furthermore, we develop an innovative language-object
+correlation module to extend the scope of existing 3D object detectors into
+open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot
+approach can outperform some supervised baselines, marking a significant stride
+towards effective 3DVG.",cs.CV,['cs.CV']
+POPDG: Popular 3D Dance Generation with PopDanceSet,Zhenye Luo · Min Ren · Xuecai Hu · Yongzhen Huang · Li Yao, ,https://arxiv.org/abs/2405.03178,,2405.03178.pdf,POPDG: Popular 3D Dance Generation with PopDanceSet,"Generating dances that are both lifelike and well-aligned with music
+continues to be a challenging task in the cross-modal domain. This paper
+introduces PopDanceSet, the first dataset tailored to the preferences of young
+audiences, enabling the generation of aesthetically oriented dances. And it
+surpasses the AIST++ dataset in music genre diversity and the intricacy and
+depth of dance movements. Moreover, the proposed POPDG model within the iDDPM
+framework enhances dance diversity and, through the Space Augmentation
+Algorithm, strengthens spatial physical connections between human body joints,
+ensuring that increased diversity does not compromise generation quality. A
+streamlined Alignment Module is also designed to improve the temporal alignment
+between dance and music. Extensive experiments show that POPDG achieves SOTA
+results on two datasets. Furthermore, the paper also expands on current
+evaluation metrics. The dataset and code are available at
+https://github.com/Luke-Luo1/POPDG.",cs.SD,"['cs.SD', 'eess.AS']"
+CLiC: Concept Learning in Context,Mehdi Safaee · Aryan Mikaeili · Or Patashnik · Daniel Cohen-Or · Ali Mahdavi Amiri, ,https://arxiv.org/abs/2311.17083,,2311.17083.pdf,CLiC: Concept Learning in Context,"This paper addresses the challenge of learning a local visual pattern of an
+object from one image, and generating images depicting objects with that
+pattern. Learning a localized concept and placing it on an object in a target
+image is a nontrivial task, as the objects may have different orientations and
+shapes. Our approach builds upon recent advancements in visual concept
+learning. It involves acquiring a visual concept (e.g., an ornament) from a
+source image and subsequently applying it to an object (e.g., a chair) in a
+target image. Our key idea is to perform in-context concept learning, acquiring
+the local visual concept within the broader context of the objects they belong
+to. To localize the concept learning, we employ soft masks that contain both
+the concept within the mask and the surrounding image area. We demonstrate our
+approach through object generation within an image, showcasing plausible
+embedding of in-context learned concepts. We also introduce methods for
+directing acquired concepts to specific locations within target images,
+employing cross-attention mechanisms, and establishing correspondences between
+source and target objects. The effectiveness of our method is demonstrated
+through quantitative and qualitative experiments, along with comparisons
+against baseline techniques.",cs.CV,['cs.CV']
+Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing,Xun Lin · Shuai Wang · RIZHAO CAI · Yizhong Liu · Ying Fu · Wenzhong Tang · Zitong YU · Alex C. Kot, ,https://arxiv.org/abs/2402.19298,,2402.19298.pdf,Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing,"Face Anti-Spoofing (FAS) is crucial for securing face recognition systems
+against presentation attacks. With advancements in sensor manufacture and
+multi-modal learning techniques, many multi-modal FAS approaches have emerged.
+However, they face challenges in generalizing to unseen attacks and deployment
+conditions. These challenges arise from (1) modality unreliability, where some
+modality sensors like depth and infrared undergo significant domain shifts in
+varying environments, leading to the spread of unreliable information during
+cross-modal feature fusion, and (2) modality imbalance, where training overly
+relies on a dominant modality hinders the convergence of others, reducing
+effectiveness against attack types that are indistinguishable sorely using the
+dominant modality. To address modality unreliability, we propose the
+Uncertainty-Guided Cross-Adapter (U-Adapter) to recognize unreliably detected
+regions within each modality and suppress the impact of unreliable regions on
+other modalities. For modality imbalance, we propose a Rebalanced Modality
+Gradient Modulation (ReGrad) strategy to rebalance the convergence speed of all
+modalities by adaptively adjusting their gradients. Besides, we provide the
+first large-scale benchmark for evaluating multi-modal FAS performance under
+domain generalization scenarios. Extensive experiments demonstrate that our
+method outperforms state-of-the-art methods. Source code and protocols will be
+released on https://github.com/OMGGGGG/mmdg.",cs.CV,['cs.CV']
+Alchemist: Parametric Control of Material Properties with Diffusion Models,Prafull Sharma · Varun Jampani · Yuanzhen Li · Xuhui Jia · Dmitry Lagun · Fredo Durand · William Freeman · Mark Matthews, ,https://arxiv.org/abs/2312.02970,,2312.02970.pdf,Alchemist: Parametric Control of Material Properties with Diffusion Models,"We propose a method to control material attributes of objects like roughness,
+metallic, albedo, and transparency in real images. Our method capitalizes on
+the generative prior of text-to-image models known for photorealism, employing
+a scalar value and instructions to alter low-level material properties.
+Addressing the lack of datasets with controlled material attributes, we
+generated an object-centric synthetic dataset with physically-based materials.
+Fine-tuning a modified pre-trained text-to-image model on this synthetic
+dataset enables us to edit material properties in real-world images while
+preserving all other attributes. We show the potential application of our model
+to material edited NeRFs.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR']"
+Noisy One-point Homographies are Surprisingly Good,Yaqing Ding · Jonathan Astermark · Magnus Oskarsson · Viktor Larsson, ,,https://vlarsson.github.io/publications/,,,,,nan
+Small Scale Data-Free Knowledge Distillation,He Liu · Yikai Wang · Huaping Liu · Fuchun Sun · Anbang Yao, ,https://arxiv.org/abs/2403.19539,,2403.19539.pdf,De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts,"Data-Free Knowledge Distillation (DFKD) is a promising task to train
+high-performance small models to enhance actual deployment without relying on
+the original training data. Existing methods commonly avoid relying on private
+data by utilizing synthetic or sampled data. However, a long-overlooked issue
+is that the severe distribution shifts between their substitution and original
+data, which manifests as huge differences in the quality of images and class
+proportions. The harmful shifts are essentially the confounder that
+significantly causes performance bottlenecks. To tackle the issue, this paper
+proposes a novel perspective with causal inference to disentangle the student
+models from the impact of such shifts. By designing a customized causal graph,
+we first reveal the causalities among the variables in the DFKD task.
+Subsequently, we propose a Knowledge Distillation Causal Intervention (KDCI)
+framework based on the backdoor adjustment to de-confound the confounder. KDCI
+can be flexibly combined with most existing state-of-the-art baselines.
+Experiments in combination with six representative DFKD methods demonstrate the
+effectiveness of our KDCI, which can obviously help existing methods under
+almost all settings, \textit{e.g.}, improving the baseline by up to 15.54\%
+accuracy on the CIFAR-100 dataset.",cs.CV,['cs.CV']
+Efficient Multitask Dense Predictor via Binarization,Yuzhang Shang · Dan Xu · Gaowen Liu · Ramana Kompella · Yan Yan, ,https://arxiv.org/abs/2405.14136,,2405.14136.pdf,Efficient Multitask Dense Predictor via Binarization,"Multi-task learning for dense prediction has emerged as a pivotal area in
+computer vision, enabling simultaneous processing of diverse yet interrelated
+pixel-wise prediction tasks. However, the substantial computational demands of
+state-of-the-art (SoTA) models often limit their widespread deployment. This
+paper addresses this challenge by introducing network binarization to compress
+resource-intensive multi-task dense predictors. Specifically, our goal is to
+significantly accelerate multi-task dense prediction models via Binary Neural
+Networks (BNNs) while maintaining and even improving model performance at the
+same time. To reach this goal, we propose a Binary Multi-task Dense Predictor,
+Bi-MTDP, and several variants of Bi-MTDP, in which a multi-task dense predictor
+is constructed via specified binarized modules. Our systematical analysis of
+this predictor reveals that performance drop from binarization is primarily
+caused by severe information degradation. To address this issue, we introduce a
+deep information bottleneck layer that enforces representations for downstream
+tasks satisfying Gaussian distribution in forward propagation. Moreover, we
+introduce a knowledge distillation mechanism to correct the direction of
+information flow in backward propagation. Intriguingly, one variant of Bi-MTDP
+outperforms full-precision (FP) multi-task dense prediction SoTAs, ARTC
+(CNN-based) and InvPT (ViT-Based). This result indicates that Bi-MTDP is not
+merely a naive trade-off between performance and efficiency, but is rather a
+benefit of the redundant information flow thanks to the multi-task
+architecture. Code is available at https://github.com/42Shawn/BiMTDP.",cs.CV,['cs.CV']
+Neural Super-Resolution for Real-time Rendering with Radiance Demodulation,Jia Li · Ziling Chen · Xiaolong Wu · Lu Wang · Beibei Wang · Lei Zhang, ,https://arxiv.org/abs/2308.06699,,2308.06699.pdf,Neural Super-Resolution for Real-time Rendering with Radiance Demodulation,"It is time-consuming to render high-resolution images in applications such as
+video games and virtual reality, and thus super-resolution technologies become
+increasingly popular for real-time rendering. However, it is challenging to
+preserve sharp texture details, keep the temporal stability and avoid the
+ghosting artifacts in real-time super-resolution rendering. To address this
+issue, we introduce radiance demodulation to separate the rendered image or
+radiance into a lighting component and a material component, considering the
+fact that the light component is smoother than the rendered image so that the
+high-resolution material component with detailed textures can be easily
+obtained. We perform the super-resolution on the lighting component only and
+re-modulate it with the high-resolution material component to obtain the final
+super-resolution image with more texture details. A reliable warping module is
+proposed by explicitly marking the occluded regions to avoid the ghosting
+artifacts. To further enhance the temporal stability, we design a
+frame-recurrent neural network and a temporal loss to aggregate the previous
+and current frames, which can better capture the spatial-temporal consistency
+among reconstructed frames. As a result, our method is able to produce
+temporally stable results in real-time rendering with high-quality details,
+even in the challenging 4 $\times$ 4 super-resolution scenarios.",cs.GR,['cs.GR']
+Multiple View Geometry Transformers for 3D Human Pose Estimation,Ziwei Liao · jialiang zhu · Chunyu Wang · Han Hu · Steven L. Waslander, ,https://arxiv.org/abs/2311.10983,,2311.10983.pdf,Multiple View Geometry Transformers for 3D Human Pose Estimation,"In this work, we aim to improve the 3D reasoning ability of Transformers in
+multi-view 3D human pose estimation. Recent works have focused on end-to-end
+learning-based transformer designs, which struggle to resolve geometric
+information accurately, particularly during occlusion. Instead, we propose a
+novel hybrid model, MVGFormer, which has a series of geometric and appearance
+modules organized in an iterative manner. The geometry modules are
+learning-free and handle all viewpoint-dependent 3D tasks geometrically which
+notably improves the model's generalization ability. The appearance modules are
+learnable and are dedicated to estimating 2D poses from image signals
+end-to-end which enables them to achieve accurate estimates even when occlusion
+occurs, leading to a model that is both accurate and generalizable to new
+cameras and geometries. We evaluate our approach for both in-domain and
+out-of-domain settings, where our model consistently outperforms
+state-of-the-art methods, and especially does so by a significant margin in the
+out-of-domain setting. We will release the code and models:
+https://github.com/XunshanMan/MVGFormer.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Efficient Scene Recovery Using Luminous Flux Prior,ZhongYu Li · Lei Zhang, ,,,,,,,nan
+ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe,Yifan Bai · Zeyang Zhao · Yihong Gong · Xing Wei, ,https://arxiv.org/abs/2312.17133,,2312.17133.pdf,ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe,"We present ARTrackV2, which integrates two pivotal aspects of tracking:
+determining where to look (localization) and how to describe (appearance
+analysis) the target object across video frames. Building on the foundation of
+its predecessor, ARTrackV2 extends the concept by introducing a unified
+generative framework to ""read out"" object's trajectory and ""retell"" its
+appearance in an autoregressive manner. This approach fosters a time-continuous
+methodology that models the joint evolution of motion and visual features,
+guided by previous estimates. Furthermore, ARTrackV2 stands out for its
+efficiency and simplicity, obviating the less efficient intra-frame
+autoregression and hand-tuned parameters for appearance updates. Despite its
+simplicity, ARTrackV2 achieves state-of-the-art performance on prevailing
+benchmark datasets while demonstrating remarkable efficiency improvement. In
+particular, ARTrackV2 achieves AO score of 79.5\% on GOT-10k, and AUC of 86.1\%
+on TrackingNet while being $3.6 \times$ faster than ARTrack. The code will be
+released.",cs.CV,['cs.CV']
+ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion,Jiayu Yang · Ziang Cheng · Yunfei Duan · Pan Ji · Hongdong Li, ,https://arxiv.org/abs/2310.10343,,2310.10343.pdf,ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion,"Given a single image of a 3D object, this paper proposes a novel method
+(named ConsistNet) that is able to generate multiple images of the same object,
+as if seen they are captured from different viewpoints, while the 3D
+(multi-view) consistencies among those multiple generated images are
+effectively exploited. Central to our method is a multi-view consistency block
+which enables information exchange across multiple single-view diffusion
+processes based on the underlying multi-view geometry principles. ConsistNet is
+an extension to the standard latent diffusion model, and consists of two
+sub-modules: (a) a view aggregation module that unprojects multi-view features
+into global 3D volumes and infer consistency, and (b) a ray aggregation module
+that samples and aggregate 3D consistent features back to each view to enforce
+consistency. Our approach departs from previous methods in multi-view image
+generation, in that it can be easily dropped-in pre-trained LDMs without
+requiring explicit pixel correspondences or depth prediction. Experiments show
+that our method effectively learns 3D consistency over a frozen Zero123
+backbone and can generate 16 surrounding views of the object within 40 seconds
+on a single A100 GPU. Our code will be made available on
+https://github.com/JiayuYANG/ConsistNet",cs.CV,['cs.CV']
+Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction,Junuk Cha · Jihyeon Kim · Jae Shin Yoon · Seungryul Baek, ,https://arxiv.org/abs/2404.00562,,2404.00562.pdf,Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction,"This paper introduces the first text-guided work for generating the sequence
+of hand-object interaction in 3D. The main challenge arises from the lack of
+labeled data where existing ground-truth datasets are nowhere near
+generalizable in interaction type and object category, which inhibits the
+modeling of diverse 3D hand-object interaction with the correct physical
+implication (e.g., contacts and semantics) from text prompts. To address this
+challenge, we propose to decompose the interaction generation task into two
+subtasks: hand-object contact generation; and hand-object motion generation.
+For contact generation, a VAE-based network takes as input a text and an object
+mesh, and generates the probability of contacts between the surfaces of hands
+and the object during the interaction. The network learns a variety of local
+geometry structure of diverse objects that is independent of the objects'
+category, and thus, it is applicable to general objects. For motion generation,
+a Transformer-based diffusion model utilizes this 3D contact map as a strong
+prior for generating physically plausible hand-object motion as a function of
+text prompts by learning from the augmented labeled dataset; where we annotate
+text labels from many existing 3D hand and object motion data. Finally, we
+further introduce a hand refiner module that minimizes the distance between the
+object surface and hand joints to improve the temporal stability of the
+object-hand contacts and to suppress the penetration artifacts. In the
+experiments, we demonstrate that our method can generate more realistic and
+diverse interactions compared to other baseline methods. We also show that our
+method is applicable to unseen objects. We will release our model and newly
+labeled data as a strong foundation for future research. Codes and data are
+available in: https://github.com/JunukCha/Text2HOI.",cs.CV,['cs.CV']
+Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image,Yiqun Mei · Yu Zeng · He Zhang · Zhixin Shu · Xuaner Zhang · Sai Bi · Jianming Zhang · HyunJoon Jung · Vishal M. Patel,https://yiqunmei.net/holo-web/,https://arxiv.org/abs/2403.09632,,2403.09632.pdf,Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image,"At the core of portrait photography is the search for ideal lighting and
+viewpoint. The process often requires advanced knowledge in photography and an
+elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric
+relighting method that is capable of synthesizing novel viewpoints, and novel
+lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN
+(EG3D) to reconstruct geometry and appearance from an input portrait as a set
+of 3D-aware features. We design a relighting module conditioned on a given
+lighting to process these features, and predict a relit 3D representation in
+the form of a tri-plane, which can render to an arbitrary viewpoint through
+volume rendering. Besides viewpoint and lighting control, Holo-Relighting also
+takes the head pose as a condition to enable head-pose-dependent lighting
+effects. With these novel designs, Holo-Relighting can generate complex
+non-Lambertian lighting effects (e.g., specular highlights and cast shadows)
+without using any explicit physical lighting priors. We train Holo-Relighting
+with data captured with a light stage, and propose two data-rendering
+techniques to improve the data quality for training the volumetric relighting
+system. Through quantitative and qualitative experiments, we demonstrate
+Holo-Relighting can achieve state-of-the-arts relighting quality with better
+photorealism, 3D consistency and controllability.",cs.CV,['cs.CV']
+Uncertainty-Guided Never-Ending Learning to Drive,Lei Lai · Eshed Ohn-Bar · Sanjay Arora · John Yi, ,,https://paperswithcode.com/paper/learning-to-drive-anywhere,,,,,nan
+Positive-Unlabeled Learning by Latent Group-Aware Meta Disambiguation,Lin Long · Haobo Wang · Zhijie Jiang · Lei Feng · Chang Yao · Gang Chen · Junbo Zhao, ,https://arxiv.org/abs/2307.15973,,2307.15973.pdf,Debiased Pairwise Learning from Positive-Unlabeled Implicit Feedback,"Learning contrastive representations from pairwise comparisons has achieved
+remarkable success in various fields, such as natural language processing,
+computer vision, and information retrieval. Collaborative filtering algorithms
+based on pairwise learning also rooted in this paradigm. A significant concern
+is the absence of labels for negative instances in implicit feedback data,
+which often results in the random selected negative instances contains false
+negatives and inevitably, biased embeddings. To address this issue, we
+introduce a novel correction method for sampling bias that yields a modified
+loss for pairwise learning called debiased pairwise loss (DPL). The key idea
+underlying DPL is to correct the biased probability estimates that result from
+false negatives, thereby correcting the gradients to approximate those of fully
+supervised data. The implementation of DPL only requires a small modification
+of the codes. Experimental studies on five public datasets validate the
+effectiveness of proposed learning method.",cs.IR,['cs.IR']
+Overcoming Generic Knowledge Loss with Selective Parameter Update,Wenxuan Zhang · Paul Janson · Rahaf Aljundi · Mohamed Elhoseiny, ,https://arxiv.org/abs/2308.12462,,2308.12462.pdf,Overcoming Generic Knowledge Loss with Selective Parameter Update,"Foundation models encompass an extensive knowledge base and offer remarkable
+transferability. However, this knowledge becomes outdated or insufficient over
+time. The challenge lies in continuously updating foundation models to
+accommodate novel information while retaining their original capabilities.
+Leveraging the fact that foundation models have initial knowledge on various
+tasks and domains, we propose a novel approach that, instead of updating all
+parameters equally, localizes the updates to a sparse set of parameters
+relevant to the task being learned. We strike a balance between efficiency and
+new task performance, while maintaining the transferability and
+generalizability of foundation models. We extensively evaluate our method on
+foundational vision-language models with a diverse spectrum of continual
+learning tasks. Our method achieves improvements on the accuracy of the newly
+learned tasks up to 7% while preserving the pretraining knowledge with a
+negligible decrease of 0.9% on a representative control set accuracy.",cs.CV,['cs.CV']
+Projecting Trackable Thermal Patterns for Dynamic Computer Vision,Mark Sheinin · Aswin C. Sankaranarayanan · Srinivasa G. Narasimhan, ,,https://www.globotreks.com/destinations/canada/day-trips-manitoba-winnipeg/,,,,,nan
+DePT: Decoupled Prompt Tuning,Ji Zhang · Shihan Wu · Lianli Gao · Heng Tao Shen · Jingkuan Song, ,https://arxiv.org/abs/2309.07439,,2309.07439.pdf,DePT: Decoupled Prompt Tuning,"This work breaks through the Base-New Tradeoff (BNT)dilemma in prompt tuning,
+i.e., the better the tuned model generalizes to the base (or target) task, the
+worse it generalizes to new tasks, and vice versa. Specifically, through an
+in-depth analysis of the learned features of the base and new tasks, we observe
+that the BNT stems from a channel bias issue, i.e., the vast majority of
+feature channels are occupied by base-specific knowledge, resulting in the
+collapse of taskshared knowledge important to new tasks. To address this, we
+propose the Decoupled Prompt Tuning (DePT) framework, which decouples
+base-specific knowledge from feature channels into an isolated feature space
+during prompt tuning, so as to maximally preserve task-shared knowledge in the
+original feature space for achieving better zero-shot generalization on new
+tasks. Importantly, our DePT is orthogonal to existing prompt tuning methods,
+hence it can improve all of them. Extensive experiments on 11 datasets show the
+strong flexibility and effectiveness of DePT. Our code and pretrained models
+are available at https://github.com/Koorye/DePT.",cs.CV,['cs.CV']
+Sharingan: A Transformer Architecture for Multi-Person Gaze Following,Samy Tafasca · Anshul Gupta · Jean-marc Odobez, ,https://arxiv.org/abs/2310.00816,,2310.00816.pdf,Sharingan: A Transformer-based Architecture for Gaze Following,"Gaze is a powerful form of non-verbal communication and social interaction
+that humans develop from an early age. As such, modeling this behavior is an
+important task that can benefit a broad set of application domains ranging from
+robotics to sociology. In particular, Gaze Following is defined as the
+prediction of the pixel-wise 2D location where a person in the image is
+looking. Prior efforts in this direction have focused primarily on CNN-based
+architectures to perform the task. In this paper, we introduce a novel
+transformer-based architecture for 2D gaze prediction. We experiment with 2
+variants: the first one retains the same task formulation of predicting a gaze
+heatmap for one person at a time, while the second one casts the problem as a
+2D point regression and allows us to perform multi-person gaze prediction with
+a single forward pass. This new architecture achieves state-of-the-art results
+on the GazeFollow and VideoAttentionTarget datasets. The code for this paper
+will be made publicly available.",cs.CV,['cs.CV']
+Fully Exploiting Every Real Sample: Super-Pixel Sample Gradient Model Stealing,Yunlong Zhao · Xiaoheng Deng · Yijing Liu · Xinjun Pei · Jiazhi Xia · Wei Chen, ,https://ar5iv.labs.arxiv.org/html/2309.10058,,2309.10058.pdf,Dual Student Networks for Data-Free Model Stealing,"Existing data-free model stealing methods use a generator to produce samples
+in order to train a student model to match the target model outputs. To this
+end, the two main challenges are estimating gradients of the target model
+without access to its parameters, and generating a diverse set of training
+samples that thoroughly explores the input space. We propose a Dual Student
+method where two students are symmetrically trained in order to provide the
+generator a criterion to generate samples that the two students disagree on. On
+one hand, disagreement on a sample implies at least one student has classified
+the sample incorrectly when compared to the target model. This incentive
+towards disagreement implicitly encourages the generator to explore more
+diverse regions of the input space. On the other hand, our method utilizes
+gradients of student models to indirectly estimate gradients of the target
+model. We show that this novel training objective for the generator network is
+equivalent to optimizing a lower bound on the generator's loss if we had access
+to the target model gradients. We show that our new optimization framework
+provides more accurate gradient estimation of the target model and better
+accuracies on benchmark classification datasets. Additionally, our approach
+balances improved query efficiency with training computation cost. Finally, we
+demonstrate that our method serves as a better proxy model for transfer-based
+adversarial attacks than existing data-free model stealing methods.",cs.LG,"['cs.LG', 'cs.CR']"
+MaskClustering:  View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation,Mi Yan · Jiazhao Zhang · Yan Zhu · He Wang,https://pku-epic.github.io/MaskClustering/,https://arxiv.org/abs/2401.07745,,2401.07745.pdf,MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation,"Open-vocabulary 3D instance segmentation is cutting-edge for its ability to
+segment 3D instances without predefined categories. However, progress in 3D
+lags behind its 2D counterpart due to limited annotated 3D data. To address
+this, recent works first generate 2D open-vocabulary masks through 2D models
+and then merge them into 3D instances based on metrics calculated between two
+neighboring frames. In contrast to these local metrics, we propose a novel
+metric, view consensus rate, to enhance the utilization of multi-view
+observations. The key insight is that two 2D masks should be deemed part of the
+same 3D instance if a significant number of other 2D masks from different views
+contain both these two masks. Using this metric as edge weight, we construct a
+global mask graph where each mask is a node. Through iterative clustering of
+masks showing high view consensus, we generate a series of clusters, each
+representing a distinct 3D instance. Notably, our model is training-free.
+Through extensive experiments on publicly available datasets, including
+ScanNet++, ScanNet200 and MatterPort3D, we demonstrate that our method achieves
+state-of-the-art performance in open-vocabulary 3D instance segmentation. Our
+project page is at https://pku-epic.github.io/MaskClustering.",cs.CV,['cs.CV']
+Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts,Jialin Wu · Xia Hu · Yaqing Wang · Bo Pang · Radu Soricut, ,https://arxiv.org/abs/2312.00968,,2312.00968.pdf,Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts,"Large multi-modal models (LMMs) exhibit remarkable performance across
+numerous tasks. However, generalist LMMs often suffer from performance
+degradation when tuned over a large collection of tasks. Recent research
+suggests that Mixture of Experts (MoE) architectures are useful for instruction
+tuning, but for LMMs of parameter size around O(50-100B), the prohibitive cost
+of replicating and storing the expert models severely limits the number of
+experts we can use. We propose Omni-SMoLA, an architecture that uses the Soft
+MoE approach to (softly) mix many multimodal low rank experts, and avoids
+introducing a significant number of new parameters compared to conventional MoE
+models. The core intuition here is that the large model provides a foundational
+backbone, while different lightweight experts residually learn specialized
+knowledge, either per-modality or multimodally. Extensive experiments
+demonstrate that the SMoLA approach helps improve the generalist performance
+across a broad range of generative vision-and-language tasks, achieving new
+SoTA generalist performance that often matches or outperforms single
+specialized LMM baselines, as well as new SoTA specialist performance.",cs.CV,"['cs.CV', 'cs.CL']"
+SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,Thuan Nguyen · Anh Tran,thuanz123.github.io/swiftbrush,https://arxiv.org/abs/2312.05239,,2312.05239.pdf,SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,"Despite their ability to generate high-resolution and diverse images from
+text prompts, text-to-image diffusion models often suffer from slow iterative
+sampling processes. Model distillation is one of the most effective directions
+to accelerate these models. However, previous distillation methods fail to
+retain the generation quality while requiring a significant amount of images
+for training, either from real data or synthetically generated by the teacher
+model. In response to this limitation, we present a novel image-free
+distillation scheme named $\textbf{SwiftBrush}$. Drawing inspiration from
+text-to-3D synthesis, in which a 3D neural radiance field that aligns with the
+input prompt can be obtained from a 2D text-to-image diffusion prior via a
+specialized loss without the use of any 3D data ground-truth, our approach
+re-purposes that same loss for distilling a pretrained multi-step text-to-image
+model to a student network that can generate high-fidelity images with just a
+single inference step. In spite of its simplicity, our model stands as one of
+the first one-step text-to-image generators that can produce images of
+comparable quality to Stable Diffusion without reliance on any training image
+data. Remarkably, SwiftBrush achieves an FID score of $\textbf{16.67}$ and a
+CLIP score of $\textbf{0.29}$ on the COCO-30K benchmark, achieving competitive
+results or even substantially surpassing existing state-of-the-art distillation
+techniques.",cs.CV,['cs.CV']
+HardMo: A Large-Scale Hardcase Dataset for Motion Capture,Jiaqi Liao · Chuanchen Luo · Yinuo Du · Yuxi Wang · Xu-Cheng Yin · Man Zhang · Zhaoxiang Zhang · Junran Peng, ,,,,,,,nan
+Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models,Huan Ling · Seung Wook Kim · Antonio Torralba · Sanja Fidler · Karsten Kreis, ,https://arxiv.org/abs/2312.13763,,2312.13763.pdf,Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models,"Text-guided diffusion models have revolutionized image and video generation
+and have also been successfully used for optimization-based 3D object
+synthesis. Here, we instead focus on the underexplored text-to-4D setting and
+synthesize dynamic, animated 3D objects using score distillation methods with
+an additional temporal dimension. Compared to previous work, we pursue a novel
+compositional generation-based approach, and combine text-to-image,
+text-to-video, and 3D-aware multiview diffusion models to provide feedback
+during 4D object optimization, thereby simultaneously enforcing temporal
+consistency, high-quality visual appearance and realistic geometry. Our method,
+called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
+deformation fields as 4D representation. Crucial to AYG is a novel method to
+regularize the distribution of the moving 3D Gaussians and thereby stabilize
+the optimization and induce motion. We also propose a motion amplification
+mechanism as well as a new autoregressive synthesis scheme to generate and
+combine multiple 4D sequences for longer generation. These techniques allow us
+to synthesize vivid dynamic scenes, outperform previous work qualitatively and
+quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
+Gaussian 4D representation, different 4D animations can be seamlessly combined,
+as we demonstrate. AYG opens up promising avenues for animation, simulation and
+digital content creation as well as synthetic data generation.",cs.CV,"['cs.CV', 'cs.LG']"
+THRONE: A Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models,Prannay Kaul · Zhizhong Li · Hao Yang · Yonatan Dukler · Ashwin Swaminathan · CJ Taylor · Stefano Soatto · Stefano Soatto, ,,,,,,,nan
+CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance Fields from Sparse Inputs,Yingji Zhong · Lanqing Hong · Zhenguo Li · Dan Xu, ,,,,,,,nan
+Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction,Xiaoyang Lyu · Chirui Chang · Peng Dai · Yangtian Sun · Xiaojuan Qi, ,https://arxiv.org/abs/2403.19314,,2403.19314.pdf,Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction,"Scene reconstruction from multi-view images is a fundamental problem in
+computer vision and graphics. Recent neural implicit surface reconstruction
+methods have achieved high-quality results; however, editing and manipulating
+the 3D geometry of reconstructed scenes remains challenging due to the absence
+of naturally decomposed object entities and complex object/background
+compositions. In this paper, we present Total-Decom, a novel method for
+decomposed 3D reconstruction with minimal human interaction. Our approach
+seamlessly integrates the Segment Anything Model (SAM) with hybrid
+implicit-explicit neural surface representations and a mesh-based
+region-growing technique for accurate 3D object decomposition. Total-Decom
+requires minimal human annotations while providing users with real-time control
+over the granularity and quality of decomposition. We extensively evaluate our
+method on benchmark datasets and demonstrate its potential for downstream
+applications, such as animation and scene editing. The code is available at
+https://github.com/CVMI-Lab/Total-Decom.git.",cs.CV,['cs.CV']
+Contrastive Pre-Training with Multi-View Fusion for No-Reference Point Cloud Quality Assessment,Ziyu Shan · Yujie Zhang · Qi Yang · Haichen Yang · Yiling Xu · Jenq-Neng Hwang · Xiaozhong Xu · Shan Liu, ,https://arxiv.org/abs/2403.10066,,2403.10066.pdf,Contrastive Pre-Training with Multi-View Fusion for No-Reference Point Cloud Quality Assessment,"No-reference point cloud quality assessment (NR-PCQA) aims to automatically
+evaluate the perceptual quality of distorted point clouds without available
+reference, which have achieved tremendous improvements due to the utilization
+of deep neural networks. However, learning-based NR-PCQA methods suffer from
+the scarcity of labeled data and usually perform suboptimally in terms of
+generalization. To solve the problem, we propose a novel contrastive
+pre-training framework tailored for PCQA (CoPA), which enables the pre-trained
+model to learn quality-aware representations from unlabeled data. To obtain
+anchors in the representation space, we project point clouds with different
+distortions into images and randomly mix their local patches to form mixed
+images with multiple distortions. Utilizing the generated anchors, we constrain
+the pre-training process via a quality-aware contrastive loss following the
+philosophy that perceptual quality is closely related to both content and
+distortion. Furthermore, in the model fine-tuning stage, we propose a
+semantic-guided multi-view fusion module to effectively integrate the features
+of projected images from multiple perspectives. Extensive experiments show that
+our method outperforms the state-of-the-art PCQA methods on popular benchmarks.
+Further investigations demonstrate that CoPA can also benefit existing
+learning-based PCQA models.",cs.CV,"['cs.CV', 'cs.MM']"
+PredToken: Predicting Unknown Tokens and Beyond with Coarse-to-Fine Iterative Decoding,Xuesong Nie · Haoyuan Jin · Yunfeng Yan · Xi Chen · Zhihang Zhu · Donglian Qi, ,http://export.arxiv.org/abs/2310.18698,,2310.18698.pdf,Triplet Attention Transformer for Spatiotemporal Predictive Learning,"Spatiotemporal predictive learning offers a self-supervised learning paradigm
+that enables models to learn both spatial and temporal patterns by predicting
+future sequences based on historical sequences. Mainstream methods are
+dominated by recurrent units, yet they are limited by their lack of
+parallelization and often underperform in real-world scenarios. To improve
+prediction quality while maintaining computational efficiency, we propose an
+innovative triplet attention transformer designed to capture both inter-frame
+dynamics and intra-frame static features. Specifically, the model incorporates
+the Triplet Attention Module (TAM), which replaces traditional recurrent units
+by exploring self-attention mechanisms in temporal, spatial, and channel
+dimensions. In this configuration: (i) temporal tokens contain abstract
+representations of inter-frame, facilitating the capture of inherent temporal
+dependencies; (ii) spatial and channel attention combine to refine the
+intra-frame representation by performing fine-grained interactions across
+spatial and channel dimensions. Alternating temporal, spatial, and
+channel-level attention allows our approach to learn more complex short- and
+long-range spatiotemporal dependencies. Extensive experiments demonstrate
+performance surpassing existing recurrent-based and recurrent-free methods,
+achieving state-of-the-art under multi-scenario examination including moving
+object trajectory prediction, traffic flow prediction, driving scene
+prediction, and human motion capture.",cs.CV,"['cs.CV', 'cs.LG']"
+U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation,You Wu · Kean Liu · Xiaoyue Mi · Fan Tang · Juan Cao · Jintao Li, ,https://arxiv.org/abs/2403.20231,,2403.20231.pdf,U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation,"Concept personalization methods enable large text-to-image models to learn
+specific subjects (e.g., objects/poses/3D models) and synthesize renditions in
+new contexts. Given that the image references are highly biased towards visual
+attributes, state-of-the-art personalization models tend to overfit the whole
+subject and cannot disentangle visual characteristics in pixel space. In this
+study, we proposed a more challenging setting, namely fine-grained visual
+appearance personalization. Different from existing methods, we allow users to
+provide a sentence describing the desired attributes. A novel decoupled
+self-augmentation strategy is proposed to generate target-related and
+non-target samples to learn user-specified visual attributes. These augmented
+data allow for refining the model's understanding of the target attribute while
+mitigating the impact of unrelated attributes. At the inference stage,
+adjustments are conducted on semantic space through the learned target and
+non-target embeddings to further enhance the disentanglement of target
+attributes. Extensive experiments on various kinds of visual attributes with
+SOTA personalization methods show the ability of the proposed method to mimic
+target visual appearance in novel contexts, thus improving the controllability
+and flexibility of personalization.",cs.CV,['cs.CV']
+OVMR: Open-Vocabulary Recognition with Multi-Modal References,Zehong Ma · Shiliang Zhang · Longhui Wei · Qi Tian, ,https://arxiv.org/abs/2306.05493,,2306.05493.pdf,Multi-Modal Classifiers for Open-Vocabulary Object Detection,"The goal of this paper is open-vocabulary object detection (OVOD)
+$\unicode{x2013}$ building a model that can detect objects beyond the set of
+categories seen at training, thus enabling the user to specify categories of
+interest at inference without the need for model retraining. We adopt a
+standard two-stage object detector architecture, and explore three ways for
+specifying novel categories: via language descriptions, via image exemplars, or
+via a combination of the two. We make three contributions: first, we prompt a
+large language model (LLM) to generate informative language descriptions for
+object classes, and construct powerful text-based classifiers; second, we
+employ a visual aggregator on image exemplars that can ingest any number of
+images as input, forming vision-based classifiers; and third, we provide a
+simple method to fuse information from language descriptions and image
+exemplars, yielding a multi-modal classifier. When evaluating on the
+challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our
+text-based classifiers outperform all previous OVOD works; (ii) our
+vision-based classifiers perform as well as text-based classifiers in prior
+work; (iii) using multi-modal classifiers perform better than either modality
+alone; and finally, (iv) our text-based and multi-modal classifiers yield
+better performance than a fully-supervised detector.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'I.4.6; I.4.8; I.4.9; I.2.10']"
+Dynamic Prompt Optimizing for Text-to-Image Generation,Wenyi Mo · Tianyu Zhang · Yalong Bai · Bing Su · Ji-Rong Wen · Qing Yang, ,https://arxiv.org/abs/2404.04095,,2404.04095.pdf,Dynamic Prompt Optimizing for Text-to-Image Generation,"Text-to-image generative models, specifically those based on diffusion models
+like Imagen and Stable Diffusion, have made substantial advancements. Recently,
+there has been a surge of interest in the delicate refinement of text prompts.
+Users assign weights or alter the injection time steps of certain words in the
+text prompts to improve the quality of generated images. However, the success
+of fine-control prompts depends on the accuracy of the text prompts and the
+careful selection of weights and time steps, which requires significant manual
+intervention. To address this, we introduce the \textbf{P}rompt
+\textbf{A}uto-\textbf{E}diting (PAE) method. Besides refining the original
+prompts for image generation, we further employ an online reinforcement
+learning strategy to explore the weights and injection time steps of each word,
+leading to the dynamic fine-control prompts. The reward function during
+training encourages the model to consider aesthetic score, semantic
+consistency, and user preferences. Experimental results demonstrate that our
+proposed method effectively improves the original prompts, generating visually
+more appealing images while maintaining semantic alignment. Code is available
+at https://github.com/Mowenyii/PAE.",cs.CV,"['cs.CV', 'cs.AI']"
+DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery,Yixuan Zhu · Ao Li · Yansong Tang · Wenliang Zhao · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2404.01424,,2404.01424.pdf,DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery,"The recovery of occluded human meshes presents challenges for current methods
+due to the difficulty in extracting effective image features under severe
+occlusion. In this paper, we introduce DPMesh, an innovative framework for
+occluded human mesh recovery that capitalizes on the profound diffusion prior
+about object structure and spatial relationships embedded in a pre-trained
+text-to-image diffusion model. Unlike previous methods reliant on conventional
+backbones for vanilla feature extraction, DPMesh seamlessly integrates the
+pre-trained denoising U-Net with potent knowledge as its image backbone and
+performs a single-step inference to provide occlusion-aware information. To
+enhance the perception capability for occluded poses, DPMesh incorporates
+well-designed guidance via condition injection, which produces effective
+controls from 2D observations for the denoising U-Net. Furthermore, we explore
+a dedicated noisy key-point reasoning approach to mitigate disturbances arising
+from occlusion and crowded scenarios. This strategy fully unleashes the
+perceptual capability of the diffusion prior, thereby enhancing accuracy.
+Extensive experiments affirm the efficacy of our framework, as we outperform
+state-of-the-art methods on both occlusion-specific and standard datasets. The
+persuasive results underscore its ability to achieve precise and robust 3D
+human mesh recovery, particularly in challenging scenarios involving occlusion
+and crowded scenes.",cs.CV,['cs.CV']
+Learning Inclusion Matching for Animation Paint Bucket Colorization,Yuekun Dai · Shangchen Zhou · Blake Li · Chongyi Li · Chen Change Loy,https://ykdai.github.io/projects/InclusionMatching,https://arxiv.org/abs/2403.18342,,2403.18342.pdf,Learning Inclusion Matching for Animation Paint Bucket Colorization,"Colorizing line art is a pivotal task in the production of hand-drawn cel
+animation. This typically involves digital painters using a paint bucket tool
+to manually color each segment enclosed by lines, based on RGB values
+predetermined by a color designer. This frame-by-frame process is both arduous
+and time-intensive. Current automated methods mainly focus on segment matching.
+This technique migrates colors from a reference to the target frame by aligning
+features within line-enclosed segments across frames. However, issues like
+occlusion and wrinkles in animations often disrupt these direct
+correspondences, leading to mismatches. In this work, we introduce a new
+learning-based inclusion matching pipeline, which directs the network to
+comprehend the inclusion relationships between segments rather than relying
+solely on direct visual correspondences. Our method features a two-stage
+pipeline that integrates a coarse color warping module with an inclusion
+matching module, enabling more nuanced and accurate colorization. To facilitate
+the training of our network, we also develope a unique dataset, referred to as
+PaintBucket-Character. This dataset includes rendered line arts alongside their
+colorized counterparts, featuring various 3D characters. Extensive experiments
+demonstrate the effectiveness and superiority of our method over existing
+techniques.",cs.CV,['cs.CV']
+Grounded Question-Answering in Long Egocentric Videos,Shangzhe Di · Weidi Xie,https://github.com/Becomebright/GroundVQA,https://arxiv.org/abs/2312.06505,,2312.06505.pdf,Grounded Question-Answering in Long Egocentric Videos,"Existing approaches to video understanding, mainly designed for short videos
+from a third-person perspective, are limited in their applicability in certain
+fields, such as robotics. In this paper, we delve into open-ended
+question-answering (QA) in long, egocentric videos, which allows individuals or
+robots to inquire about their own past visual experiences. This task presents
+unique challenges, including the complexity of temporally grounding queries
+within extensive video content, the high resource demands for precise data
+annotation, and the inherent difficulty of evaluating open-ended answers due to
+their ambiguous nature. Our proposed approach tackles these challenges by (i)
+integrating query grounding and answering within a unified model to reduce
+error propagation; (ii) employing large language models for efficient and
+scalable data synthesis; and (iii) introducing a close-ended QA task for
+evaluation, to manage answer ambiguity. Extensive experiments demonstrate the
+effectiveness of our method, which also achieves state-of-the-art performance
+on the QaEgo4D and Ego4D-NLQ benchmarks. Code, data, and models are available
+at https://github.com/Becomebright/GroundVQA.",cs.CV,['cs.CV']
+SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models,Feifei Wang · Zhentao Tan · Tianyi Wei · Yue Wu · Qidong Huang, ,https://arxiv.org/abs/2312.07865,,2312.07865.pdf,SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models,"Despite the success of diffusion-based customization methods on visual
+content creation, increasing concerns have been raised about such techniques
+from both privacy and political perspectives. To tackle this issue, several
+anti-customization methods have been proposed in very recent months,
+predominantly grounded in adversarial attacks. Unfortunately, most of these
+methods adopt straightforward designs, such as end-to-end optimization with a
+focus on adversarially maximizing the original training loss, thereby
+neglecting nuanced internal properties intrinsic to the diffusion model, and
+even leading to ineffective optimization in some diffusion time steps.In this
+paper, we strive to bridge this gap by undertaking a comprehensive exploration
+of these inherent properties, to boost the performance of current
+anti-customization approaches. Two aspects of properties are investigated: 1)
+We examine the relationship between time step selection and the model's
+perception in the frequency domain of images and find that lower time steps can
+give much more contributions to adversarial noises. This inspires us to propose
+an adaptive greedy search for optimal time steps that seamlessly integrates
+with existing anti-customization methods. 2) We scrutinize the roles of
+features at different layers during denoising and devise a sophisticated
+feature-based optimization framework for anti-customization.Experiments on
+facial benchmarks demonstrate that our approach significantly increases
+identity disruption, thereby protecting user privacy and copyright. Our code is
+available at: https://github.com/somuchtome/SimAC.",cs.CV,['cs.CV']
+DYSON: Dynamic Feature Space Self-Organization for Online Task-Free Class Incremental Learning,Yuhang He · YingJie Chen · Yuhan Jin · Songlin Dong · Xing Wei · Yihong Gong, ,https://arxiv.org/abs/2405.08533,,2405.08533.pdf,Dynamic Feature Learning and Matching for Class-Incremental Learning,"Class-incremental learning (CIL) has emerged as a means to learn new classes
+incrementally without catastrophic forgetting of previous classes. Recently,
+CIL has undergone a paradigm shift towards dynamic architectures due to their
+superior performance. However, these models are still limited by the following
+aspects: (i) Data augmentation (DA), which are tightly coupled with CIL,
+remains under-explored in dynamic architecture scenarios. (ii) Feature
+representation. The discriminativeness of dynamic feature are sub-optimal and
+possess potential for refinement. (iii) Classifier. The misalignment between
+dynamic feature and classifier constrains the capabilities of the model. To
+tackle the aforementioned drawbacks, we propose the Dynamic Feature Learning
+and Matching (DFLM) model in this paper from above three perspectives.
+Specifically, we firstly introduce class weight information and non-stationary
+functions to extend the mix DA method for dynamically adjusting the focus on
+memory during training. Then, von Mises-Fisher (vMF) classifier is employed to
+effectively model the dynamic feature distribution and implicitly learn their
+discriminative properties. Finally, the matching loss is proposed to facilitate
+the alignment between the learned dynamic features and the classifier by
+minimizing the distribution distance. Extensive experiments on CIL benchmarks
+validate that our proposed model achieves significant performance improvements
+over existing methods.",cs.CV,['cs.CV']
+NightCC: Nighttime Color Constancy via  Adaptive Channel Masking,Shuwei Li · Robby T. Tan, ,,,,,,,nan
+G$^3$-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding,Yuan Wang · Yali Li · Shengjin Wang, ,https://arxiv.org/abs/2403.08182,,2403.08182.pdf,SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention,"3D visual grounding aims to automatically locate the 3D region of the
+specified object given the corresponding textual description. Existing works
+fail to distinguish similar objects especially when multiple referred objects
+are involved in the description. Experiments show that direct matching of
+language and visual modal has limited capacity to comprehend complex
+referential relationships in utterances. It is mainly due to the interference
+caused by redundant visual information in cross-modal alignment. To strengthen
+relation-orientated mapping between different modalities, we propose SeCG, a
+semantic-enhanced relational learning model based on a graph network with our
+designed memory graph attention layer. Our method replaces original
+language-independent encoding with cross-modal encoding in visual analysis.
+More text-related feature expressions are obtained through the guidance of
+global semantics and implicit relationships. Experimental results on ReferIt3D
+and ScanRefer benchmarks show that the proposed method outperforms the existing
+state-of-the-art methods, particularly improving the localization performance
+for the multi-relation challenges.",cs.CV,['cs.CV']
+Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation,Yunhe Gao, ,https://arxiv.org/abs/2306.02416,,2306.02416.pdf,Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation,"A major focus of clinical imaging workflow is disease diagnosis and
+management, leading to medical imaging datasets strongly tied to specific
+clinical objectives. This scenario has led to the prevailing practice of
+developing task-specific segmentation models, without gaining insights from
+widespread imaging cohorts. Inspired by the training program of medical
+radiology residents, we propose a shift towards universal medical image
+segmentation, a paradigm aiming to build medical image understanding foundation
+models by leveraging the diversity and commonality across clinical targets,
+body regions, and imaging modalities. Towards this goal, we develop Hermes, a
+novel context-prior learning approach to address the challenges of data
+heterogeneity and annotation differences in medical image segmentation. In a
+large collection of eleven diverse datasets (2,438 3D images) across five
+modalities (CT, PET, T1, T2 and cine MRI) and multiple body regions, we
+demonstrate the merit of the universal paradigm over the traditional paradigm
+on addressing multiple tasks within a single model. By exploiting the synergy
+across tasks, Hermes achieves state-of-the-art performance on all testing
+datasets and shows superior model scalability. Results on two additional
+datasets reveals Hermes' strong performance for transfer learning, incremental
+learning, and generalization to downstream tasks. Hermes's learned priors
+demonstrate an appealing trait to reflect the intricate relations among tasks
+and modalities, which aligns with the established anatomical and imaging
+principles in radiology. The code is available:
+https://github.com/yhygao/universal-medical-image-segmentation.",cs.CV,['cs.CV']
+Generative Quanta Color Imaging,Vishal Purohit · Junjie Luo · Yiheng Chi · Qi Guo · Stanley H. Chan · Qiang Qiu, ,https://arxiv.org/abs/2403.19066,,2403.19066.pdf,Generative Quanta Color Imaging,"The astonishing development of single-photon cameras has created an
+unprecedented opportunity for scientific and industrial imaging. However, the
+high data throughput generated by these 1-bit sensors creates a significant
+bottleneck for low-power applications. In this paper, we explore the
+possibility of generating a color image from a single binary frame of a
+single-photon camera. We evidently find this problem being particularly
+difficult to standard colorization approaches due to the substantial degree of
+exposure variation. The core innovation of our paper is an exposure synthesis
+model framed under a neural ordinary differential equation (Neural ODE) that
+allows us to generate a continuum of exposures from a single observation. This
+innovation ensures consistent exposure in binary images that colorizers take
+on, resulting in notably enhanced colorization. We demonstrate applications of
+the method in single-image and burst colorization and show superior generative
+performance over baselines. Project website can be found at
+https://vishal-s-p.github.io/projects/2023/generative_quanta_color.html.",cs.CV,"['cs.CV', 'cs.AI']"
+Polarization Wavefront Lidar: Learning Large Scene Reconstruction from Polarized Wavefronts,Dominik Scheuble · Chenyang Lei · Mario Bijelic · Seung-Hwan Baek · Felix Heide, ,,https://cg.postech.ac.kr/2024/03/01/9-papers-are-accepted-to-cvpr-2024/,,,,,nan
+MirageRoom: 3D Scene Segmentation with 2D Pre-trained Models by Mirage Projection,Haowen Sun · Yueqi Duan · Juncheng Yan · Yifan Liu · Jiwen Lu, ,https://arxiv.org/abs/2403.06403,,2403.06403.pdf,PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models,"Recent success of vision foundation models have shown promising performance
+for the 2D perception tasks. However, it is difficult to train a 3D foundation
+network directly due to the limited dataset and it remains under explored
+whether existing foundation models can be lifted to 3D space seamlessly. In
+this paper, we present PointSeg, a novel training-free paradigm that leverages
+off-the-shelf vision foundation models to address 3D scene perception tasks.
+PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to
+align their corresponding pixels across frames. Concretely, we design a
+two-branch prompts learning structure to construct the 3D point-box prompts
+pairs, combining with the bidirectional matching strategy for accurate point
+and proposal prompts generation. Then, we perform the iterative post-refinement
+adaptively when cooperated with different vision foundation models. Moreover,
+we design a affinity-aware merging algorithm to improve the final ensemble
+masks. PointSeg demonstrates impressive segmentation performance across various
+datasets, all without training. Specifically, our approach significantly
+surpasses the state-of-the-art specialist model by 13.4$\%$, 11.3$\%$, and
+12$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top
+of that, PointSeg can incorporate with various segmentation models and even
+surpasses the supervised methods.",cs.CV,['cs.CV']
+Overcoming Data Limitations for High-Quality Video Diffusion Models,Haoxin Chen · Yong Zhang · Xiaodong Cun · Menghan Xia · Xintao Wang · CHAO WENG · Ying Shan, ,,,,,,,nan
+FakeInversion: Learning to Detect Images from Unseen Text-to-Image Models by Inverting Stable Diffusion,George Cazenavette · Avneesh Sud · Thomas Leung · Ben Usman, ,https://ar5iv.labs.arxiv.org/html/2210.06998,,2210.06998.pdf,DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models,"Text-to-image generation models that generate images based on prompt
+descriptions have attracted an increasing amount of attention during the past
+few months. Despite their encouraging performance, these models raise concerns
+about the misuse of their generated fake images. To tackle this problem, we
+pioneer a systematic study on the detection and attribution of fake images
+generated by text-to-image generation models. Concretely, we first build a
+machine learning classifier to detect the fake images generated by various
+text-to-image generation models. We then attribute these fake images to their
+source models, such that model owners can be held responsible for their models'
+misuse. We further investigate how prompts that generate fake images affect
+detection and attribution. We conduct extensive experiments on four popular
+text-to-image generation models, including DALL$\cdot$E 2, Stable Diffusion,
+GLIDE, and Latent Diffusion, and two benchmark prompt-image datasets. Empirical
+results show that (1) fake images generated by various models can be
+distinguished from real ones, as there exists a common artifact shared by fake
+images from different models; (2) fake images can be effectively attributed to
+their source models, as different models leave unique fingerprints in their
+generated images; (3) prompts with the ``person'' topic or a length between 25
+and 75 enable models to generate fake images with higher authenticity. All
+findings contribute to the community's insight into the threats caused by
+text-to-image generation models. We appeal to the community's consideration of
+the counterpart solutions, like ours, against the rapidly-evolving fake image
+generation.",cs.CR,"['cs.CR', 'cs.CV', 'cs.LG']"
+"Separating the ""Chirp"" from the ""Chat"": Self-supervised Visual Grounding of Sound and Language",Mark Hamilton · Andrew Zisserman · John Hershey · William Freeman, ,https://arxiv.org/abs/2404.19696,,,Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners,"3D visual grounding is a challenging task that often requires direct and
+dense supervision, notably the semantic label for each object in the scene. In
+this paper, we instead study the naturally supervised setting that learns from
+only 3D scene and QA pairs, where prior works underperform. We propose the
+Language-Regularized Concept Learner (LARC), which uses constraints from
+language as regularization to significantly improve the accuracy of
+neuro-symbolic concept learners in the naturally supervised setting. Our
+approach is based on two core insights: the first is that language constraints
+(e.g., a word's relation to another) can serve as effective regularization for
+structured representations in neuro-symbolic models; the second is that we can
+query large language models to distill such constraints from language
+properties. We show that LARC improves performance of prior works in naturally
+supervised 3D visual grounding, and demonstrates a wide range of 3D visual
+reasoning capabilities-from zero-shot composition, to data efficiency and
+transferability. Our method represents a promising step towards regularizing
+structured visual reasoning frameworks with language-based priors, for learning
+in settings without dense supervision.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models,Weiwei Cao · Jianpeng Zhang · Yingda Xia · Tony C. W. MOK · Zi Li · Xianghua Ye · Le Lu · Jian Zheng · Yuxing Tang · Ling Zhang, ,https://arxiv.org/abs/2404.04936,,2404.04936.pdf,Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models,"Radiologists highly desire fully automated versatile AI for medical imaging
+interpretation. However, the lack of extensively annotated large-scale
+multi-disease datasets has hindered the achievement of this goal. In this
+paper, we explore the feasibility of leveraging language as a naturally
+high-quality supervision for chest CT imaging. In light of the limited
+availability of image-report pairs, we bootstrap the understanding of 3D chest
+CT images by distilling chest-related diagnostic knowledge from an extensively
+pre-trained 2D X-ray expert model. Specifically, we propose a language-guided
+retrieval method to match each 3D CT image with its semantically closest 2D
+X-ray image, and perform pair-wise and semantic relation knowledge
+distillation. Subsequently, we use contrastive learning to align images and
+reports within the same patient while distinguishing them from the other
+patients. However, the challenge arises when patients have similar semantic
+diagnoses, such as healthy patients, potentially confusing if treated as
+negatives. We introduce a robust contrastive learning that identifies and
+corrects these false negatives. We train our model with over 12,000 pairs of
+chest CT images and radiology reports. Extensive experiments across multiple
+scenarios, including zero-shot learning, report generation, and fine-tuning
+processes, demonstrate the model's feasibility in interpreting chest CT images.",cs.CV,['cs.CV']
+Towards Automated Movie Trailer Generation,Dawit Argaw Argaw · Mattia Soldan · Alejandro Pardo · Chen Zhao · Fabian Caba Heilbron · Joon Chung · Bernard Ghanem, ,https://arxiv.org/abs/2404.03477,,2404.03477.pdf,Towards Automated Movie Trailer Generation,"Movie trailers are an essential tool for promoting films and attracting
+audiences. However, the process of creating trailers can be time-consuming and
+expensive. To streamline this process, we propose an automatic trailer
+generation framework that generates plausible trailers from a full movie by
+automating shot selection and composition. Our approach draws inspiration from
+machine translation techniques and models the movies and trailers as sequences
+of shots, thus formulating the trailer generation problem as a
+sequence-to-sequence task. We introduce Trailer Generation Transformer (TGT), a
+deep-learning framework utilizing an encoder-decoder architecture. TGT movie
+encoder is tasked with contextualizing each movie shot representation via
+self-attention, while the autoregressive trailer decoder predicts the feature
+representation of the next trailer shot, accounting for the relevance of shots'
+temporal order in trailers. Our TGT significantly outperforms previous methods
+on a comprehensive suite of metrics.",cs.CV,['cs.CV']
+COCONut: Modernizing COCO Segmentation,Xueqing Deng · Qihang Yu · Peng Wang · Xiaohui Shen · Liang-Chieh Chen, ,,,,,,,nan
+Investigating Compositional Challenges in Vision-Language Models for Visual Grounding,Yunan Zeng · Yan Huang · Jinjin Zhang · Zequn Jie · Zhenhua Chai · Liang Wang, ,https://arxiv.org/html/2405.17104v1,,2405.17104v1.pdf,LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding,"Visual grounding is an essential tool that links user-provided text queries
+with query-specific regions within an image. Despite advancements in visual
+grounding models, their ability to comprehend complex queries remains limited.
+To overcome this limitation, we introduce LLM-Optic, an innovative method that
+utilizes Large Language Models (LLMs) as an optical lens to enhance existing
+visual grounding models in comprehending complex text queries involving
+intricate text structures, multiple objects, or object spatial relationships,
+situations that current models struggle with. LLM-Optic first employs an LLM as
+a Text Grounder to interpret complex text queries and accurately identify
+objects the user intends to locate. Then a pre-trained visual grounding model
+is used to generate candidate bounding boxes given the refined query by the
+Text Grounder. After that, LLM-Optic annotates the candidate bounding boxes
+with numerical marks to establish a connection between text and specific image
+regions, thereby linking two distinct modalities. Finally, it employs a Large
+Multimodal Model (LMM) as a Visual Grounder to select the marked candidate
+objects that best correspond to the original text query. Through LLM-Optic, we
+have achieved universal visual grounding, which allows for the detection of
+arbitrary objects specified by arbitrary human language input. Importantly, our
+method achieves this enhancement without requiring additional training or
+fine-tuning. Extensive experiments across various challenging benchmarks
+demonstrate that LLM-Optic achieves state-of-the-art zero-shot visual grounding
+capabilities.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+View-decoupled Transformer for Person Re-identification under Aerial-ground Camera Network,Quan Zhang · Lei Wang · Vishal M. Patel · Xiaohua Xie · Jianhuang Lai, ,https://arxiv.org/abs/2403.14513,,2403.14513.pdf,View-decoupled Transformer for Person Re-identification under Aerial-ground Camera Network,"Existing person re-identification methods have achieved remarkable advances
+in appearance-based identity association across homogeneous cameras, such as
+ground-ground matching. However, as a more practical scenario, aerial-ground
+person re-identification (AGPReID) among heterogeneous cameras has received
+minimal attention. To alleviate the disruption of discriminative identity
+representation by dramatic view discrepancy as the most significant challenge
+in AGPReID, the view-decoupled transformer (VDT) is proposed as a simple yet
+effective framework. Two major components are designed in VDT to decouple
+view-related and view-unrelated features, namely hierarchical subtractive
+separation and orthogonal loss, where the former separates these two features
+inside the VDT, and the latter constrains these two to be independent. In
+addition, we contribute a large-scale AGPReID dataset called CARGO, consisting
+of five/eight aerial/ground cameras, 5,000 identities, and 108,563 images.
+Experiments on two datasets show that VDT is a feasible and effective solution
+for AGPReID, surpassing the previous method on mAP/Rank1 by up to 5.0%/2.7% on
+CARGO and 3.7%/5.2% on AG-ReID, keeping the same magnitude of computational
+complexity. Our project is available at https://github.com/LinlyAC/VDT-AGPReID",cs.CV,['cs.CV']
+Towards Accurate Post-training Quantization for Diffusion Models,Changyuan Wang · Ziwei Wang · Xiuwei Xu · Yansong Tang · Jie Zhou · Jiwen Lu, ,https://arxiv.org/abs/2404.05662,,2404.05662.pdf,Towards Accurate Binarization of Diffusion Model,"With the advancement of diffusion models (DMs) and the substantially
+increased computational requirements, quantization emerges as a practical
+solution to obtain compact and efficient low-bit DMs. However, the highly
+discrete representation leads to severe accuracy degradation, hindering the
+quantization of diffusion models to ultra-low bit-widths. This paper proposes a
+novel quantization-aware training approach for DMs, namely BinaryDM. The
+proposed method pushes DMs' weights toward accurate and efficient binarization,
+considering the representation and computation properties. From the
+representation perspective, we present a Learnable Multi-basis Binarizer (LMB)
+to recover the representations generated by the binarized DM. The LMB enhances
+detailed information through the flexible combination of dual binary bases
+while applying to parameter-sparse locations of DM architectures to achieve
+minor burdens. From the optimization perspective, a Low-rank Representation
+Mimicking (LRM) is applied to assist the optimization of binarized DMs. The LRM
+mimics the representations of full-precision DMs in low-rank space, alleviating
+the direction ambiguity of the optimization process caused by fine-grained
+alignment. Moreover, a quick progressive warm-up is applied to BinaryDM,
+avoiding convergence difficulties by layerwisely progressive quantization at
+the beginning of training. Comprehensive experiments demonstrate that BinaryDM
+achieves significant accuracy and efficiency gains compared to SOTA
+quantization methods of DMs under ultra-low bit-widths. With 1.1-bit weight and
+4-bit activation (W1.1A4), BinaryDM achieves as low as 7.11 FID and saves the
+performance from collapse (baseline FID 39.69). As the first binarization
+method for diffusion models, W1.1A4 BinaryDM achieves impressive 9.3 times OPs
+and 24.8 times model size savings, showcasing its substantial potential for
+edge deployment.",cs.CV,['cs.CV']
+Density-Adaptive Model Based on Motif Matrix for Multi-Agent Trajectory Prediction,Di Wen · Haoran Xu · Zhaocheng He · Zhe Wu · Guang Tan · Peixi Peng, ,,https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/itr2.12502,,,,,nan
+Improving Transferable Targeted Adversarial Attacks with Model Self-Enhancement,Han Wu · Guanyan Ou · Weibin Wu · Zibin Zheng, ,https://arxiv.org/abs/2312.04913,,2312.04913.pdf,SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation,"Current Visual-Language Pre-training (VLP) models are vulnerable to
+adversarial examples. These adversarial examples present substantial security
+risks to VLP models, as they can leverage inherent weaknesses in the models,
+resulting in incorrect predictions. In contrast to white-box adversarial
+attacks, transfer attacks (where the adversary crafts adversarial examples on a
+white-box model to fool another black-box model) are more reflective of
+real-world scenarios, thus making them more meaningful for research. By
+summarizing and analyzing existing research, we identified two factors that can
+influence the efficacy of transfer attacks on VLP models: inter-modal
+interaction and data diversity. Based on these insights, we propose a
+self-augment-based transfer attack method, termed SA-Attack. Specifically,
+during the generation of adversarial images and adversarial texts, we apply
+different data augmentation methods to the image modality and text modality,
+respectively, with the aim of improving the adversarial transferability of the
+generated adversarial images and texts. Experiments conducted on the FLickr30K
+and COCO datasets have validated the effectiveness of our method. Our code will
+be available after this paper is accepted.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CR', 'cs.LG']"
+Disentangled Prompt Representation for Domain Generalization,De Cheng · Zhipeng Xu · XINYANG JIANG · Nannan Wang · Dongsheng Li · Xinbo Gao, ,https://arxiv.org/abs/2403.08506,,,DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning,"Federated learning (FL) has emerged as a powerful paradigm for learning from
+decentralized data, and federated domain generalization further considers the
+test dataset (target domain) is absent from the decentralized training data
+(source domains). However, most existing FL methods assume that domain labels
+are provided during training, and their evaluation imposes explicit constraints
+on the number of domains, which must strictly match the number of clients.
+Because of the underutilization of numerous edge devices and additional
+cross-client domain annotations in the real world, such restrictions may be
+impractical and involve potential privacy leaks. In this paper, we propose an
+efficient and novel approach, called Disentangled Prompt Tuning (DiPrompT), a
+method that tackles the above restrictions by learning adaptive prompts for
+domain generalization in a distributed manner. Specifically, we first design
+two types of prompts, i.e., global prompt to capture general knowledge across
+all clients and domain prompts to capture domain-specific knowledge. They
+eliminate the restriction on the one-to-one mapping between source domains and
+local clients. Furthermore, a dynamic query metric is introduced to
+automatically search the suitable domain label for each sample, which includes
+two-substep text-image alignments based on prompt tuning without
+labor-intensive annotation. Extensive experiments on multiple datasets
+demonstrate that our DiPrompT achieves superior domain generalization
+performance over state-of-the-art FL methods when domain labels are not
+provided, and even outperforms many centralized learning methods using domain
+labels.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner,Mengfei Xia · Yujun Shen · Changsong Lei · Yu Zhou · Deli Zhao · Ran Yi · Wenping Wang · Yong-Jin Liu, ,https://arxiv.org/abs/2310.09469,,2310.09469.pdf,Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner,"A diffusion model, which is formulated to produce an image using thousands of
+denoising steps, usually suffers from a slow inference speed. Existing
+acceleration algorithms simplify the sampling by skipping most steps yet
+exhibit considerable performance degradation. By viewing the generation of
+diffusion models as a discretized integrating process, we argue that the
+quality drop is partly caused by applying an inaccurate integral direction to a
+timestep interval. To rectify this issue, we propose a timestep aligner that
+helps find a more accurate integral direction for a particular interval at the
+minimum cost. Specifically, at each denoising step, we replace the original
+parameterization by conditioning the network on a new timestep, which is
+obtained by aligning the sampling distribution to the real distribution.
+Extensive experiments show that our plug-in design can be trained efficiently
+and boost the inference performance of various state-of-the-art acceleration
+methods, especially when there are few denoising steps. For example, when using
+10 denoising steps on the popular LSUN Bedroom dataset, we improve the FID of
+DDIM from 9.65 to 6.07, simply by adopting our method for a more appropriate
+set of timesteps. Code will be made publicly available.",cs.CV,['cs.CV']
+AutoAD III: The Prequel -- Back to the Pixels,Tengda Han · Max Bain · Arsha Nagrani · Gül Varol · Weidi Xie · Andrew Zisserman, ,https://arxiv.org/abs/2404.14412v1,,2404.14412v1.pdf,AutoAD III: The Prequel -- Back to the Pixels,"Generating Audio Description (AD) for movies is a challenging task that
+requires fine-grained visual understanding and an awareness of the characters
+and their names. Currently, visual language models for AD generation are
+limited by a lack of suitable training data, and also their evaluation is
+hampered by using performance measures not specialized to the AD domain. In
+this paper, we make three contributions: (i) We propose two approaches for
+constructing AD datasets with aligned video data, and build training and
+evaluation datasets using these. These datasets will be publicly released; (ii)
+We develop a Q-former-based architecture which ingests raw video and generates
+AD, using frozen pre-trained visual encoders and large language models; and
+(iii) We provide new evaluation metrics to benchmark AD quality that are
+well-matched to human performance. Taken together, we improve the state of the
+art on AD generation.",cs.CV,['cs.CV']
+Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting,Taeho Kang · Youngki Lee,https://tho-kn.github.io/projects/EgoTAP/,https://arxiv.org/abs/2402.18330,,2402.18330.pdf,Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting,"We present EgoTAP, a heatmap-to-3D pose lifting method for highly accurate
+stereo egocentric 3D pose estimation. Severe self-occlusion and out-of-view
+limbs in egocentric camera views make accurate pose estimation a challenging
+problem. To address the challenge, prior methods employ joint
+heatmaps-probabilistic 2D representations of the body pose, but heatmap-to-3D
+pose conversion still remains an inaccurate process. We propose a novel
+heatmap-to-3D lifting method composed of the Grid ViT Encoder and the
+Propagation Network. The Grid ViT Encoder summarizes joint heatmaps into
+effective feature embedding using self-attention. Then, the Propagation Network
+estimates the 3D pose by utilizing skeletal information to better estimate the
+position of obscure joints. Our method significantly outperforms the previous
+state-of-the-art qualitatively and quantitatively demonstrated by a 23.9\%
+reduction of error in an MPJPE metric. Our source code is available in GitHub.",cs.CV,['cs.CV']
+Data Valuation and Detections in Federated Learning,Wenqian Li · Shuran Fu · Fengrui Zhang · Yan Pang,https://github.com/muz1lee/MOTdata/tree/main,https://arxiv.org/abs/2311.05304v2,,2311.05304v2.pdf,Data Valuation and Detections in Federated Learning,"Federated Learning (FL) enables collaborative model training while preserving
+the privacy of raw data. A challenge in this framework is the fair and
+efficient valuation of data, which is crucial for incentivizing clients to
+contribute high-quality data in the FL task. In scenarios involving numerous
+data clients within FL, it is often the case that only a subset of clients and
+datasets are pertinent to a specific learning task, while others might have
+either a negative or negligible impact on the model training process. This
+paper introduces a novel privacy-preserving method for evaluating client
+contributions and selecting relevant datasets without a pre-specified training
+algorithm in an FL task. Our proposed approach FedBary, utilizes Wasserstein
+distance within the federated context, offering a new solution for data
+valuation in the FL framework. This method ensures transparent data valuation
+and efficient computation of the Wasserstein barycenter and reduces the
+dependence on validation datasets. Through extensive empirical experiments and
+theoretical analyses, we demonstrate the potential of this data valuation
+method as a promising avenue for FL research.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CR']"
+WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion,Khiem Vuong · N. Dinesh Reddy · Robert Tamburo · Srinivasa G. Narasimhan, ,https://arxiv.org/abs/2403.19022,,2403.19022.pdf,WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion,"Current methods for 2D and 3D object understanding struggle with severe
+occlusions in busy urban environments, partly due to the lack of large-scale
+labeled ground-truth annotations for learning occlusion. In this work, we
+introduce a novel framework for automatically generating a large, realistic
+dataset of dynamic objects under occlusions using freely available time-lapse
+imagery. By leveraging off-the-shelf 2D (bounding box, segmentation, keypoint)
+and 3D (pose, shape) predictions as pseudo-groundtruth, unoccluded 3D objects
+are identified automatically and composited into the background in a clip-art
+style, ensuring realistic appearances and physically accurate occlusion
+configurations. The resulting clip-art image with pseudo-groundtruth enables
+efficient training of object reconstruction methods that are robust to
+occlusions. Our method demonstrates significant improvements in both 2D and 3D
+reconstruction, particularly in scenarios with heavily occluded objects like
+vehicles and people in urban scenes.",cs.CV,['cs.CV']
+Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models,Yushi Hu · Otilia Stretcu · Chun-Ta Lu · Krishnamurthy Viswanathan · Kenji Hata · Enming Luo · Ranjay Krishna · Ariel Fuxman, ,https://arxiv.org/abs/2312.03052,,2312.03052.pdf,Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models,"Solving complex visual tasks such as ""Who invented the musical instrument on
+the right?"" involves a composition of skills: understanding space, recognizing
+instruments, and also retrieving prior knowledge. Recent work shows promise by
+decomposing such tasks using a large language model (LLM) into an executable
+program that invokes specialized vision models. However, generated programs are
+error-prone: they omit necessary steps, include spurious ones, and are unable
+to recover when the specialized models give incorrect outputs. Moreover, they
+require loading multiple models, incurring high latency and computation costs.
+We propose Visual Program Distillation (VPD), an instruction tuning framework
+that produces a vision-language model (VLM) capable of solving complex visual
+tasks with a single forward pass. VPD distills the reasoning ability of LLMs by
+using them to sample multiple candidate programs, which are then executed and
+verified to identify a correct one. It translates each correct program into a
+language description of the reasoning steps, which are then distilled into a
+VLM. Extensive experiments show that VPD improves the VLM's ability to count,
+understand spatial relations, and reason compositionally. Our VPD-trained
+PaLI-X outperforms all prior VLMs, achieving state-of-the-art performance
+across complex vision tasks, including MMBench, OK-VQA, A-OKVQA, TallyQA, POPE,
+and Hateful Memes. An evaluation with human annotators also confirms that VPD
+improves model response factuality and consistency. Finally, experiments on
+content moderation demonstrate that VPD is also helpful for adaptation to
+real-world applications with limited data.",cs.CV,"['cs.CV', 'cs.CL']"
+Learning Multi-dimensional Human Preference for Text-to-Image Generation,Sixian Zhang · Bohan Wang · Junqiang Wu · Yan Li · Tingting Gao · Di ZHANG · Zhongyuan Wang,https://wangbohan97.github.io/MPS/,,,,,,,nan
+IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation,Mengshun Hu · Kui Jiang · Zhihang Zhong · Zheng Wang · Yinqiang Zheng, ,https://arxiv.org/abs/2404.13534,,2404.13534.pdf,Motion-aware Latent Diffusion Models for Video Frame Interpolation,"With the advancement of AIGC, video frame interpolation (VFI) has become a
+crucial component in existing video generation frameworks, attracting
+widespread research interest. For the VFI task, the motion estimation between
+neighboring frames plays a crucial role in avoiding motion ambiguity. However,
+existing VFI methods always struggle to accurately predict the motion
+information between consecutive frames, and this imprecise estimation leads to
+blurred and visually incoherent interpolated frames. In this paper, we propose
+a novel diffusion framework, motion-aware latent diffusion models (MADiff),
+which is specifically designed for the VFI task. By incorporating motion priors
+between the conditional neighboring frames with the target interpolated frame
+predicted throughout the diffusion sampling procedure, MADiff progressively
+refines the intermediate outcomes, culminating in generating both visually
+smooth and realistic results. Extensive experiments conducted on benchmark
+datasets demonstrate that our method achieves state-of-the-art performance
+significantly outperforming existing approaches, especially under challenging
+scenarios involving dynamic textures with complex motion.",cs.CV,['cs.CV']
+MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation,Hanzhe Hu · Zhizhuo Zhou · Varun Jampani · Shubham Tulsiani, ,https://arxiv.org/abs/2404.03656,,2404.03656.pdf,MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation,"We present MVD-Fusion: a method for single-view 3D inference via generative
+modeling of multi-view-consistent RGB-D images. While recent methods pursuing
+3D inference advocate learning novel-view generative models, these generations
+are not 3D-consistent and require a distillation process to generate a 3D
+output. We instead cast the task of 3D inference as directly generating
+mutually-consistent multiple views and build on the insight that additionally
+inferring depth can provide a mechanism for enforcing this consistency.
+Specifically, we train a denoising diffusion model to generate multi-view RGB-D
+images given a single RGB input image and leverage the (intermediate noisy)
+depth estimates to obtain reprojection-based conditioning to maintain
+multi-view consistency. We train our model using large-scale synthetic dataset
+Obajverse as well as the real-world CO3D dataset comprising of generic camera
+viewpoints. We demonstrate that our approach can yield more accurate synthesis
+compared to recent state-of-the-art, including distillation-based 3D inference
+and prior multi-view generation methods. We also evaluate the geometry induced
+by our multi-view depth prediction and find that it yields a more accurate
+representation than other direct 3D inference approaches.",cs.CV,['cs.CV']
+Video ReCap: Recursive Captioning of Hour-Long Videos,Md Mohaiminul Islam · Vu Bao Ngan Ho · Xitong Yang · Tushar Nagarajan · Lorenzo Torresani · Gedas Bertasius, ,https://arxiv.org/abs/2402.13250,,2402.13250.pdf,Video ReCap: Recursive Captioning of Hour-Long Videos,"Most video captioning models are designed to process short video clips of few
+seconds and output text describing low-level visual concepts (e.g., objects,
+scenes, atomic actions). However, most real-world videos last for minutes or
+hours and have a complex hierarchical structure spanning different temporal
+granularities. We propose Video ReCap, a recursive video captioning model that
+can process video inputs of dramatically different lengths (from 1 second to 2
+hours) and output video captions at multiple hierarchy levels. The recursive
+video-language architecture exploits the synergy between different video
+hierarchies and can process hour-long videos efficiently. We utilize a
+curriculum learning training scheme to learn the hierarchical structure of
+videos, starting from clip-level captions describing atomic actions, then
+focusing on segment-level descriptions, and concluding with generating
+summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
+augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
+recursive model can flexibly generate captions at different hierarchy levels
+while also being useful for other complex video understanding tasks, such as
+VideoQA on EgoSchema. Data, code, and models are available at:
+https://sites.google.com/view/vidrecap",cs.CV,['cs.CV']
+SuperSVG: Superpixel-based Scalable Vector Graphics Synthesis,Teng Hu · Ran Yi · Baihong Qian · Jiangning Zhang · Paul L. Rosin · Yu-Kun Lai, ,https://arxiv.org/html/2405.02962v1,,2405.02962v1.pdf,VectorPainter: A Novel Approach to Stylized Vector Graphics Synthesis with Vectorized Strokes,"We propose a novel method, VectorPainter, for the task of stylized vector
+graphics synthesis. Given a text prompt and a reference style image,
+VectorPainter generates a vector graphic that aligns in content with the text
+prompt and remains faithful in style to the reference image. We recognize that
+the key to this task lies in fully leveraging the intrinsic properties of
+vector graphics. Innovatively, we conceptualize the stylization process as the
+rearrangement of vectorized strokes extracted from the reference image.
+VectorPainter employs an optimization-based pipeline. It begins by extracting
+vectorized strokes from the reference image, which are then used to initialize
+the synthesis process. To ensure fidelity to the reference style, a novel style
+preservation loss is introduced. Extensive experiments have been conducted to
+demonstrate that our method is capable of aligning with the text description
+while remaining faithful to the reference image.",cs.CV,['cs.CV']
+GeoAuxNet: Towards Universal 3D Representation Learning for Multi-sensor Point Clouds,Shengjun Zhang · Xin Fei · Yueqi Duan, ,https://arxiv.org/abs/2403.19220,,2403.19220.pdf,GeoAuxNet: Towards Universal 3D Representation Learning for Multi-sensor Point Clouds,"Point clouds captured by different sensors such as RGB-D cameras and LiDAR
+possess non-negligible domain gaps. Most existing methods design different
+network architectures and train separately on point clouds from various
+sensors. Typically, point-based methods achieve outstanding performances on
+even-distributed dense point clouds from RGB-D cameras, while voxel-based
+methods are more efficient for large-range sparse LiDAR point clouds. In this
+paper, we propose geometry-to-voxel auxiliary learning to enable voxel
+representations to access point-level geometric information, which supports
+better generalisation of the voxel-based backbone with additional
+interpretations of multi-sensor point clouds. Specifically, we construct
+hierarchical geometry pools generated by a voxel-guided dynamic point network,
+which efficiently provide auxiliary fine-grained geometric information adapted
+to different stages of voxel features. We conduct experiments on joint
+multi-sensor datasets to demonstrate the effectiveness of GeoAuxNet. Enjoying
+elaborate geometric information, our method outperforms other models
+collectively trained on multi-sensor datasets, and achieve competitive results
+with the-state-of-art experts on each single dataset.",cs.CV,['cs.CV']
+BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics,Wenqian Zhang · Molin Huang · Yuxuan Zhou · Juze Zhang · Jingyi Yu · Jingya Wang · Lan Xu,https://github.com/Godheritage/BOTH2Hands,https://arxiv.org/abs/2312.07937,,2312.07937.pdf,BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics,"The recently emerging text-to-motion advances have spired numerous attempts
+for convenient and interactive human motion generation. Yet, existing methods
+are largely limited to generating body motions only without considering the
+rich two-hand motions, let alone handling various conditions like body dynamics
+or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal
+dataset for two-hand motion generation. Our dataset includes accurate motion
+tracking for the human body and hands and provides pair-wised finger-level hand
+annotations and body descriptions. We further provide a strong baseline method,
+BOTH2Hands, for the novel task: generating vivid two-hand motions from both
+implicit body dynamics and explicit text prompts. We first warm up two parallel
+body-to-hand and text-to-hand diffusion models and then utilize the
+cross-attention transformer for motion blending. Extensive experiments and
+cross-validations demonstrate the effectiveness of our approach and dataset for
+generating convincing two-hand motions from the hybrid body-and-textual
+conditions. Our dataset and code will be disseminated to the community for
+future research.",cs.CV,['cs.CV']
+Paint3D: Paint Anything 3D with Lighting-less Texture Diffusion Models,Xianfang Zeng · Xin Chen · Zhongqi Qi · Wen Liu · Zibo Zhao · Zhibin Wang · Bin Fu · Yong Liu · Gang Yu, ,https://arxiv.org/abs/2312.13913,,2312.13913.pdf,Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models,"This paper presents Paint3D, a novel coarse-to-fine generative framework that
+is capable of producing high-resolution, lighting-less, and diverse 2K UV
+texture maps for untextured 3D meshes conditioned on text or image inputs. The
+key challenge addressed is generating high-quality textures without embedded
+illumination information, which allows the textures to be re-lighted or
+re-edited within modern graphics pipelines. To achieve this, our method first
+leverages a pre-trained depth-aware 2D diffusion model to generate
+view-conditional images and perform multi-view texture fusion, producing an
+initial coarse texture map. However, as 2D models cannot fully represent 3D
+shapes and disable lighting effects, the coarse texture map exhibits incomplete
+areas and illumination artifacts. To resolve this, we train separate UV
+Inpainting and UVHD diffusion models specialized for the shape-aware refinement
+of incomplete areas and the removal of illumination artifacts. Through this
+coarse-to-fine process, Paint3D can produce high-quality 2K UV textures that
+maintain semantic consistency while being lighting-less, significantly
+advancing the state-of-the-art in texturing 3D objects.",cs.CV,['cs.CV']
+Overload: Latency Attacks on Object Detection for Edge Devices,Erh-Chung Chen · Pin-Yu Chen · I-Hsin Chung · Che-Rung Lee, ,https://ar5iv.labs.arxiv.org/html/2304.05370,,2304.05370.pdf,Overload: Latency Attacks on Object Detection for Edge Devices,"Nowadays, the deployment of deep learning-based applications is an essential
+task owing to the increasing demands on intelligent services. In this paper, we
+investigate latency attacks on deep learning applications. Unlike common
+adversarial attacks for misclassification, the goal of latency attacks is to
+increase the inference time, which may stop applications from responding to the
+requests within a reasonable time. This kind of attack is ubiquitous for
+various applications, and we use object detection to demonstrate how such kind
+of attacks work. We also design a framework named Overload to generate latency
+attacks at scale. Our method is based on a newly formulated optimization
+problem and a novel technique, called spatial attention. This attack serves to
+escalate the required computing costs during the inference time, consequently
+leading to an extended inference time for object detection. It presents a
+significant threat, especially to systems with limited computing resources. We
+conducted experiments using YOLOv5 models on Nvidia NX. Compared to existing
+methods, our method is simpler and more effective. The experimental results
+show that with latency attacks, the inference time of a single image can be
+increased ten times longer in reference to the normal setting. Moreover, our
+findings pose a potential new threat to all object detection tasks requiring
+non-maximum suppression (NMS), as our attack is NMS-agnostic.",cs.CV,['cs.CV']
+OmniGlue: Generalizable Feature Matching with Foundation Model Guidance,Hanwen Jiang · Arjun Karpur · Bingyi Cao · Qixing Huang · André Araujo, ,https://arxiv.org/abs/2405.12979,,2405.12979.pdf,OmniGlue: Generalizable Feature Matching with Foundation Model Guidance,"The image matching field has been witnessing a continuous emergence of novel
+learnable feature matching techniques, with ever-improving performance on
+conventional benchmarks. However, our investigation shows that despite these
+gains, their potential for real-world applications is restricted by their
+limited generalization capabilities to novel image domains. In this paper, we
+introduce OmniGlue, the first learnable image matcher that is designed with
+generalization as a core principle. OmniGlue leverages broad knowledge from a
+vision foundation model to guide the feature matching process, boosting
+generalization to domains not seen at training time. Additionally, we propose a
+novel keypoint position-guided attention mechanism which disentangles spatial
+and appearance information, leading to enhanced matching descriptors. We
+perform comprehensive experiments on a suite of $7$ datasets with varied image
+domains, including scene-level, object-centric and aerial images. OmniGlue's
+novel components lead to relative gains on unseen domains of $20.9\%$ with
+respect to a directly comparable reference model, while also outperforming the
+recent LightGlue method by $9.5\%$ relatively.Code and model can be found at
+https://hwjiang1510.github.io/OmniGlue",cs.CV,['cs.CV']
+InstaGen: Enhancing Object Detection by Training on Synthetic Dataset,Chengjian Feng · Yujie Zhong · Zequn Jie · Weidi Xie · Lin Ma, ,https://arxiv.org/abs/2402.05937,,2402.05937.pdf,InstaGen: Enhancing Object Detection by Training on Synthetic Dataset,"In this paper, we present a novel paradigm to enhance the ability of object
+detector, e.g., expanding categories or improving detection performance, by
+training on synthetic dataset generated from diffusion models. Specifically, we
+integrate an instance-level grounding head into a pre-trained, generative
+diffusion model, to augment it with the ability of localising instances in the
+generated images. The grounding head is trained to align the text embedding of
+category names with the regional visual feature of the diffusion model, using
+supervision from an off-the-shelf object detector, and a novel self-training
+scheme on (novel) categories not covered by the detector. We conduct thorough
+experiments to show that, this enhanced version of diffusion model, termed as
+InstaGen, can serve as a data synthesizer, to enhance object detectors by
+training on its generated samples, demonstrating superior performance over
+existing state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse
+(+1.2 to 5.2 AP) scenarios. Project page with code:
+https://fcjian.github.io/InstaGen.",cs.CV,['cs.CV']
+LTM: Lightweight Textured Mesh Extraction and Refinement of Large Unbounded Scenes for Efficient Storage and Real-time Rendering,Jaehoon Choi · Rajvi Shah · Qinbo Li · Yipeng Wang · Ayush Saraf · Changil Kim · Jia-Bin Huang · Dinesh Manocha · Suhib Alsisan · Johannes Kopf,https://jh-choi.github.io/LTMM,https://arxiv.org/html/2404.15891v2,,2404.15891v2.pdf,OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation,"Recent advancements in 3D reconstruction technologies have paved the way for
+high-quality and real-time rendering of complex 3D scenes. Despite these
+achievements, a notable challenge persists: it is difficult to precisely
+reconstruct specific objects from large scenes. Current scene reconstruction
+techniques frequently result in the loss of object detail textures and are
+unable to reconstruct object portions that are occluded or unseen in views. To
+address this challenge, we delve into the meticulous 3D reconstruction of
+specific objects within large scenes and propose a framework termed OMEGAS:
+Object Mesh Extraction from Large Scenes Guided by GAussian Segmentation.
+OMEGAS employs a multi-step approach, grounded in several excellent
+off-the-shelf methodologies. Specifically, initially, we utilize the Segment
+Anything Model (SAM) to guide the segmentation of 3D Gaussian Splatting (3DGS),
+thereby creating a basic 3DGS model of the target object. Then, we leverage
+large-scale diffusion priors to further refine the details of the 3DGS model,
+especially aimed at addressing invisible or occluded object portions from the
+original scene views. Subsequently, by re-rendering the 3DGS model onto the
+scene views, we achieve accurate object segmentation and effectively remove the
+background. Finally, these target-only images are used to improve the 3DGS
+model further and extract the definitive 3D object mesh by the SuGaR model. In
+various scenarios, our experiments demonstrate that OMEGAS significantly
+surpasses existing scene reconstruction methods. Our project page is at:
+https://github.com/CrystalWlz/OMEGAS",cs.CV,['cs.CV']
+Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models,Pablo Marcos-Manchón · Roberto Alcover-Couso · Juan SanMiguel · Jose M. Martinez,https://github.com/vpulab/ovam,https://arxiv.org/abs/2403.14291v1,,2403.14291v1.pdf,Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models,"Diffusion models represent a new paradigm in text-to-image generation. Beyond
+generating high-quality images from text prompts, models such as Stable
+Diffusion have been successfully extended to the joint generation of semantic
+segmentation pseudo-masks. However, current extensions primarily rely on
+extracting attentions linked to prompt words used for image synthesis. This
+approach limits the generation of segmentation masks derived from word tokens
+not contained in the text prompt. In this work, we introduce Open-Vocabulary
+Attention Maps (OVAM)-a training-free method for text-to-image diffusion models
+that enables the generation of attention maps for any word. In addition, we
+propose a lightweight optimization process based on OVAM for finding tokens
+that generate accurate attention maps for an object class with a single
+annotation. We evaluate these tokens within existing state-of-the-art Stable
+Diffusion extensions. The best-performing model improves its mIoU from 52.1 to
+86.6 for the synthetic images' pseudo-masks, demonstrating that our optimized
+tokens are an efficient way to improve the performance of existing methods
+without architectural changes or retraining.",cs.CV,['cs.CV']
+Leveraging Frame Affinity for sRGB-to-RAW Video De-rendering,Chen Zhang · Wencheng Han · Yang Zhou · Jianbing Shen · Cheng-Zhong Xu · Wentao Liu, ,https://arxiv.org/abs/2404.09490,,2404.09490.pdf,Leveraging Temporal Contextualization for Video Action Recognition,"Pretrained vision-language models have shown effectiveness in video
+understanding. However, recent studies have not sufficiently leveraged
+essential temporal information from videos, simply averaging frame-wise
+representations or referencing consecutive frames. We introduce Temporally
+Contextualized CLIP (TC-CLIP), a pioneering framework for video understanding
+that effectively and efficiently leverages comprehensive video information. We
+propose Temporal Contextualization (TC), a novel layer-wise temporal
+information infusion mechanism for video that extracts core information from
+each frame, interconnects relevant information across the video to summarize
+into context tokens, and ultimately leverages the context tokens during the
+feature encoding process. Furthermore, our Video-conditional Prompting (VP)
+module manufactures context tokens to generate informative prompts in text
+modality. We conduct extensive experiments in zero-shot, few-shot,
+base-to-novel, and fully-supervised action recognition to validate the
+superiority of our TC-CLIP. Ablation studies for TC and VP guarantee our design
+choices. Code is available at https://github.com/naver-ai/tc-clip",cs.CV,['cs.CV']
+UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion,Junsheng Zhou · Weiqi Zhang · Baorui Ma · Kanle Shi · Yu-Shen Liu · Zhizhong Han, ,https://arxiv.org/abs/2404.06851,,2404.06851.pdf,UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion,"Diffusion models have shown remarkable results for image generation, editing
+and inpainting. Recent works explore diffusion models for 3D shape generation
+with neural implicit functions, i.e., signed distance function and occupancy
+function. However, they are limited to shapes with closed surfaces, which
+prevents them from generating diverse 3D real-world contents containing open
+surfaces. In this work, we present UDiFF, a 3D diffusion model for unsigned
+distance fields (UDFs) which is capable to generate textured 3D shapes with
+open surfaces from text conditions or unconditionally. Our key idea is to
+generate UDFs in spatial-frequency domain with an optimal wavelet
+transformation, which produces a compact representation space for UDF
+generation. Specifically, instead of selecting an appropriate wavelet
+transformation which requires expensive manual efforts and still leads to large
+information loss, we propose a data-driven approach to learn the optimal
+wavelet transformation for UDFs. We evaluate UDiFF to show our advantages by
+numerical and visual comparisons with the latest methods on widely used
+benchmarks. Page: https://weiqi-zhang.github.io/UDiFF.",cs.CV,['cs.CV']
+OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation,Xiongwei Wu · Sicheng Yu · Ee-Peng Lim · Chong Wah Ngo, ,https://arxiv.org/abs/2404.01409,,2404.01409.pdf,OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation,"In the realm of food computing, segmenting ingredients from images poses
+substantial challenges due to the large intra-class variance among the same
+ingredients, the emergence of new ingredients, and the high annotation costs
+associated with large food segmentation datasets. Existing approaches primarily
+utilize a closed-vocabulary and static text embeddings setting. These methods
+often fall short in effectively handling the ingredients, particularly new and
+diverse ones. In response to these limitations, we introduce OVFoodSeg, a
+framework that adopts an open-vocabulary setting and enhances text embeddings
+with visual context. By integrating vision-language models (VLMs), our approach
+enriches text embedding with image-specific information through two innovative
+modules, eg, an image-to-text learner FoodLearner and an Image-Informed Text
+Encoder. The training process of OVFoodSeg is divided into two stages: the
+pre-training of FoodLearner and the subsequent learning phase for segmentation.
+The pre-training phase equips FoodLearner with the capability to align visual
+information with corresponding textual representations that are specifically
+related to food, while the second phase adapts both the FoodLearner and the
+Image-Informed Text Encoder for the segmentation task. By addressing the
+deficiencies of previous models, OVFoodSeg demonstrates a significant
+improvement, achieving an 4.9\% increase in mean Intersection over Union (mIoU)
+on the FoodSeg103 dataset, setting a new milestone for food image segmentation.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM']"
+LaneCPP: Continuous 3D Lane Detection using Physical Priors,Maximilian Pittner · Joel Janai · Alexandru Paul Condurache, ,https://arxiv.org/abs/2401.08036,,2401.08036.pdf,3D Lane Detection from Front or Surround-View using Joint-Modeling & Matching,"3D lanes offer a more comprehensive understanding of the road surface
+geometry than 2D lanes, thereby providing crucial references for driving
+decisions and trajectory planning. While many efforts aim to improve prediction
+accuracy, we recognize that an efficient network can bring results closer to
+lane modeling. However, if the modeling data is imprecise, the results might
+not accurately capture the real-world scenario. Therefore, accurate lane
+modeling is essential to align prediction results closely with the environment.
+This study centers on efficient and accurate lane modeling, proposing a joint
+modeling approach that combines Bezier curves and interpolation methods.
+Furthermore, based on this lane modeling approach, we developed a Global2Local
+Lane Matching method with Bezier Control-Point and Key-Point, which serve as a
+comprehensive solution that leverages hierarchical features with two
+mathematical models to ensure a precise match. We also introduce a novel 3D
+Spatial Encoder, representing an exploration of 3D surround-view lane detection
+research. The framework is suitable for front-view or surround-view 3D lane
+detection. By directly outputting the key points of lanes in 3D space, it
+overcomes the limitations of anchor-based methods, enabling accurate prediction
+of closed-loop or U-shaped lanes and effective adaptation to complex road
+conditions. This innovative method establishes a new benchmark in front-view 3D
+lane detection on the Openlane dataset and achieves competitive performance in
+surround-view 2D lane detection on the Argoverse2 dataset.",cs.CV,['cs.CV']
+MonoNPHM: Dynamic Head Reconstruction from Monocular Videos,Simon Giebenhain · Tobias Kirschstein · Markos Georgopoulos · Martin Rünz · Lourdes Agapito · Matthias Nießner,https://simongiebenhain.github.io/MonoNPHM/,https://arxiv.org/abs/2312.06740,,2312.06740.pdf,MonoNPHM: Dynamic Head Reconstruction from Monocular Videos,"We present Monocular Neural Parametric Head Models (MonoNPHM) for dynamic 3D
+head reconstructions from monocular RGB videos. To this end, we propose a
+latent appearance space that parameterizes a texture field on top of a neural
+parametric model. We constrain predicted color values to be correlated with the
+underlying geometry such that gradients from RGB effectively influence latent
+geometry codes during inverse rendering. To increase the representational
+capacity of our expression space, we augment our backward deformation field
+with hyper-dimensions, thus improving color and geometry representation in
+topologically challenging expressions. Using MonoNPHM as a learned prior, we
+approach the task of 3D head reconstruction using signed distance field based
+volumetric rendering. By numerically inverting our backward deformation field,
+we incorporated a landmark loss using facial anchor points that are closely
+tied to our canonical geometry representation. To evaluate the task of dynamic
+face reconstruction from monocular RGB videos we record 20 challenging Kinect
+sequences under casual conditions. MonoNPHM outperforms all baselines with a
+significant margin, and makes an important step towards easily accessible
+neural parametric face models through RGB tracking.",cs.CV,['cs.CV']
+Retrieval-Augmented Egocentric Video Captioning,Jilan Xu · Yifei Huang · Junlin Hou · Guo Chen · Yuejie Zhang · Rui Feng · Weidi Xie, ,https://arxiv.org/abs/2401.00789,,2401.00789.pdf,Retrieval-Augmented Egocentric Video Captioning,"Understanding human actions from videos of first-person view poses
+significant challenges. Most prior approaches explore representation learning
+on egocentric videos only, while overlooking the potential benefit of
+exploiting existing large-scale third-person videos. In this paper, (1) we
+develop EgoInstructor, a retrieval-augmented multimodal captioning model that
+automatically retrieves semantically relevant third-person instructional videos
+to enhance the video captioning of egocentric videos. (2) For training the
+cross-view retrieval module, we devise an automatic pipeline to discover
+ego-exo video pairs from distinct large-scale egocentric and exocentric
+datasets. (3) We train the cross-view retrieval module with a novel EgoExoNCE
+loss that pulls egocentric and exocentric video features closer by aligning
+them to shared text features that describe similar actions. (4) Through
+extensive experiments, our cross-view retrieval module demonstrates superior
+performance across seven benchmarks. Regarding egocentric video captioning,
+EgoInstructor exhibits significant improvements by leveraging third-person
+videos as references.",cs.CV,['cs.CV']
+Relaxed Contrastive Learning for Federated Learning,Seonguk Seo · Jinkyu Kim · Geeho Kim · Bohyung Han, ,https://arxiv.org/abs/2401.04928,,2401.04928.pdf,Relaxed Contrastive Learning for Federated Learning,"We propose a novel contrastive learning framework to effectively address the
+challenges of data heterogeneity in federated learning. We first analyze the
+inconsistency of gradient updates across clients during local training and
+establish its dependence on the distribution of feature representations,
+leading to the derivation of the supervised contrastive learning (SCL)
+objective to mitigate local deviations. In addition, we show that a na\""ive
+adoption of SCL in federated learning leads to representation collapse,
+resulting in slow convergence and limited performance gains. To address this
+issue, we introduce a relaxed contrastive learning loss that imposes a
+divergence penalty on excessively similar sample pairs within each class. This
+strategy prevents collapsed representations and enhances feature
+transferability, facilitating collaborative training and leading to significant
+performance improvements. Our framework outperforms all existing federated
+learning approaches by huge margins on the standard benchmarks through
+extensive experimental results.",cs.LG,['cs.LG']
+Rewrite the stars,Xu Ma · Xiyang Dai · Yue Bai · Yizhou Wang · Yun Fu, ,https://arxiv.org/abs/2403.19967,,2403.19967.pdf,Rewrite the Stars,"Recent studies have drawn attention to the untapped potential of the ""star
+operation"" (element-wise multiplication) in network design. While intuitive
+explanations abound, the foundational rationale behind its application remains
+largely unexplored. Our study attempts to reveal the star operation's ability
+to map inputs into high-dimensional, non-linear feature spaces -- akin to
+kernel tricks -- without widening the network. We further introduce StarNet, a
+simple yet powerful prototype, demonstrating impressive performance and low
+latency under compact network structure and efficient budget. Like stars in the
+sky, the star operation appears unremarkable but holds a vast universe of
+potential. Our work encourages further exploration across tasks, with codes
+available at https://github.com/ma-xu/Rewrite-the-Stars.",cs.CV,['cs.CV']
+PointInfinity: Resolution-Invariant Point Diffusion Models,Zixuan Huang · Justin Johnson · Shoubhik Debnath · James Rehg · Chao-Yuan Wu,https://zixuanh.com/projects/pointinfinity,https://arxiv.org/abs/2404.03566v1,,2404.03566v1.pdf,PointInfinity: Resolution-Invariant Point Diffusion Models,"We present PointInfinity, an efficient family of point cloud diffusion
+models. Our core idea is to use a transformer-based architecture with a
+fixed-size, resolution-invariant latent representation. This enables efficient
+training with low-resolution point clouds, while allowing high-resolution point
+clouds to be generated during inference. More importantly, we show that scaling
+the test-time resolution beyond the training resolution improves the fidelity
+of generated point clouds and surfaces. We analyze this phenomenon and draw a
+link to classifier-free guidance commonly used in diffusion models,
+demonstrating that both allow trading off fidelity and variability during
+inference. Experiments on CO3D show that PointInfinity can efficiently generate
+high-resolution point clouds (up to 131k points, 31 times more than Point-E)
+with state-of-the-art quality.",cs.CV,['cs.CV']
+JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation,Yu Zeng · Vishal M. Patel · Haochen Wang · Xun Huang · Ting-Chun Wang · Ming-Yu Liu · Yogesh Balaji,https://research.nvidia.com/labs/dir/jedi/,https://arxiv.org/html/2307.04725v2,,2307.04725v2.pdf,AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,"With the advance of text-to-image (T2I) diffusion models (e.g., Stable
+Diffusion) and corresponding personalization techniques such as DreamBooth and
+LoRA, everyone can manifest their imagination into high-quality images at an
+affordable cost. However, adding motion dynamics to existing high-quality
+personalized T2Is and enabling them to generate animations remains an open
+challenge. In this paper, we present AnimateDiff, a practical framework for
+animating personalized T2I models without requiring model-specific tuning. At
+the core of our framework is a plug-and-play motion module that can be trained
+once and seamlessly integrated into any personalized T2Is originating from the
+same base T2I. Through our proposed training strategy, the motion module
+effectively learns transferable motion priors from real-world videos. Once
+trained, the motion module can be inserted into a personalized T2I model to
+form a personalized animation generator. We further propose MotionLoRA, a
+lightweight fine-tuning technique for AnimateDiff that enables a pre-trained
+motion module to adapt to new motion patterns, such as different shot types, at
+a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA
+on several public representative personalized T2I models collected from the
+community. The results demonstrate that our approaches help these models
+generate temporally smooth animation clips while preserving the visual quality
+and motion diversity. Codes and pre-trained weights are available at
+https://github.com/guoyww/AnimateDiff.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models,Shweta Mahajan · Tanzila Rahman · Kwang Moo Yi · Leonid Sigal, ,https://arxiv.org/abs/2312.12416,,2312.12416.pdf,Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models,"The quality of the prompts provided to text-to-image diffusion models
+determines how faithful the generated content is to the user's intent, often
+requiring `prompt engineering'. To harness visual concepts from target images
+without prompt engineering, current approaches largely rely on embedding
+inversion by optimizing and then mapping them to pseudo-tokens. However,
+working with such high-dimensional vector representations is challenging
+because they lack semantics and interpretability, and only allow simple vector
+operations when using them. Instead, this work focuses on inverting the
+diffusion model to obtain interpretable language prompts directly. The
+challenge of doing this lies in the fact that the resulting optimization
+problem is fundamentally discrete and the space of prompts is exponentially
+large; this makes using standard optimization techniques, such as stochastic
+gradient descent, difficult. To this end, we utilize a delayed projection
+scheme to optimize for prompts representative of the vocabulary space in the
+model. Further, we leverage the findings that different timesteps of the
+diffusion process cater to different levels of detail in an image. The later,
+noisy, timesteps of the forward diffusion process correspond to the semantic
+information, and therefore, prompt inversion in this range provides tokens
+representative of the image semantics. We show that our approach can identify
+semantically interpretable and meaningful prompts for a target image which can
+be used to synthesize diverse images with similar content. We further
+illustrate the application of the optimized prompts in evolutionary image
+generation and concept removal.",cs.CV,"['cs.CV', 'cs.LG']"
+Pixel Aligned Language Models,Jiarui Xu · Xingyi Zhou · Shen Yan · Xiuye Gu · Anurag Arnab · Chen Sun · Xiaolong Wang · Cordelia Schmid,https://jerryxu.net/PixelLLM/,https://arxiv.org/abs/2312.09237,,2312.09237.pdf,Pixel Aligned Language Models,"Large language models have achieved great success in recent years, so as
+their variants in vision. Existing vision-language models can describe images
+in natural languages, answer visual-related questions, or perform complex
+reasoning about the image. However, it is yet unclear how localization tasks,
+such as word grounding or referring localization, can be performed using large
+language models. In this work, we aim to develop a vision-language model that
+can take locations, for example, a set of points or boxes, as either inputs or
+outputs. When taking locations as inputs, the model performs
+location-conditioned captioning, which generates captions for the indicated
+object or region. When generating locations as outputs, our model regresses
+pixel coordinates for each output word generated by the language model, and
+thus performs dense word grounding. Our model is pre-trained on the Localized
+Narrative dataset, which contains pixel-word-aligned captioning from human
+attention. We show our model can be applied to various location-aware
+vision-language tasks, including referring localization, location-conditioned
+captioning, and dense object captioning, archiving state-of-the-art performance
+on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .",cs.CV,['cs.CV']
+Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning,Xinshun Wang · Zhongbin Fang · Xia Li · Xiangtai Li · Chen Chen · Mengyuan Liu, ,https://arxiv.org/abs/2312.03703,,2312.03703.pdf,Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning,"In-context learning provides a new perspective for multi-task modeling for
+vision and NLP. Under this setting, the model can perceive tasks from prompts
+and accomplish them without any extra task-specific head predictions or model
+fine-tuning. However, Skeleton sequence modeling via in-context learning
+remains unexplored. Directly applying existing in-context models from other
+areas onto skeleton sequences fails due to the inter-frame and cross-task pose
+similarity that makes it outstandingly hard to perceive the task correctly from
+a subtle context. To address this challenge, we propose Skeleton-in-Context
+(SiC), an effective framework for in-context skeleton sequence modeling. Our
+SiC is able to handle multiple skeleton-based tasks simultaneously after a
+single training process and accomplish each task from context according to the
+given prompt. It can further generalize to new, unseen tasks according to
+customized prompts. To facilitate context perception, we additionally propose a
+task-unified prompt, which adaptively learns tasks of different natures, such
+as partial joint-level generation, sequence-level prediction, or 2D-to-3D
+motion prediction. We conduct extensive experiments to evaluate the
+effectiveness of our SiC on multiple tasks, including motion prediction, pose
+estimation, joint completion, and future pose estimation. We also evaluate its
+generalization capability on unseen tasks such as motion-in-between. These
+experiments show that our model achieves state-of-the-art multi-task
+performance and even outperforms single-task methods on certain tasks.",cs.CV,['cs.CV']
+CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment,Hyeongmin Lee · Kyoungkook Kang · Jungseul Ok · Sunghyun Cho, ,https://arxiv.org/abs/2404.01123,,2404.01123.pdf,CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment,"Recent image tone adjustment (or enhancement) approaches have predominantly
+adopted supervised learning for learning human-centric perceptual assessment.
+However, these approaches are constrained by intrinsic challenges of supervised
+learning. Primarily, the requirement for expertly-curated or retouched images
+escalates the data acquisition expenses. Moreover, their coverage of target
+style is confined to stylistic variants inferred from the training data. To
+surmount the above challenges, we propose an unsupervised learning-based
+approach for text-based image tone adjustment method, CLIPtone, that extends an
+existing image enhancement method to accommodate natural language descriptions.
+Specifically, we design a hyper-network to adaptively modulate the pretrained
+parameters of the backbone model based on text description. To assess whether
+the adjusted image aligns with the text description without ground truth image,
+we utilize CLIP, which is trained on a vast set of language-image pairs and
+thus encompasses knowledge of human perception. The major advantages of our
+approach are three fold: (i) minimal data collection expenses, (ii) support for
+a range of adjustments, and (iii) the ability to handle novel text descriptions
+unseen in training. Our approach's efficacy is demonstrated through
+comprehensive experiments, including a user study.",cs.CV,"['cs.CV', 'cs.GR', 'eess.IV']"
+PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor,Jaewon Jung · Hongsun Jang · Jaeyong Song · Jinho Lee,https://github.com/jaewonalive/PeerAiD,https://arxiv.org/abs/2403.06668,,2403.06668.pdf,PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor,"Adversarial robustness of the neural network is a significant concern when it
+is applied to security-critical domains. In this situation, adversarial
+distillation is a promising option which aims to distill the robustness of the
+teacher network to improve the robustness of a small student network. Previous
+works pretrain the teacher network to make it robust against the adversarial
+examples aimed at itself. However, the adversarial examples are dependent on
+the parameters of the target network. The fixed teacher network inevitably
+degrades its robustness against the unseen transferred adversarial examples
+which target the parameters of the student network in the adversarial
+distillation process. We propose PeerAiD to make a peer network learn the
+adversarial examples of the student network instead of adversarial examples
+aimed at itself. PeerAiD is an adversarial distillation that trains the peer
+network and the student network simultaneously in order to specialize the peer
+network for defending the student network. We observe that such peer networks
+surpass the robustness of the pretrained robust teacher model against
+adversarial examples aimed at the student network. With this peer network and
+adversarial distillation, PeerAiD achieves significantly higher robustness of
+the student network with AutoAttack (AA) accuracy by up to 1.66%p and improves
+the natural accuracy of the student network by up to 4.72%p with ResNet-18 on
+TinyImageNet dataset. Code is available at
+https://github.com/jaewonalive/PeerAiD.",cs.LG,"['cs.LG', 'cs.CV']"
+MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer,Jianjian Cao · Peng Ye · Shengze Li · Chong Yu · Yansong Tang · Jiwen Lu · Tao Chen,https://github.com/double125/MADTP,https://arxiv.org/abs/2403.02991,,2403.02991.pdf,MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer,"Vision-Language Transformers (VLTs) have shown great success recently, but
+are meanwhile accompanied by heavy computation costs, where a major reason can
+be attributed to the large number of visual and language tokens. Existing token
+pruning research for compressing VLTs mainly follows a single-modality-based
+scheme yet ignores the critical role of aligning different modalities for
+guiding the token pruning process, causing the important tokens for one
+modality to be falsely pruned in another modality branch. Meanwhile, existing
+VLT pruning works also lack the flexibility to dynamically compress each layer
+based on different input samples. To this end, we propose a novel framework
+named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for
+accelerating various VLTs. Specifically, we first introduce a well-designed
+Multi-modality Alignment Guidance (MAG) module that can align features of the
+same semantic concept from different modalities, to ensure the pruned tokens
+are less important for all modalities. We further design a novel Dynamic Token
+Pruning (DTP) module, which can adaptively adjust the token compression ratio
+in each layer based on different input instances. Extensive experiments on
+various benchmarks demonstrate that MADTP significantly reduces the
+computational complexity of kinds of multimodal models while preserving
+competitive performance. Notably, when applied to the BLIP model in the NLVR2
+dataset, MADTP can reduce the GFLOPs by 80% with less than 4% performance
+degradation.",cs.CV,['cs.CV']
+VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models,Hyeonho Jeong · Geon Yeong Park · Jong Chul Ye,https://video-motion-customization.github.io/,https://arxiv.org/abs/2312.00845,,2312.00845.pdf,VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models,"Text-to-video diffusion models have advanced video generation significantly.
+However, customizing these models to generate videos with tailored motions
+presents a substantial challenge. In specific, they encounter hurdles in (a)
+accurately reproducing motion from a target video, and (b) creating diverse
+visual variations. For example, straightforward extensions of static image
+customization methods to video often lead to intricate entanglements of
+appearance and motion data. To tackle this, here we present the Video Motion
+Customization (VMC) framework, a novel one-shot tuning approach crafted to
+adapt temporal attention layers within video diffusion models. Our approach
+introduces a novel motion distillation objective using residual vectors between
+consecutive frames as a motion reference. The diffusion process then preserves
+low-frequency motion trajectories while mitigating high-frequency
+motion-unrelated noise in image space. We validate our method against
+state-of-the-art video generative models across diverse real-world motions and
+contexts. Our codes, data and the project demo can be found at
+https://video-motion-customization.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting,Zijie Chen · Lichao Zhang · Fangsheng Weng · Lili Pan · ZHENZHONG Lan,https://github.com/zzjchen/Tailored-Visions,https://arxiv.org/abs/2310.08129,,2310.08129.pdf,Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting,"Despite significant progress in the field, it is still challenging to create
+personalized visual representations that align closely with the desires and
+preferences of individual users. This process requires users to articulate
+their ideas in words that are both comprehensible to the models and accurately
+capture their vision, posing difficulties for many users. In this paper, we
+tackle this challenge by leveraging historical user interactions with the
+system to enhance user prompts. We propose a novel approach that involves
+rewriting user prompts based on a newly collected large-scale text-to-image
+dataset with over 300k prompts from 3115 users. Our rewriting model enhances
+the expressiveness and alignment of user prompts with their intended visual
+outputs. Experimental results demonstrate the superiority of our methods over
+baseline approaches, as evidenced in our new offline evaluation method and
+online tests. Our code and dataset are available at
+https://github.com/zzjchen/Tailored-Visions.",cs.CV,['cs.CV']
+VideoBooth: Diffusion-based Video Generation with Image Prompts,Yuming Jiang · Tianxing Wu · Shuai Yang · Chenyang Si · Dahua Lin · Yu Qiao · Chen Change Loy · Ziwei Liu, ,https://arxiv.org/abs/2312.00777,,2312.00777.pdf,VideoBooth: Diffusion-based Video Generation with Image Prompts,"Text-driven video generation witnesses rapid progress. However, merely using
+text prompts is not enough to depict the desired subject appearance that
+accurately aligns with users' intents, especially for customized content
+creation. In this paper, we study the task of video generation with image
+prompts, which provide more accurate and direct content control beyond the text
+prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
+dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
+manner. Coarse visual embeddings from image encoder provide high-level
+encodings of image prompts, while fine visual embeddings from the proposed
+attention injection module provide multi-scale and detailed encoding of image
+prompts. These two complementary embeddings can faithfully capture the desired
+appearance. 2) In the attention injection module at fine level, multi-scale
+image prompts are fed into different cross-frame attention layers as additional
+keys and values. This extra spatial information refines the details in the
+first frame and then it is propagated to the remaining frames, which maintains
+temporal consistency. Extensive experiments demonstrate that VideoBooth
+achieves state-of-the-art performance in generating customized high-quality
+videos with subjects specified in image prompts. Notably, VideoBooth is a
+generalizable framework where a single model works for a wide range of image
+prompts with feed-forward pass.",cs.CV,['cs.CV']
+FreeU: Free Lunch in Diffusion U-Net,Chenyang Si · Ziqi Huang · Yuming Jiang · Ziwei Liu,https://chenyangsi.top/FreeU/,https://arxiv.org/abs/2309.11497,,2309.11497.pdf,FreeU: Free Lunch in Diffusion U-Net,"In this paper, we uncover the untapped potential of diffusion U-Net, which
+serves as a ""free lunch"" that substantially improves the generation quality on
+the fly. We initially investigate the key contributions of the U-Net
+architecture to the denoising process and identify that its main backbone
+primarily contributes to denoising, whereas its skip connections mainly
+introduce high-frequency features into the decoder module, causing the network
+to overlook the backbone semantics. Capitalizing on this discovery, we propose
+a simple yet effective method-termed ""FreeU"" - that enhances generation quality
+without additional training or finetuning. Our key insight is to strategically
+re-weight the contributions sourced from the U-Net's skip connections and
+backbone feature maps, to leverage the strengths of both components of the
+U-Net architecture. Promising results on image and video generation tasks
+demonstrate that our FreeU can be readily integrated to existing diffusion
+models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion,
+to improve the generation quality with only a few lines of code. All you need
+is to adjust two scaling factors during inference. Project page:
+https://chenyangsi.top/FreeU/.",cs.CV,['cs.CV']
+One-Shot Structure-Aware Stylized Image Synthesis,Hansam Cho · Jonghyun Lee · Seunggyu Chang · Yonghyun Jeong,https://github.com/hansam95/OSASIS,https://arxiv.org/abs/2402.17275,,2402.17275.pdf,One-Shot Structure-Aware Stylized Image Synthesis,"While GAN-based models have been successful in image stylization tasks, they
+often struggle with structure preservation while stylizing a wide range of
+input images. Recently, diffusion models have been adopted for image
+stylization but still lack the capability to maintain the original quality of
+input images. Building on this, we propose OSASIS: a novel one-shot stylization
+method that is robust in structure preservation. We show that OSASIS is able to
+effectively disentangle the semantics from the structure of an image, allowing
+it to control the level of content and style implemented to a given input. We
+apply OSASIS to various experimental settings, including stylization with
+out-of-domain reference images and stylization with text-driven manipulation.
+Results show that OSASIS outperforms other stylization methods, especially for
+input images that were rarely encountered during training, providing a
+promising solution to stylization via diffusion models.",cs.CV,['cs.CV']
+OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning,Noor Ahmed · Anna Kukleva · Bernt Schiele, ,https://arxiv.org/abs/2403.18550,,2403.18550.pdf,OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning,"Few-Shot Class-Incremental Learning (FSCIL) introduces a paradigm in which
+the problem space expands with limited data. FSCIL methods inherently face the
+challenge of catastrophic forgetting as data arrives incrementally, making
+models susceptible to overwriting previously acquired knowledge. Moreover,
+given the scarcity of labeled samples available at any given time, models may
+be prone to overfitting and find it challenging to strike a balance between
+extensive pretraining and the limited incremental data. To address these
+challenges, we propose the OrCo framework built on two core principles:
+features' orthogonality in the representation space, and contrastive learning.
+In particular, we improve the generalization of the embedding space by
+employing a combination of supervised and self-supervised contrastive losses
+during the pretraining phase. Additionally, we introduce OrCo loss to address
+challenges arising from data limitations during incremental sessions. Through
+feature space perturbations and orthogonality between classes, the OrCo loss
+maximizes margins and reserves space for the following incremental data. This,
+in turn, ensures the accommodation of incoming classes in the feature space
+without compromising previously acquired knowledge. Our experimental results
+showcase state-of-the-art performance across three benchmark datasets,
+including mini-ImageNet, CIFAR100, and CUB datasets. Code is available at
+https://github.com/noorahmedds/OrCo",cs.CV,['cs.CV']
+ZeroShape: Regression-based Zero-shot Shape Reconstruction,Zixuan Huang · Stefan Stojanov · Anh Thai · Varun Jampani · James Rehg, ,https://arxiv.org/abs/2312.14198,,2312.14198.pdf,ZeroShape: Regression-based Zero-shot Shape Reconstruction,"We study the problem of single-image zero-shot 3D shape reconstruction.
+Recent works learn zero-shot shape reconstruction through generative modeling
+of 3D assets, but these models are computationally expensive at train and
+inference time. In contrast, the traditional approach to this problem is
+regression-based, where deterministic models are trained to directly regress
+the object shape. Such regression methods possess much higher computational
+efficiency than generative methods. This raises a natural question: is
+generative modeling necessary for high performance, or conversely, are
+regression-based approaches still competitive? To answer this, we design a
+strong regression-based model, called ZeroShape, based on the converging
+findings in this field and a novel insight. We also curate a large real-world
+evaluation benchmark, with objects from three different real-world 3D datasets.
+This evaluation benchmark is more diverse and an order of magnitude larger than
+what prior works use to quantitatively evaluate their models, aiming at
+reducing the evaluation variance in our field. We show that ZeroShape not only
+achieves superior performance over state-of-the-art methods, but also
+demonstrates significantly higher computational and data efficiency.",cs.CV,['cs.CV']
+Robust Self-calibration of Focal Lengths from the Fundamental Matrix,Viktor Kocur · Daniel Kyselica · Zuzana Kukelova,https://github.com/kocurvik/robust_self_calibration,https://arxiv.org/abs/2311.16304,,2311.16304.pdf,Robust Self-calibration of Focal Lengths from the Fundamental Matrix,"The problem of self-calibration of two cameras from a given fundamental
+matrix is one of the basic problems in geometric computer vision. Under the
+assumption of known principal points and square pixels, the well-known Bougnoux
+formula offers a means to compute the two unknown focal lengths. However, in
+many practical situations, the formula yields inaccurate results due to
+commonly occurring singularities. Moreover, the estimates are sensitive to
+noise in the computed fundamental matrix and to the assumed positions of the
+principal points. In this paper, we therefore propose an efficient and robust
+iterative method to estimate the focal lengths along with the principal points
+of the cameras given a fundamental matrix and priors for the estimated camera
+parameters. In addition, we study a computationally efficient check of models
+generated within RANSAC that improves the accuracy of the estimated models
+while reducing the total computational time. Extensive experiments on real and
+synthetic data show that our iterative method brings significant improvements
+in terms of the accuracy of the estimated focal lengths over the Bougnoux
+formula and other state-of-the-art methods, even when relying on inaccurate
+priors.",cs.CV,['cs.CV']
+GauHuman: Articulated Gaussian Splatting from Monocular Human Videos,Shoukang Hu · Tao Hu · Ziwei Liu, ,,https://paperswithcode.com/paper/gauhuman-articulated-gaussian-splatting-from,,,,,nan
+Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation,Hyunwoo Ryu · Jiwoo Kim · Hyunseok An · Junwoo Chang · Joohwan Seo · Taehan Kim · Yubin Kim · Chaewon Hwang · Jongeun Choi · Roberto Horowitz,https://sites.google.com/view/diffusion-edfs,https://arxiv.org/abs/2309.02685,,2309.02685.pdf,Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation,"Diffusion generative modeling has become a promising approach for learning
+robotic manipulation tasks from stochastic human demonstrations. In this paper,
+we present Diffusion-EDFs, a novel SE(3)-equivariant diffusion-based approach
+for visual robotic manipulation tasks. We show that our proposed method
+achieves remarkable data efficiency, requiring only 5 to 10 human
+demonstrations for effective end-to-end training in less than an hour.
+Furthermore, our benchmark experiments demonstrate that our approach has
+superior generalizability and robustness compared to state-of-the-art methods.
+Lastly, we validate our methods with real hardware experiments. Project
+Website: https://sites.google.com/view/diffusion-edfs/home",cs.RO,"['cs.RO', 'cs.AI', 'cs.LG']"
+DREAM: Diffusion Rectification and Estimation-Adaptive Models,Jinxin Zhou · Tianyu Ding · Tianyi Chen · Jiachen Jiang · Ilya Zharkov · Zhihui Zhu · Luming Liang, ,https://arxiv.org/abs/2312.00210,,2312.00210.pdf,DREAM: Diffusion Rectification and Estimation-Adaptive Models,"We present DREAM, a novel training framework representing Diffusion
+Rectification and Estimation Adaptive Models, requiring minimal code changes
+(just three lines) yet significantly enhancing the alignment of training with
+sampling in diffusion models. DREAM features two components: diffusion
+rectification, which adjusts training to reflect the sampling process, and
+estimation adaptation, which balances perception against distortion. When
+applied to image super-resolution (SR), DREAM adeptly navigates the tradeoff
+between minimizing distortion and preserving high image quality. Experiments
+demonstrate DREAM's superiority over standard diffusion-based SR methods,
+showing a $2$ to $3\times $ faster training convergence and a $10$ to
+$20\times$ reduction in sampling steps to achieve comparable results. We hope
+DREAM will inspire a rethinking of diffusion model training paradigms.",cs.CV,"['cs.CV', 'cs.AI']"
+Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers,Hongjie Wang · Bhishma Dedhia · Niraj Jha,https://jha-lab.github.io/zerotprune/,https://ar5iv.labs.arxiv.org/html/2305.17328,,2305.17328.pdf,Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers,"Deployment of Transformer models on edge devices is becoming increasingly
+challenging due to the exponentially growing inference cost that scales
+quadratically with the number of tokens in the input sequence. Token pruning is
+an emerging solution to address this challenge due to its ease of deployment on
+various Transformer backbones. However, most token pruning methods require
+computationally expensive fine-tuning, which is undesirable in many edge
+deployment cases. In this work, we propose Zero-TPrune, the first zero-shot
+method that considers both the importance and similarity of tokens in
+performing token pruning. It leverages the attention graph of pre-trained
+Transformer models to produce an importance distribution for tokens via our
+proposed Weighted Page Rank (WPR) algorithm. This distribution further guides
+token partitioning for efficient similarity-based pruning. Due to the
+elimination of the fine-tuning overhead, Zero-TPrune can prune large models at
+negligible computational cost, switch between different pruning configurations
+at no computational cost, and perform hyperparameter tuning efficiently. We
+evaluate the performance of Zero-TPrune on vision tasks by applying it to
+various vision Transformer backbones and testing them on ImageNet. Without any
+fine-tuning, Zero-TPrune reduces the FLOPs cost of DeiT-S by 34.7% and improves
+its throughput by 45.3% with only 0.4% accuracy loss. Compared with
+state-of-the-art pruning methods that require fine-tuning, Zero-TPrune not only
+eliminates the need for fine-tuning after pruning but also does so with only
+0.1% accuracy loss. Compared with state-of-the-art fine-tuning-free pruning
+methods, Zero-TPrune reduces accuracy loss by up to 49% with similar FLOPs
+budgets. Project webpage: https://jha-lab.github.io/zerotprune.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'eess.IV']"
+FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects,Bowen Wen · Wei Yang · Jan Kautz · Stan Birchfield, ,https://arxiv.org/abs/2312.08344,,2312.08344.pdf,FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects,"We present FoundationPose, a unified foundation model for 6D object pose
+estimation and tracking, supporting both model-based and model-free setups. Our
+approach can be instantly applied at test-time to a novel object without
+fine-tuning, as long as its CAD model is given, or a small number of reference
+images are captured. We bridge the gap between these two setups with a neural
+implicit representation that allows for effective novel view synthesis, keeping
+the downstream pose estimation modules invariant under the same unified
+framework. Strong generalizability is achieved via large-scale synthetic
+training, aided by a large language model (LLM), a novel transformer-based
+architecture, and contrastive learning formulation. Extensive evaluation on
+multiple public datasets involving challenging scenarios and objects indicate
+our unified approach outperforms existing methods specialized for each task by
+a large margin. In addition, it even achieves comparable results to
+instance-level methods despite the reduced assumptions. Project page:
+https://nvlabs.github.io/FoundationPose/",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Improving Bird’s Eye View Semantic Segmentation by Task Decomposition,Tianhao Zhao · Yongcan Chen · Yu Wu · Tianyang Liu · Bo Du · Peilun Xiao · shi qiu · Hongda Yang · Guozhen Li · yi yang · Yutian Lin, ,https://arxiv.org/abs/2404.01925v1,,2404.01925v1.pdf,Improving Bird's Eye View Semantic Segmentation by Task Decomposition,"Semantic segmentation in bird's eye view (BEV) plays a crucial role in
+autonomous driving. Previous methods usually follow an end-to-end pipeline,
+directly predicting the BEV segmentation map from monocular RGB inputs.
+However, the challenge arises when the RGB inputs and BEV targets from distinct
+perspectives, making the direct point-to-point predicting hard to optimize. In
+this paper, we decompose the original BEV segmentation task into two stages,
+namely BEV map reconstruction and RGB-BEV feature alignment. In the first
+stage, we train a BEV autoencoder to reconstruct the BEV segmentation maps
+given corrupted noisy latent representation, which urges the decoder to learn
+fundamental knowledge of typical BEV patterns. The second stage involves
+mapping RGB input images into the BEV latent space of the first stage, directly
+optimizing the correlations between the two views at the feature level. Our
+approach simplifies the complexity of combining perception and generation into
+distinct steps, equipping the model to handle intricate and challenging scenes
+effectively. Besides, we propose to transform the BEV segmentation map from the
+Cartesian to the polar coordinate system to establish the column-wise
+correspondence between RGB images and BEV maps. Moreover, our method requires
+neither multi-scale features nor camera intrinsic parameters for depth
+estimation and saves computational overhead. Extensive experiments on nuScenes
+and Argoverse show the effectiveness and efficiency of our method. Code is
+available at https://github.com/happytianhao/TaDe.",cs.CV,"['cs.CV', 'cs.AI']"
+Optimal Transport Aggregation for Visual Place Recognition,Sergio Izquierdo · Javier Civera,https://serizba.github.io/salad.html,https://arxiv.org/abs/2311.15937,,2311.15937.pdf,Optimal Transport Aggregation for Visual Place Recognition,"The task of Visual Place Recognition (VPR) aims to match a query image
+against references from an extensive database of images from different places,
+relying solely on visual cues. State-of-the-art pipelines focus on the
+aggregation of features extracted from a deep backbone, in order to form a
+global descriptor for each image. In this context, we introduce SALAD (Sinkhorn
+Algorithm for Locally Aggregated Descriptors), which reformulates NetVLAD's
+soft-assignment of local features to clusters as an optimal transport problem.
+In SALAD, we consider both feature-to-cluster and cluster-to-feature relations
+and we also introduce a 'dustbin' cluster, designed to selectively discard
+features deemed non-informative, enhancing the overall descriptor quality.
+Additionally, we leverage and fine-tune DINOv2 as a backbone, which provides
+enhanced description power for the local features, and dramatically reduces the
+required training time. As a result, our single-stage method not only surpasses
+single-stage baselines in public VPR datasets, but also surpasses two-stage
+methods that add a re-ranking with significantly higher cost. Code and models
+are available at https://github.com/serizba/salad.",cs.CV,['cs.CV']
+DAP: A Dynamic Adversarial Patch for Evading Person Detectors,Amira Guesmi · Ruitian Ding · Muhammad Abdullah Hanif · Ihsen Alouani · Muhammad Shafique, ,,https://dblp.org/rec/journals/corr/abs-2305-11618,,,,,nan
+UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures,Mingyuan Zhou · Rakib Hyder · Ziwei Xuan · Guo-Jun Qi,https://usrc-sea.github.io/UltrAvatar/,https://arxiv.org/abs/2401.11078,,2401.11078.pdf,UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures,"Recent advances in 3D avatar generation have gained significant attentions.
+These breakthroughs aim to produce more realistic animatable avatars, narrowing
+the gap between virtual and real-world experiences. Most of existing works
+employ Score Distillation Sampling (SDS) loss, combined with a differentiable
+renderer and text condition, to guide a diffusion model in generating 3D
+avatars. However, SDS often generates oversmoothed results with few facial
+details, thereby lacking the diversity compared with ancestral sampling. On the
+other hand, other works generate 3D avatar from a single image, where the
+challenges of unwanted lighting effects, perspective views, and inferior image
+quality make them difficult to reliably reconstruct the 3D face meshes with the
+aligned complete textures. In this paper, we propose a novel 3D avatar
+generation approach termed UltrAvatar with enhanced fidelity of geometry, and
+superior quality of physically based rendering (PBR) textures without unwanted
+lighting. To this end, the proposed approach presents a diffuse color
+extraction model and an authenticity guided texture diffusion model. The former
+removes the unwanted lighting effects to reveal true diffuse colors so that the
+generated avatars can be rendered under various lighting conditions. The latter
+follows two gradient-based guidances for generating PBR textures to render
+diverse face-identity features and details better aligning with 3D mesh
+geometry. We demonstrate the effectiveness and robustness of the proposed
+method, outperforming the state-of-the-art methods by a large margin in the
+experiments.",cs.CV,['cs.CV']
+IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing,Shaofei Wang · Bozidar Antic · Andreas Geiger · Siyu Tang, ,https://arxiv.org/abs/2312.05210,,2312.05210.pdf,IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing,"We present IntrinsicAvatar, a novel approach to recovering the intrinsic
+properties of clothed human avatars including geometry, albedo, material, and
+environment lighting from only monocular videos. Recent advancements in
+human-based neural rendering have enabled high-quality geometry and appearance
+reconstruction of clothed humans from just monocular videos. However, these
+methods bake intrinsic properties such as albedo, material, and environment
+lighting into a single entangled neural representation. On the other hand, only
+a handful of works tackle the problem of estimating geometry and disentangled
+appearance properties of clothed humans from monocular videos. They usually
+achieve limited quality and disentanglement due to approximations of secondary
+shading effects via learned MLPs. In this work, we propose to model secondary
+shading effects explicitly via Monte-Carlo ray tracing. We model the rendering
+process of clothed humans as a volumetric scattering process, and combine ray
+tracing with body articulation. Our approach can recover high-quality geometry,
+albedo, material, and lighting properties of clothed humans from a single
+monocular video, without requiring supervised pre-training using ground truth
+materials. Furthermore, since we explicitly model the volumetric scattering
+process and ray tracing, our model naturally generalizes to novel poses,
+enabling animation of the reconstructed avatar in novel lighting conditions.",cs.CV,['cs.CV']
+A Call to Reflect on Evaluation Practices for Age Estimation: Comparative Analysis of the State-of-the-Art and a Unified Benchmark,Jakub Paplham · Vojtech Franc, ,https://arxiv.org/abs/2307.04570v2,,2307.04570v2.pdf,A Call to Reflect on Evaluation Practices for Age Estimation: Comparative Analysis of the State-of-the-Art and a Unified Benchmark,"Comparing different age estimation methods poses a challenge due to the
+unreliability of published results stemming from inconsistencies in the
+benchmarking process. Previous studies have reported continuous performance
+improvements over the past decade using specialized methods; however, our
+findings challenge these claims. This paper identifies two trivial, yet
+persistent issues with the currently used evaluation protocol and describes how
+to resolve them. We describe our evaluation protocol in detail and provide
+specific examples of how the protocol should be used. We utilize the protocol
+to offer an extensive comparative analysis for state-of-the-art facial age
+estimation methods. Surprisingly, we find that the performance differences
+between the methods are negligible compared to the effect of other factors,
+such as facial alignment, facial coverage, image resolution, model
+architecture, or the amount of data used for pretraining. We use the gained
+insights to propose using FaRL as the backbone model and demonstrate its
+efficiency. The results emphasize the importance of consistent data
+preprocessing practices for reliable and meaningful comparisons. We make our
+source code public at
+https://github.com/paplhjak/Facial-Age-Estimation-Benchmark.",cs.CV,"['cs.CV', 'cs.LG']"
+REACTO: Reconstructing Articulated Objects from a Single Video,Chaoyue Song · Jiacheng Wei · Chuan-Sheng Foo · Guosheng Lin · Fayao Liu,https://chaoyuesong.github.io/REACTO/,https://arxiv.org/abs/2404.11151,,2404.11151.pdf,REACTO: Reconstructing Articulated Objects from a Single Video,"In this paper, we address the challenge of reconstructing general articulated
+3D objects from a single video. Existing works employing dynamic neural
+radiance fields have advanced the modeling of articulated objects like humans
+and animals from videos, but face challenges with piece-wise rigid general
+articulated objects due to limitations in their deformation models. To tackle
+this, we propose Quasi-Rigid Blend Skinning, a novel deformation model that
+enhances the rigidity of each part while maintaining flexible deformation of
+the joints. Our primary insight combines three distinct approaches: 1) an
+enhanced bone rigging system for improved component modeling, 2) the use of
+quasi-sparse skinning weights to boost part rigidity and reconstruction
+fidelity, and 3) the application of geodesic point assignment for precise
+motion and seamless deformation. Our method outperforms previous works in
+producing higher-fidelity 3D reconstructions of general articulated objects, as
+demonstrated on both real and synthetic datasets. Project page:
+https://chaoyuesong.github.io/REACTO.",cs.CV,['cs.CV']
+DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation,Junming Chen · Yunfei Liu · Jianan Wang · Ailing Zeng · Yu Li · Qifeng Chen,https://jeremycjm.github.io/proj/DiffSHEG/,https://arxiv.org/abs/2401.04747,,2401.04747.pdf,DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation,"We propose DiffSHEG, a Diffusion-based approach for Speech-driven Holistic 3D
+Expression and Gesture generation with arbitrary length. While previous works
+focused on co-speech gesture or expression generation individually, the joint
+generation of synchronized expressions and gestures remains barely explored. To
+address this, our diffusion-based co-speech motion generation transformer
+enables uni-directional information flow from expression to gesture,
+facilitating improved matching of joint expression-gesture distributions.
+Furthermore, we introduce an outpainting-based sampling strategy for arbitrary
+long sequence generation in diffusion models, offering flexibility and
+computational efficiency. Our method provides a practical solution that
+produces high-quality synchronized expression and gesture generation driven by
+speech. Evaluated on two public datasets, our approach achieves
+state-of-the-art performance both quantitatively and qualitatively.
+Additionally, a user study confirms the superiority of DiffSHEG over prior
+approaches. By enabling the real-time generation of expressive and synchronized
+motions, DiffSHEG showcases its potential for various applications in the
+development of digital humans and embodied agents.",cs.SD,"['cs.SD', 'cs.AI', 'cs.CV', 'cs.GR', 'eess.AS']"
+Building Vision-Language Models on Solid Foundations with Masked Distillation,Sepehr Sameni · Kushal Kafle · Hao Tan · Simon Jenni, ,https://arxiv.org/abs/2311.03149,,2311.03149.pdf,Asymmetric Masked Distillation for Pre-Training Small Foundation Models,"Self-supervised foundation models have shown great potential in computer
+vision thanks to the pre-training paradigm of masked autoencoding. Scale is a
+primary factor influencing the performance of these foundation models. However,
+these large foundation models often result in high computational cost. This
+paper focuses on pre-training relatively small vision transformer models that
+could be efficiently adapted to downstream tasks. Specifically, taking
+inspiration from knowledge distillation in model compression, we propose a new
+asymmetric masked distillation (AMD) framework for pre-training relatively
+small models with autoencoding. The core of AMD is to devise an asymmetric
+masking strategy, where the teacher model is enabled to see more context
+information with a lower masking ratio, while the student model is still
+equipped with a high masking ratio. We design customized multi-layer feature
+alignment between the teacher encoder and student encoder to regularize the
+pre-training of student MAE. To demonstrate the effectiveness and versatility
+of AMD, we apply it to both ImageMAE and VideoMAE for pre-training relatively
+small ViT models. AMD achieved 84.6% classification accuracy on IN1K using the
+ViT-B model. And AMD achieves 73.3% classification accuracy using the ViT-B
+model on the Something-in-Something V2 dataset, a 3.7% improvement over the
+original ViT-B model from VideoMAE. We also transfer AMD pre-trained models to
+downstream tasks and obtain consistent performance improvement over the
+original masked autoencoding. The code and models are available at
+https://github.com/MCG-NJU/AMD.",cs.CV,['cs.CV']
+Small Steps and Level Sets: Fitting Neural Surface Models with Point Guidance,Chamin Hewa Koneputugodage · Yizhak Ben-Shabat · Dylan Campbell · Stephen Gould, ,http://export.arxiv.org/abs/2310.07997,,2310.07997.pdf,PG-NeuS: Robust and Efficient Point Guidance for Multi-View Neural Surface Reconstruction,"Recently, learning multi-view neural surface reconstruction with the
+supervision of point clouds or depth maps has been a promising way. However,
+due to the underutilization of prior information, current methods still
+struggle with the challenges of limited accuracy and excessive time complexity.
+In addition, prior data perturbation is also an important but rarely considered
+issue. To address these challenges, we propose a novel point-guided method
+named PG-NeuS, which achieves accurate and efficient reconstruction while
+robustly coping with point noise. Specifically, aleatoric uncertainty of the
+point cloud is modeled to capture the distribution of noise, leading to noise
+robustness. Furthermore, a Neural Projection module connecting points and
+images is proposed to add geometric constraints to implicit surface, achieving
+precise point guidance. To better compensate for geometric bias between volume
+rendering and point modeling, high-fidelity points are filtered into a Bias
+Network to further improve details representation. Benefiting from the
+effective point guidance, even with a lightweight network, the proposed PG-NeuS
+achieves fast convergence with an impressive 11x speedup compared to NeuS.
+Extensive experiments show that our method yields high-quality surfaces with
+high efficiency, especially for fine-grained details and smooth regions,
+outperforming the state-of-the-art methods. Moreover, it exhibits strong
+robustness to noisy data and sparse data.",cs.CV,"['cs.CV', 'cs.AI']"
+Self-correcting LLM-controlled Diffusion,Tsung-Han Wu · Long Lian · Joseph Gonzalez · Boyi Li · Trevor Darrell,https://self-correcting-llm-diffusion.github.io/,https://arxiv.org/abs/2311.16090,,2311.16090.pdf,Self-correcting LLM-controlled Diffusion Models,"Text-to-image generation has witnessed significant progress with the advent
+of diffusion models. Despite the ability to generate photorealistic images,
+current text-to-image diffusion models still often struggle to accurately
+interpret and follow complex input text prompts. In contrast to existing models
+that aim to generate images only with their best effort, we introduce
+Self-correcting LLM-controlled Diffusion (SLD). SLD is a framework that
+generates an image from the input prompt, assesses its alignment with the
+prompt, and performs self-corrections on the inaccuracies in the generated
+image. Steered by an LLM controller, SLD turns text-to-image generation into an
+iterative closed-loop process, ensuring correctness in the resulting image. SLD
+is not only training-free but can also be seamlessly integrated with diffusion
+models behind API access, such as DALL-E 3, to further boost the performance of
+state-of-the-art diffusion models. Experimental results show that our approach
+can rectify a majority of incorrect generations, particularly in generative
+numeracy, attribute binding, and spatial relationships. Furthermore, by simply
+adjusting the instructions to the LLM, SLD can perform image editing tasks,
+bridging the gap between text-to-image generation and image editing pipelines.
+We will make our code available for future research and applications.",cs.CV,['cs.CV']
+Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval,Yucheng Suo · Fan Ma · Linchao Zhu · Yi Yang, ,https://arxiv.org/abs/2403.16005,,2403.16005.pdf,Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval,"We study the zero-shot Composed Image Retrieval (ZS-CIR) task, which is to
+retrieve the target image given a reference image and a description without
+training on the triplet datasets. Previous works generate pseudo-word tokens by
+projecting the reference image features to the text embedding space. However,
+they focus on the global visual representation, ignoring the representation of
+detailed attributes, e.g., color, object number and layout. To address this
+challenge, we propose a Knowledge-Enhanced Dual-stream zero-shot composed image
+retrieval framework (KEDs). KEDs implicitly models the attributes of the
+reference images by incorporating a database. The database enriches the
+pseudo-word tokens by providing relevant images and captions, emphasizing
+shared attribute information in various aspects. In this way, KEDs recognizes
+the reference image from diverse perspectives. Moreover, KEDs adopts an extra
+stream that aligns pseudo-word tokens with textual concepts, leveraging
+pseudo-triplets mined from image-text pairs. The pseudo-word tokens generated
+in this stream are explicitly aligned with fine-grained semantics in the text
+embedding space. Extensive experiments on widely used benchmarks, i.e.
+ImageNet-R, COCO object, Fashion-IQ and CIRR, show that KEDs outperforms
+previous zero-shot composed image retrieval methods.",cs.CV,['cs.CV']
+MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training,Pavan Kumar Anasosalu Vasu · Hadi Pouransari · Fartash Faghri · Raviteja Vemulapalli · Oncel Tuzel,https://github.com/apple/ml-mobileclip,https://arxiv.org/abs/2311.17049,,2311.17049.pdf,MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training,"Contrastive pretraining of image-text foundation models, such as CLIP,
+demonstrated excellent zero-shot performance and improved robustness on a wide
+range of downstream tasks. However, these models utilize large
+transformer-based encoders with significant memory and latency overhead which
+pose challenges for deployment on mobile devices. In this work, we introduce
+MobileCLIP -- a new family of efficient image-text models optimized for runtime
+performance along with a novel and efficient training approach, namely
+multi-modal reinforced training. The proposed training approach leverages
+knowledge transfer from an image captioning model and an ensemble of strong
+CLIP encoders to improve the accuracy of efficient models. Our approach avoids
+train-time compute overhead by storing the additional knowledge in a reinforced
+dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for
+zero-shot classification and retrieval tasks on several datasets. Our
+MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to
+previous best CLIP model based on ViT-B/16. We further demonstrate the
+effectiveness of our multi-modal reinforced training by training a CLIP model
+based on ViT-B/16 image backbone and achieving +2.9% average performance
+improvement on 38 evaluation benchmarks compared to the previous best.
+Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$
+improved learning efficiency when compared with non-reinforced CLIP training.
+Code and models are available at https://github.com/apple/ml-mobileclip .",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Locally Adaptive Neural 3D Morphable Models,Michail Tarasiou · Rolandos Alexandros Potamias · Eimear O' Sullivan · Stylianos Ploumpis · Stefanos Zafeiriou, ,https://arxiv.org/abs/2401.02937,,2401.02937.pdf,Locally Adaptive Neural 3D Morphable Models,"We present the Locally Adaptive Morphable Model (LAMM), a highly flexible
+Auto-Encoder (AE) framework for learning to generate and manipulate 3D meshes.
+We train our architecture following a simple self-supervised training scheme in
+which input displacements over a set of sparse control vertices are used to
+overwrite the encoded geometry in order to transform one training sample into
+another. During inference, our model produces a dense output that adheres
+locally to the specified sparse geometry while maintaining the overall
+appearance of the encoded object. This approach results in state-of-the-art
+performance in both disentangling manipulated geometry and 3D mesh
+reconstruction. To the best of our knowledge LAMM is the first end-to-end
+framework that enables direct local control of 3D vertex geometry in a single
+forward pass. A very efficient computational graph allows our network to train
+with only a fraction of the memory required by previous methods and run faster
+during inference, generating 12k vertex meshes at $>$60fps on a single CPU
+thread. We further leverage local geometry control as a primitive for higher
+level editing operations and present a set of derivative capabilities such as
+swapping and sampling object parts. Code and pretrained models can be found at
+https://github.com/michaeltrs/LAMM.",cs.CV,['cs.CV']
+Finsler-Laplace-Beltrami Operators with Application to Shape Analysis,Simon Weber · Thomas Dagès · Maolin Gao · Daniel Cremers, ,https://arxiv.org/abs/2404.03999,,2404.03999.pdf,Finsler-Laplace-Beltrami Operators with Application to Shape Analysis,"The Laplace-Beltrami operator (LBO) emerges from studying manifolds equipped
+with a Riemannian metric. It is often called the Swiss army knife of geometry
+processing as it allows to capture intrinsic shape information and gives rise
+to heat diffusion, geodesic distances, and a multitude of shape descriptors. It
+also plays a central role in geometric deep learning. In this work, we explore
+Finsler manifolds as a generalization of Riemannian manifolds. We revisit the
+Finsler heat equation and derive a Finsler heat kernel and a
+Finsler-Laplace-Beltrami Operator (FLBO): a novel theoretically justified
+anisotropic Laplace-Beltrami operator (ALBO). In experimental evaluations we
+demonstrate that the proposed FLBO is a valuable alternative to the traditional
+Riemannian-based LBO and ALBOs for spatial filtering and shape correspondence
+estimation. We hope that the proposed Finsler heat kernel and the FLBO will
+inspire further exploration of Finsler geometry in the computer vision
+community.",cs.CV,['cs.CV']
+InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks,Zhe Chen · Jiannan Wu · Wenhai Wang · Weijie Su · Guo Chen · Sen Xing · Zhong Muyan · Qing-Long Zhang · Xizhou Zhu · Lewei Lu · Bin Li · Ping Luo · Tong Lu · Yu Qiao · Jifeng Dai, ,https://arxiv.org/abs/2312.14238,,2312.14238.pdf,InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks,"The exponential growth of large language models (LLMs) has opened up numerous
+possibilities for multimodal AGI systems. However, the progress in vision and
+vision-language foundation models, which are also critical elements of
+multi-modal AGI, has not kept pace with LLMs. In this work, we design a
+large-scale vision-language foundation model (InternVL), which scales up the
+vision foundation model to 6 billion parameters and progressively aligns it
+with the LLM, using web-scale image-text data from various sources. This model
+can be broadly applied to and achieve state-of-the-art performance on 32
+generic visual-linguistic benchmarks including visual perception tasks such as
+image-level or pixel-level recognition, vision-language tasks such as zero-shot
+image/video classification, zero-shot image/video-text retrieval, and link with
+LLMs to create multi-modal dialogue systems. It has powerful visual
+capabilities and can be a good alternative to the ViT-22B. We hope that our
+research could contribute to the development of multi-modal large models. Code
+and models are available at https://github.com/OpenGVLab/InternVL.",cs.CV,['cs.CV']
+LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry,Weirong Chen · Le Chen · Rui Wang · Marc Pollefeys,https://chiaki530.github.io/projects/leapvo/,https://arxiv.org/abs/2401.01887v1,,2401.01887v1.pdf,LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry,"Visual odometry estimates the motion of a moving camera based on visual
+input. Existing methods, mostly focusing on two-view point tracking, often
+ignore the rich temporal context in the image sequence, thereby overlooking the
+global motion patterns and providing no assessment of the full trajectory
+reliability. These shortcomings hinder performance in scenarios with occlusion,
+dynamic objects, and low-texture areas. To address these challenges, we present
+the Long-term Effective Any Point Tracking (LEAP) module. LEAP innovatively
+combines visual, inter-track, and temporal cues with mindfully selected anchors
+for dynamic track estimation. Moreover, LEAP's temporal probabilistic
+formulation integrates distribution updates into a learnable iterative
+refinement module to reason about point-wise uncertainty. Based on these
+traits, we develop LEAP-VO, a robust visual odometry system adept at handling
+occlusions and dynamic scenes. Our mindful integration showcases a novel
+practice by employing long-term point tracking as the front-end. Extensive
+experiments demonstrate that the proposed pipeline significantly outperforms
+existing baselines across various visual odometry benchmarks.",cs.CV,['cs.CV']
+MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding,Bo He · Hengduo Li · Young Kyun Jang · Menglin Jia · Xuefei Cao · Ashish Shah · Abhinav Shrivastava · Ser-Nam Lim, ,https://arxiv.org/html/2404.05726v2,,2404.05726v2.pdf,MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding,"With the success of large language models (LLMs), integrating the vision
+model into LLMs to build vision-language foundation models has gained much more
+interest recently. However, existing LLM-based large multimodal models (e.g.,
+Video-LLaMA, VideoChat) can only take in a limited number of frames for short
+video understanding. In this study, we mainly focus on designing an efficient
+and effective model for long-term video understanding. Instead of trying to
+process more frames simultaneously like most existing work, we propose to
+process videos in an online manner and store past video information in a memory
+bank. This allows our model to reference historical video content for long-term
+analysis without exceeding LLMs' context length constraints or GPU memory
+limits. Our memory bank can be seamlessly integrated into current multimodal
+LLMs in an off-the-shelf manner. We conduct extensive experiments on various
+video understanding tasks, such as long-video understanding, video question
+answering, and video captioning, and our model can achieve state-of-the-art
+performances across multiple datasets. Code available at
+https://boheumd.github.io/MA-LMM/.",cs.CV,['cs.CV']
+Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption,Buzhen Huang · Chen Li · Chongyang Xu · Liang Pan · Yangang Wang · Gim Hee Lee, ,https://arxiv.org/abs/2404.11291,,2404.11291.pdf,Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption,"Existing multi-person human reconstruction approaches mainly focus on
+recovering accurate poses or avoiding penetration, but overlook the modeling of
+close interactions. In this work, we tackle the task of reconstructing closely
+interactive humans from a monocular video. The main challenge of this task
+comes from insufficient visual information caused by depth ambiguity and severe
+inter-person occlusion. In view of this, we propose to leverage knowledge from
+proxemic behavior and physics to compensate the lack of visual information.
+This is based on the observation that human interaction has specific patterns
+following the social proxemics. Specifically, we first design a latent
+representation based on Vector Quantised-Variational AutoEncoder (VQ-VAE) to
+model human interaction. A proxemics and physics guided diffusion model is then
+introduced to denoise the initial distribution. We design the diffusion model
+as dual branch with each branch representing one individual such that the
+interaction can be modeled via cross attention. With the learned priors of
+VQ-VAE and physical constraint as the additional information, our proposed
+approach is capable of estimating accurate poses that are also proxemics and
+physics plausible. Experimental results on Hi4D, 3DPW, and CHI3D demonstrate
+that our method outperforms existing approaches. The code is available at
+\url{https://github.com/boycehbz/HumanInteraction}.",cs.CV,['cs.CV']
+The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement,Gabriele Trivigno · Carlo Masone · Barbara Caputo · Torsten Sattler, ,https://arxiv.org/abs/2404.10438v1,,2404.10438v1.pdf,The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement,"Pose refinement is an interesting and practically relevant research
+direction. Pose refinement can be used to (1) obtain a more accurate pose
+estimate from an initial prior (e.g., from retrieval), (2) as pre-processing,
+i.e., to provide a better starting point to a more expensive pose estimator,
+(3) as post-processing of a more accurate localizer. Existing approaches focus
+on learning features / scene representations for the pose refinement task. This
+involves training an implicit scene representation or learning features while
+optimizing a camera pose-based loss. A natural question is whether training
+specific features / representations is truly necessary or whether similar
+results can be already achieved with more generic features. In this work, we
+present a simple approach that combines pre-trained features with a particle
+filter and a renderable representation of the scene. Despite its simplicity, it
+achieves state-of-the-art results, demonstrating that one can easily build a
+pose refiner without the need for specific training. The code is at
+https://github.com/ga1i13o/mcloc_poseref",cs.CV,['cs.CV']
+GDA: Generalized Diffusion for Robust Test-time Adaptation,Yun-Yun Tsai · Fu-Chen Chen · Albert Chen · Junfeng Yang · Che-Chun Su · Min Sun · Cheng-Hao Kuo, ,https://arxiv.org/abs/2404.00095,,2404.00095.pdf,GDA: Generalized Diffusion for Robust Test-time Adaptation,"Machine learning models struggle with generalization when encountering
+out-of-distribution (OOD) samples with unexpected distribution shifts. For
+vision tasks, recent studies have shown that test-time adaptation employing
+diffusion models can achieve state-of-the-art accuracy improvements on OOD
+samples by generating new samples that align with the model's domain without
+the need to modify the model's weights. Unfortunately, those studies have
+primarily focused on pixel-level corruptions, thereby lacking the
+generalization to adapt to a broader range of OOD types. We introduce
+Generalized Diffusion Adaptation (GDA), a novel diffusion-based test-time
+adaptation method robust against diverse OOD types. Specifically, GDA
+iteratively guides the diffusion by applying a marginal entropy loss derived
+from the model, in conjunction with style and content preservation losses
+during the reverse sampling process. In other words, GDA considers the model's
+output behavior with the semantic information of the samples as a whole, which
+can reduce ambiguity in downstream tasks during the generation process.
+Evaluation across various popular model architectures and OOD benchmarks shows
+that GDA consistently outperforms prior work on diffusion-driven adaptation.
+Notably, it achieves the highest classification accuracy improvements, ranging
+from 4.4\% to 5.02\% on ImageNet-C and 2.5\% to 7.4\% on Rendition, Sketch, and
+Stylized benchmarks. This performance highlights GDA's generalization to a
+broader range of OOD benchmarks.",cs.CV,['cs.CV']
+RecDiffusion: Rectangling for Image Stitching with Diffusion Models,Tianhao Zhou · Li Haipeng · Ziyi Wang · Ao Luo · Chenlin Zhang · Jiajun Li · Bing Zeng · Shuaicheng Liu, ,https://arxiv.org/abs/2403.19164,,2403.19164.pdf,RecDiffusion: Rectangling for Image Stitching with Diffusion Models,"Image stitching from different captures often results in non-rectangular
+boundaries, which is often considered unappealing. To solve non-rectangular
+boundaries, current solutions involve cropping, which discards image content,
+inpainting, which can introduce unrelated content, or warping, which can
+distort non-linear features and introduce artifacts. To overcome these issues,
+we introduce a novel diffusion-based learning framework, \textbf{RecDiffusion},
+for image stitching rectangling. This framework combines Motion Diffusion
+Models (MDM) to generate motion fields, effectively transitioning from the
+stitched image's irregular borders to a geometrically corrected intermediary.
+Followed by Content Diffusion Models (CDM) for image detail refinement.
+Notably, our sampling process utilizes a weighted map to identify regions
+needing correction during each iteration of CDM. Our RecDiffusion ensures
+geometric accuracy and overall visual appeal, surpassing all previous methods
+in both quantitative and qualitative measures when evaluated on public
+benchmarks. Code is released at https://github.com/lhaippp/RecDiffusion.",cs.CV,['cs.CV']
+"Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts",Qin Liu · Jaemin Cho · Mohit Bansal · Marc Niethammer,https://github.com/uncbiag/SegNext,https://arxiv.org/abs/2404.00741,,2404.00741.pdf,"Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts","The goal of interactive image segmentation is to delineate specific regions
+within an image via visual or language prompts. Low-latency and high-quality
+interactive segmentation with diverse prompts remain challenging for existing
+specialist and generalist models. Specialist models, with their limited prompts
+and task-specific designs, experience high latency because the image must be
+recomputed every time the prompt is updated, due to the joint encoding of image
+and visual prompts. Generalist models, exemplified by the Segment Anything
+Model (SAM), have recently excelled in prompt diversity and efficiency, lifting
+image segmentation to the foundation model era. However, for high-quality
+segmentations, SAM still lags behind state-of-the-art specialist models despite
+SAM being trained with x100 more segmentation masks. In this work, we delve
+deep into the architectural differences between the two types of models. We
+observe that dense representation and fusion of visual prompts are the key
+design choices contributing to the high segmentation quality of specialist
+models. In light of this, we reintroduce this dense design into the generalist
+models, to facilitate the development of generalist models with high
+segmentation quality. To densely represent diverse visual prompts, we propose
+to use a dense map to capture five types: clicks, boxes, polygons, scribbles,
+and masks. Thus, we propose SegNext, a next-generation interactive segmentation
+approach offering low latency, high quality, and diverse prompt support. Our
+method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS,
+both quantitatively and qualitatively.",cs.CV,['cs.CV']
+Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior,Fangfu Liu · Diankun Wu · Yi Wei · Yongming Rao · Yueqi Duan,https://liuff19.github.io/Sherpa3D/,https://arxiv.org/abs/2312.06655,,2312.06655.pdf,Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior,"Recently, 3D content creation from text prompts has demonstrated remarkable
+progress by utilizing 2D and 3D diffusion models. While 3D diffusion models
+ensure great multi-view consistency, their ability to generate high-quality and
+diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion
+models find a distillation approach that achieves excellent generalization and
+rich details without any 3D data. However, 2D lifting methods suffer from
+inherent view-agnostic ambiguity thereby leading to serious multi-face Janus
+issues, where text prompts fail to provide sufficient guidance to learn
+coherent 3D results. Instead of retraining a costly viewpoint-aware model, we
+study how to fully exploit easily accessible coarse 3D knowledge to enhance the
+prompts and guide 2D lifting optimization for refinement. In this paper, we
+propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity,
+generalizability, and geometric consistency simultaneously. Specifically, we
+design a pair of guiding strategies derived from the coarse 3D prior generated
+by the 3D diffusion model: a structural guidance for geometric fidelity and a
+semantic guidance for 3D coherence. Employing the two types of guidance, the 2D
+diffusion model enriches the 3D content with diversified and high-quality
+results. Extensive experiments show the superiority of our Sherpa3D over the
+state-of-the-art text-to-3D methods in terms of quality and 3D consistency.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Identifying Important Group of Pixels using Interactions,Kosuke Sumiyasu · Kazuhiko Kawamoto · Hiroshi Kera, ,https://arxiv.org/abs/2401.03785,,2401.03785.pdf,Identifying Important Group of Pixels using Interactions,"To better understand the behavior of image classifiers, it is useful to
+visualize the contribution of individual pixels to the model prediction. In
+this study, we propose a method, MoXI ($\textbf{Mo}$del e$\textbf{X}$planation
+by $\textbf{I}$nteractions), that efficiently and accurately identifies a group
+of pixels with high prediction confidence. The proposed method employs
+game-theoretic concepts, Shapley values and interactions, taking into account
+the effects of individual pixels and the cooperative influence of pixels on
+model confidence. Theoretical analysis and experiments demonstrate that our
+method better identifies the pixels that are highly contributing to the model
+outputs than widely-used visualization by Grad-CAM, Attention rollout, and
+Shapley value. While prior studies have suffered from the exponential
+computational cost in the computation of Shapley value and interactions, we
+show that this can be reduced to quadratic cost for our task. The code is
+available at https://github.com/KosukeSumiyasu/MoXI.",cs.CV,"['cs.CV', 'cs.LG']"
+DualAD: Disentangling the Dynamic and Static World for End-to-End Driving,Simon Doll · Niklas Hanselmann · Lukas Schneider · Richard Schulz · Marius Cordts · Markus Enzweiler · Hendrik Lensch, ,https://arxiv.org/html/2306.16927v2,,2306.16927v2.pdf,End-to-end Autonomous Driving: Challenges and Frontiers,"The autonomous driving community has witnessed a rapid growth in approaches
+that embrace an end-to-end algorithm framework, utilizing raw sensor input to
+generate vehicle motion plans, instead of concentrating on individual tasks
+such as detection and motion prediction. End-to-end systems, in comparison to
+modular pipelines, benefit from joint feature optimization for perception and
+planning. This field has flourished due to the availability of large-scale
+datasets, closed-loop evaluation, and the increasing need for autonomous
+driving algorithms to perform effectively in challenging scenarios. In this
+survey, we provide a comprehensive analysis of more than 270 papers, covering
+the motivation, roadmap, methodology, challenges, and future trends in
+end-to-end autonomous driving. We delve into several critical challenges,
+including multi-modality, interpretability, causal confusion, robustness, and
+world models, amongst others. Additionally, we discuss current advancements in
+foundation models and visual pre-training, as well as how to incorporate these
+techniques within the end-to-end driving framework. we maintain an active
+repository that contains up-to-date literature and open-source projects at
+https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CV', 'cs.LG']"
+A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing,Li Maomao · Yu Li · Tianyu Yang · Yunfei Liu · Dongxu Yue · Zhihui Lin · Dong Xu, ,https://arxiv.org/abs/2312.05856,,2312.05856.pdf,A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing,"This paper presents a video inversion approach for zero-shot video editing,
+which models the input video with low-rank representation during the inversion
+process. The existing video editing methods usually apply the typical 2D DDIM
+inversion or naive spatial-temporal DDIM inversion before editing, which
+leverages time-varying representation for each frame to derive noisy latent.
+Unlike most existing approaches, we propose a Spatial-Temporal
+Expectation-Maximization (STEM) inversion, which formulates the dense video
+feature under an expectation-maximization manner and iteratively estimates a
+more compact basis set to represent the whole video. Each frame applies the
+fixed and global representation for inversion, which is more friendly for
+temporal consistency during reconstruction and editing. Extensive qualitative
+and quantitative experiments demonstrate that our STEM inversion can achieve
+consistent improvement on two state-of-the-art video editing methods. Project
+page: https://stem-inv.github.io/page/.",cs.CV,['cs.CV']
+"Time-, Memory- and Parameter-Efficient Visual Adaptation",Otniel-Bogdan Mercea · Alexey Gritsenko · Cordelia Schmid · Anurag Arnab, ,https://arxiv.org/abs/2402.02887,,2402.02887.pdf,"Time-, Memory- and Parameter-Efficient Visual Adaptation","As foundation models become more popular, there is a growing need to
+efficiently finetune them for downstream tasks. Although numerous adaptation
+methods have been proposed, they are designed to be efficient only in terms of
+how many parameters are trained. They, however, typically still require
+backpropagating gradients throughout the model, meaning that their
+training-time and -memory cost does not reduce as significantly. We propose an
+adaptation method which does not backpropagate gradients through the backbone.
+We achieve this by designing a lightweight network in parallel that operates on
+features from the frozen, pretrained backbone. As a result, our method is
+efficient not only in terms of parameters, but also in training-time and memory
+usage. Our approach achieves state-of-the-art accuracy-parameter trade-offs on
+the popular VTAB benchmark, and we further show how we outperform prior works
+with respect to training-time and -memory usage too. We further demonstrate the
+training efficiency and scalability of our method by adapting a vision
+transformer backbone of 4 billion parameters for the computationally demanding
+task of video classification, without any intricate model parallelism. Here, we
+outperform a prior adaptor-based method which could only scale to a 1 billion
+parameter backbone, or fully-finetuning a smaller backbone, with the same GPU
+and less training time.",cs.CV,"['cs.CV', 'cs.LG']"
+WildlifeMapper:  Aerial Image Analysis for Multi-Species Detection and Identification,Satish Kumar · Bowen Zhang · Chandrakanth Gudavalli · Connor Levenson · Lacey Hughey · Jared Stabach · Irene Amoke · Gordon Ojwang · Joseph Mukeka · Howard Frederick · Stephen Mwiu · Joseph Ochieng Ogutu · B S Manjunath, ,https://arxiv.org/abs/2311.12956,,2311.12956.pdf,Innovative Horizons in Aerial Imagery: LSKNet Meets DiffusionDet for Advanced Object Detection,"In the realm of aerial image analysis, object detection plays a pivotal role,
+with significant implications for areas such as remote sensing, urban planning,
+and disaster management. This study addresses the inherent challenges in this
+domain, notably the detection of small objects, managing densely packed
+elements, and accounting for diverse orientations. We present an in-depth
+evaluation of an object detection model that integrates the Large Selective
+Kernel Network (LSKNet)as its backbone with the DiffusionDet head, utilizing
+the iSAID dataset for empirical analysis. Our approach encompasses the
+introduction of novel methodologies and extensive ablation studies. These
+studies critically assess various aspects such as loss functions, box
+regression techniques, and classification strategies to refine the model's
+precision in object detection. The paper details the experimental application
+of the LSKNet backbone in synergy with the DiffusionDet heads, a combination
+tailored to meet the specific challenges in aerial image object detection. The
+findings of this research indicate a substantial enhancement in the model's
+performance, especially in the accuracy-time tradeoff. The proposed model
+achieves a mean average precision (MAP) of approximately 45.7%, which is a
+significant improvement, outperforming the RCNN model by 4.7% on the same
+dataset. This advancement underscores the effectiveness of the proposed
+modifications and sets a new benchmark in aerial image analysis, paving the way
+for more accurate and efficient object detection methodologies. The code is
+publicly available at https://github.com/SashaMatsun/LSKDiffDet",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincaré Ball,Simon Weber · Barış Zöngür · Nikita Araslanov · Daniel Cremers, ,https://arxiv.org/abs/2404.03778,,2404.03778.pdf,Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincaré Ball,"Hierarchy is a natural representation of semantic taxonomies, including the
+ones routinely used in image segmentation. Indeed, recent work on semantic
+segmentation reports improved accuracy from supervised training leveraging
+hierarchical label structures. Encouraged by these results, we revisit the
+fundamental assumptions behind that work. We postulate and then empirically
+verify that the reasons for the observed improvement in segmentation accuracy
+may be entirely unrelated to the use of the semantic hierarchy. To demonstrate
+this, we design a range of cross-domain experiments with a representative
+hierarchical approach. We find that on the new testing domains, a flat
+(non-hierarchical) segmentation network, in which the parents are inferred from
+the children, has superior segmentation accuracy to the hierarchical approach
+across the board. Complementing these findings and inspired by the intrinsic
+properties of hyperbolic spaces, we study a more principled approach to
+hierarchical segmentation using the Poincar\'e ball model. The hyperbolic
+representation largely outperforms the previous (Euclidean) hierarchical
+approach as well and is on par with our flat Euclidean baseline in terms of
+segmentation accuracy. However, it additionally exhibits surprisingly strong
+calibration quality of the parent nodes in the semantic hierarchy, especially
+on the more challenging domains. Our combined analysis suggests that the
+established practice of hierarchical segmentation may be limited to in-domain
+settings, whereas flat classifiers generalize substantially better, especially
+if they are modeled in the hyperbolic space.",cs.CV,['cs.CV']
+OTE: Exploring Accurate Scene Text Recognition Using One Token,Jianjun Xu · Yuxin Wang · Hongtao Xie · Yongdong Zhang, ,https://arxiv.org/html/2403.07518v1,,2403.07518v1.pdf,Open-Vocabulary Scene Text Recognition via Pseudo-Image Labeling and Margin Loss,"Scene text recognition is an important and challenging task in computer
+vision. However, most prior works focus on recognizing pre-defined words, while
+there are various out-of-vocabulary (OOV) words in real-world applications.
+  In this paper, we propose a novel open-vocabulary text recognition framework,
+Pseudo-OCR, to recognize OOV words. The key challenge in this task is the lack
+of OOV training data. To solve this problem, we first propose a pseudo label
+generation module that leverages character detection and image inpainting to
+produce substantial pseudo OOV training data from real-world images. Unlike
+previous synthetic data, our pseudo OOV data contains real characters and
+backgrounds to simulate real-world applications. Secondly, to reduce noises in
+pseudo data, we present a semantic checking mechanism to filter semantically
+meaningful data. Thirdly, we introduce a quality-aware margin loss to boost the
+training with pseudo data. Our loss includes a margin-based part to enhance the
+classification ability, and a quality-aware part to penalize low-quality
+samples in both real and pseudo data.
+  Extensive experiments demonstrate that our approach outperforms the
+state-of-the-art on eight datasets and achieves the first rank in the ICDAR2022
+challenge.",cs.CV,['cs.CV']
+Language-Driven Anchors for Zero-Shot Adversarial Robustness,Xiao Li · Wei Zhang · Yining Liu · Zhanhao Hu · Bo Zhang · Xiaolin Hu,https://github.com/LixiaoTHU/LAAT,,https://paperswithcode.com/search?q=author:Xiaolin+Hu&order_by=stars,,,,,nan
+DAVE -- A Detect-and-Verify Paradigm for Low-Shot Counting,Jer Pelhan · Alan Lukezic · Vitjan Zavrtanik · Matej Kristan, ,https://arxiv.org/abs/2404.16622,,2404.16622.pdf,DAVE -- A Detect-and-Verify Paradigm for Low-Shot Counting,"Low-shot counters estimate the number of objects corresponding to a selected
+category, based on only few or no exemplars annotated in the image. The current
+state-of-the-art estimates the total counts as the sum over the object location
+density map, but does not provide individual object locations and sizes, which
+are crucial for many applications. This is addressed by detection-based
+counters, which, however fall behind in the total count accuracy. Furthermore,
+both approaches tend to overestimate the counts in the presence of other object
+classes due to many false positives. We propose DAVE, a low-shot counter based
+on a detect-and-verify paradigm, that avoids the aforementioned issues by first
+generating a high-recall detection set and then verifying the detections to
+identify and remove the outliers. This jointly increases the recall and
+precision, leading to accurate counts. DAVE outperforms the top density-based
+counters by ~20% in the total count MAE, it outperforms the most recent
+detection-based counter by ~20% in detection quality and sets a new
+state-of-the-art in zero-shot as well as text-prompt-based counting.",cs.CV,['cs.CV']
+SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes,Soubhik Sanyal · Partha Ghosh · Jinlong Yang · Michael J. Black · Justus Thies · Timo Bolkart,https://sculpt.is.tue.mpg.de/,https://arxiv.org/html/2308.10638v2,,2308.10638v2.pdf,SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes,"We present SCULPT, a novel 3D generative model for clothed and textured 3D
+meshes of humans. Specifically, we devise a deep neural network that learns to
+represent the geometry and appearance distribution of clothed human bodies.
+Training such a model is challenging, as datasets of textured 3D meshes for
+humans are limited in size and accessibility. Our key observation is that there
+exist medium-sized 3D scan datasets like CAPE, as well as large-scale 2D image
+datasets of clothed humans and multiple appearances can be mapped to a single
+geometry. To effectively learn from the two data modalities, we propose an
+unpaired learning procedure for pose-dependent clothed and textured human
+meshes. Specifically, we learn a pose-dependent geometry space from 3D scan
+data. We represent this as per vertex displacements w.r.t. the SMPL model.
+Next, we train a geometry conditioned texture generator in an unsupervised way
+using the 2D image data. We use intermediate activations of the learned
+geometry model to condition our texture generator. To alleviate entanglement
+between pose and clothing type, and pose and clothing appearance, we condition
+both the texture and geometry generators with attribute labels such as clothing
+types for the geometry, and clothing colors for the texture generator. We
+automatically generated these conditioning labels for the 2D images based on
+the visual question answering model BLIP and CLIP. We validate our method on
+the SCULPT dataset, and compare to state-of-the-art 3D generative models for
+clothed human bodies. Our code and data can be found at
+https://sculpt.is.tue.mpg.de.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+Brush2Prompt: Contextual Prompt Generator for Object Inpainting,Mang Tik Chiu · Yuqian Zhou · Lingzhi Zhang · Zhe Lin · Connelly Barnes · Sohrab Amirghodsi · Eli Shechtman · Humphrey Shi, ,https://ar5iv.labs.arxiv.org/html/2204.07845,,2204.07845.pdf,Shape-guided Object Inpainting,"Previous works on image inpainting mainly focus on inpainting background or
+partially missing objects, while the problem of inpainting an entire missing
+object remains unexplored. This work studies a new image inpainting task, i.e.
+shape-guided object inpainting. Given an incomplete input image, the goal is to
+fill in the hole by generating an object based on the context and implicit
+guidance given by the hole shape. Since previous methods for image inpainting
+are mainly designed for background inpainting, they are not suitable for this
+task. Therefore, we propose a new data preparation method and a novel
+Contextual Object Generator (CogNet) for the object inpainting task. On the
+data side, we incorporate object priors into training data by using object
+instances as holes. The CogNet has a two-stream architecture that combines the
+standard bottom-up image completion process with a top-down object generation
+process. A predictive class embedding module bridges the two streams by
+predicting the class of the missing object from the bottom-up features, from
+which a semantic object map is derived as the input of the top-down stream.
+Experiments demonstrate that the proposed method can generate realistic objects
+that fit the context in terms of both visual appearance and semantic meanings.
+Code can be found at the project page
+\url{https://zengxianyu.github.io/objpaint}",cs.CV,"['cs.CV', 'cs.MM']"
+AV-RIR: Audio-Visual Room Impulse Response Estimation,Anton Ratnarajah · Sreyan Ghosh · Sonal Kumar · Purva Chiniya · Dinesh Manocha,https://anton-jeran.github.io/AVRIR/,https://arxiv.org/abs/2312.00834,,2312.00834.pdf,AV-RIR: Audio-Visual Room Impulse Response Estimation,"Accurate estimation of Room Impulse Response (RIR), which captures an
+environment's acoustic properties, is important for speech processing and AR/VR
+applications. We propose AV-RIR, a novel multi-modal multi-task learning
+approach to accurately estimate the RIR from a given reverberant speech signal
+and the visual cues of its corresponding environment. AV-RIR builds on a novel
+neural codec-based architecture that effectively captures environment geometry
+and materials properties and solves speech dereverberation as an auxiliary task
+by using multi-task learning. We also propose Geo-Mat features that augment
+material information into visual cues and CRIP that improves late reverberation
+components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical
+results show that AV-RIR quantitatively outperforms previous audio-only and
+visual-only approaches by achieving 36% - 63% improvement across various
+acoustic metrics in RIR estimation. Additionally, it also achieves higher
+preference scores in human evaluation. As an auxiliary benefit, dereverbed
+speech from AV-RIR shows competitive performance with the state-of-the-art in
+various spoken language processing tasks and outperforms reverberation time
+error score in the real-world AVSpeech dataset. Qualitative examples of both
+synthesized reverberant speech and enhanced speech can be found at
+https://www.youtube.com/watch?v=tTsKhviukAE.",cs.SD,"['cs.SD', 'cs.CV']"
+DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization,Jisu Nam · Heesu Kim · DongJae Lee · Siyoon Jin · Seungryong Kim · Seunggyu Chang, ,https://arxiv.org/abs/2402.09812,,2402.09812.pdf,DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization,"The objective of text-to-image (T2I) personalization is to customize a
+diffusion model to a user-provided reference concept, generating diverse images
+of the concept aligned with the target prompts. Conventional methods
+representing the reference concepts using unique text embeddings often fail to
+accurately mimic the appearance of the reference. To address this, one solution
+may be explicitly conditioning the reference images into the target denoising
+process, known as key-value replacement. However, prior works are constrained
+to local editing since they disrupt the structure path of the pre-trained T2I
+model. To overcome this, we propose a novel plug-in method, called
+DreamMatcher, which reformulates T2I personalization as semantic matching.
+Specifically, DreamMatcher replaces the target values with reference values
+aligned by semantic matching, while leaving the structure path unchanged to
+preserve the versatile capability of pre-trained T2I models for generating
+diverse structures. We also introduce a semantic-consistent masking strategy to
+isolate the personalized concept from irrelevant regions introduced by the
+target prompts. Compatible with existing T2I models, DreamMatcher shows
+significant improvements in complex scenarios. Intensive analyses demonstrate
+the effectiveness of our approach.",cs.CV,['cs.CV']
+"FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features",Andre Rochow · Max Schwarz · Sven Behnke,https://andrerochow.github.io/fsrt,https://arxiv.org/abs/2404.09736,,2404.09736.pdf,"FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features","The task of face reenactment is to transfer the head motion and facial
+expressions from a driving video to the appearance of a source image, which may
+be of a different person (cross-reenactment). Most existing methods are
+CNN-based and estimate optical flow from the source image to the current
+driving frame, which is then inpainted and refined to produce the output
+animation. We propose a transformer-based encoder for computing a set-latent
+representation of the source image(s). We then predict the output color of a
+query pixel using a transformer-based decoder, which is conditioned with
+keypoints and a facial expression vector extracted from the driving frame.
+Latent representations of the source person are learned in a self-supervised
+manner that factorize their appearance, head pose, and facial expressions.
+Thus, they are perfectly suited for cross-reenactment. In contrast to most
+related work, our method naturally extends to multiple source images and can
+thus adapt to person-specific facial dynamics. We also propose data
+augmentation and regularization schemes that are necessary to prevent
+overfitting and support generalizability of the learned representations. We
+evaluated our approach in a randomized user study. The results indicate
+superior performance compared to the state-of-the-art in terms of motion
+transfer quality and temporal consistency.",cs.CV,['cs.CV']
+Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models,Hongjie Wang · Difan Liu · Yan Kang · Yijun Li · Zhe Lin · Niraj Jha · Yuchen Liu,https://atedm.github.io/,https://arxiv.org/abs/2405.05252,,2405.05252.pdf,Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models,"Diffusion Models (DMs) have exhibited superior performance in generating
+high-quality and diverse images. However, this exceptional performance comes at
+the cost of expensive architectural design, particularly due to the attention
+module heavily used in leading models. Existing works mainly adopt a retraining
+process to enhance DM efficiency. This is computationally expensive and not
+very scalable. To this end, we introduce the Attention-driven Training-free
+Efficient Diffusion Model (AT-EDM) framework that leverages attention maps to
+perform run-time pruning of redundant tokens, without the need for any
+retraining. Specifically, for single-denoising-step pruning, we develop a novel
+ranking algorithm, Generalized Weighted Page Rank (G-WPR), to identify
+redundant tokens, and a similarity-based recovery method to restore tokens for
+the convolution operation. In addition, we propose a Denoising-Steps-Aware
+Pruning (DSAP) approach to adjust the pruning budget across different denoising
+timesteps for better generation quality. Extensive evaluations show that AT-EDM
+performs favorably against prior art in terms of efficiency (e.g., 38.8% FLOPs
+saving and up to 1.53x speed-up over Stable Diffusion XL) while maintaining
+nearly the same FID and CLIP scores as the full model. Project webpage:
+https://atedm.github.io.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'eess.IV', 'eess.SP']"
+Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis,Atefeh Khoshkhahtinat · Ali Zafari · Piyush Mehta · Nasser Nasrabadi, ,https://arxiv.org/abs/2403.16258,,2403.16258.pdf,Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis,"While replacing Gaussian decoders with a conditional diffusion model enhances
+the perceptual quality of reconstructions in neural image compression, their
+lack of inductive bias for image data restricts their ability to achieve
+state-of-the-art perceptual levels. To address this limitation, we adopt a
+non-isotropic diffusion model at the decoder side. This model imposes an
+inductive bias aimed at distinguishing between frequency contents, thereby
+facilitating the generation of high-quality images. Moreover, our framework is
+equipped with a novel entropy model that accurately models the probability
+distribution of latent representation by exploiting spatio-channel correlations
+in latent space, while accelerating the entropy decoding step. This
+channel-wise entropy model leverages both local and global spatial contexts
+within each channel chunk. The global spatial context is built upon the
+Transformer, which is specifically designed for image compression tasks. The
+designed Transformer employs a Laplacian-shaped positional encoding, the
+learnable parameters of which are adaptively adjusted for each channel cluster.
+Our experiments demonstrate that our proposed framework yields better
+perceptual quality compared to cutting-edge generative-based codecs, and the
+proposed entropy model contributes to notable bitrate savings.",eess.IV,"['eess.IV', 'cs.CV', 'cs.IT', 'cs.LG', 'math.IT']"
+Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models,Pengze Zhang · Hubery Yin · Chen Li · Xiaohua Xie,https://pangzecheung.github.io/SingDiffusion/,https://arxiv.org/abs/2403.08381,,2403.08381.pdf,Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models,"Most diffusion models assume that the reverse process adheres to a Gaussian
+distribution. However, this approximation has not been rigorously validated,
+especially at singularities, where t=0 and t=1. Improperly dealing with such
+singularities leads to an average brightness issue in applications, and limits
+the generation of images with extreme brightness or darkness. We primarily
+focus on tackling singularities from both theoretical and practical
+perspectives. Initially, we establish the error bounds for the reverse process
+approximation, and showcase its Gaussian characteristics at singularity time
+steps. Based on this theoretical insight, we confirm the singularity at t=1 is
+conditionally removable while it at t=0 is an inherent property. Upon these
+significant conclusions, we propose a novel plug-and-play method SingDiffusion
+to address the initial singular time step sampling, which not only effectively
+resolves the average brightness issue for a wide range of diffusion models
+without extra training efforts, but also enhances their generation capability
+in achieving notable lower FID scores.",cs.CV,['cs.CV']
+"Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning",Nikhil Singh · Chih-Wei Wu · Iroro Orife · Kalayeh, ,https://arxiv.org/abs/2404.17753,,,Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification,"CLIP showcases exceptional cross-modal matching capabilities due to its
+training on image-text contrastive learning tasks. However, without specific
+optimization for unimodal scenarios, its performance in single-modality feature
+extraction might be suboptimal. Despite this, some studies have directly used
+CLIP's image encoder for tasks like few-shot classification, introducing a
+misalignment between its pre-training objectives and feature extraction
+methods. This inconsistency can diminish the quality of the image's feature
+representation, adversely affecting CLIP's effectiveness in target tasks. In
+this paper, we view text features as precise neighbors of image features in
+CLIP's space and present a novel CrOss-moDal nEighbor Representation(CODER)
+based on the distance structure between images and their neighbor texts. This
+feature extraction method aligns better with CLIP's pre-training objectives,
+thereby fully leveraging CLIP's robust cross-modal capabilities. The key to
+construct a high-quality CODER lies in how to create a vast amount of
+high-quality and diverse texts to match with images. We introduce the Auto Text
+Generator(ATG) to automatically generate the required texts in a data-free and
+training-free manner. We apply CODER to CLIP's zero-shot and few-shot image
+classification tasks. Experiment results across various datasets and models
+confirm CODER's effectiveness. Code is available
+at:https://github.com/YCaigogogo/CVPR24-CODER.",cs.CV,"['cs.CV', 'cs.AI']"
+MaGGIe: Masked Guided Gradual Human Instance Matting,Chuong Huynh · Seoung Wug Oh · Abhinav Shrivastava · Joon-Young Lee,https://maggie-matt.github.io,https://arxiv.org/abs/2404.16035v1,,2404.16035v1.pdf,MaGGIe: Masked Guided Gradual Human Instance Matting,"Human matting is a foundation task in image and video processing, where human
+foreground pixels are extracted from the input. Prior works either improve the
+accuracy by additional guidance or improve the temporal consistency of a single
+instance across frames. We propose a new framework MaGGIe, Masked Guided
+Gradual Human Instance Matting, which predicts alpha mattes progressively for
+each human instances while maintaining the computational cost, precision, and
+consistency. Our method leverages modern architectures, including transformer
+attention and sparse convolution, to output all instance mattes simultaneously
+without exploding memory and latency. Although keeping constant inference costs
+in the multiple-instance scenario, our framework achieves robust and versatile
+performance on our proposed synthesized benchmarks. With the higher quality
+image and video matting benchmarks, the novel multi-instance synthesis approach
+from publicly available sources is introduced to increase the generalization of
+models in real-world scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+Aligning and Prompting Everything All at Once for Universal Visual Perception,Yunhang Shen · Chaoyou Fu · Peixian Chen · Mengdan Zhang · Ke Li · Xing Sun · Yunsheng Wu · Shaohui Lin · Rongrong Ji, ,https://arxiv.org/abs/2312.02153v1,,2312.02153v1.pdf,Aligning and Prompting Everything All at Once for Universal Visual Perception,"Vision foundation models have been explored recently to build general-purpose
+vision systems. However, predominant paradigms, driven by casting
+instance-level tasks as an object-word alignment, bring heavy cross-modality
+interaction, which is not effective in prompting object detection and visual
+grounding. Another line of work that focuses on pixel-level tasks often
+encounters a large annotation gap of things and stuff, and suffers from mutual
+interference between foreground-object and background-class segmentation. In
+stark contrast to the prevailing methods, we present APE, a universal visual
+perception model for aligning and prompting everything all at once in an image
+to perform diverse tasks, i.e., detection, segmentation, and grounding, as an
+instance-level sentence-object matching paradigm. Specifically, APE advances
+the convergence of detection and grounding by reformulating language-guided
+grounding as open-vocabulary detection, which efficiently scales up model
+prompting to thousands of category vocabularies and region descriptions while
+maintaining the effectiveness of cross-modality fusion. To bridge the
+granularity gap of different pixel-level tasks, APE equalizes semantic and
+panoptic segmentation to proxy instance learning by considering any isolated
+regions as individual instances. APE aligns vision and language representation
+on broad data with natural and challenging characteristics all at once without
+task-specific fine-tuning. The extensive experiments on over 160 datasets
+demonstrate that, with only one-suit of weights, APE outperforms (or is on par
+with) the state-of-the-art models, proving that an effective yet universal
+perception for anything aligning and prompting is indeed feasible. Codes and
+trained models are released at https://github.com/shenyunhang/APE.",cs.CV,['cs.CV']
+A General and Efficient Training for Transformer via Token Expansion,Wenxuan Huang · Yunhang Shen · Jiao Xie · Baochang Zhang · Gaoqi He · Ke Li · Xing Sun · Shaohui Lin, ,https://arxiv.org/abs/2404.00672v1,,2404.00672v1.pdf,A General and Efficient Training for Transformer via Token Expansion,"The remarkable performance of Vision Transformers (ViTs) typically requires
+an extremely large training cost. Existing methods have attempted to accelerate
+the training of ViTs, yet typically disregard method universality with accuracy
+dropping. Meanwhile, they break the training consistency of the original
+transformers, including the consistency of hyper-parameters, architecture, and
+strategy, which prevents them from being widely applied to different
+Transformer networks. In this paper, we propose a novel token growth scheme
+Token Expansion (termed ToE) to achieve consistent training acceleration for
+ViTs. We introduce an ""initialization-expansion-merging"" pipeline to maintain
+the integrity of the intermediate feature distribution of original
+transformers, preventing the loss of crucial learnable information in the
+training process. ToE can not only be seamlessly integrated into the training
+and fine-tuning process of transformers (e.g., DeiT and LV-ViT), but also
+effective for efficient training frameworks (e.g., EfficientTrain), without
+twisting the original training hyper-parameters, architecture, and introducing
+additional training strategies. Extensive experiments demonstrate that ToE
+achieves about 1.3x faster for the training of ViTs in a lossless manner, or
+even with performance gains over the full-token training baselines. Code is
+available at https://github.com/Osilly/TokenExpansion .",cs.LG,"['cs.LG', 'cs.AI', 'cs.CL', 'cs.CV']"
+RankED: Addressing Imbalance and Uncertainty in Edge Detection Using Ranking-based Losses,bedrettin cetinkaya · Sinan Kalkan · Emre Akbas,https://ranked-cvpr24.github.io/,https://arxiv.org/abs/2403.01795,,2403.01795.pdf,RankED: Addressing Imbalance and Uncertainty in Edge Detection Using Ranking-based Losses,"Detecting edges in images suffers from the problems of (P1) heavy imbalance
+between positive and negative classes as well as (P2) label uncertainty owing
+to disagreement between different annotators. Existing solutions address P1
+using class-balanced cross-entropy loss and dice loss and P2 by only predicting
+edges agreed upon by most annotators. In this paper, we propose RankED, a
+unified ranking-based approach that addresses both the imbalance problem (P1)
+and the uncertainty problem (P2). RankED tackles these two problems with two
+components: One component which ranks positive pixels over negative pixels, and
+the second which promotes high confidence edge pixels to have more label
+certainty. We show that RankED outperforms previous studies and sets a new
+state-of-the-art on NYUD-v2, BSDS500 and Multi-cue datasets. Code is available
+at https://ranked-cvpr24.github.io.",cs.CV,['cs.CV']
+Solving the Catastrophic Forgetting Problem in Generalized Category Discovery,Xinzi Cao · Xiawu Zheng · Guanhong Wang · Weijiang Yu · Yunhang Shen · Ke Li · Yutong Lu · Yonghong Tian, ,https://arxiv.org/abs/2308.12112,,2308.12112.pdf,Generalized Continual Category Discovery,"Most of Continual Learning (CL) methods push the limit of supervised learning
+settings, where an agent is expected to learn new labeled tasks and not forget
+previous knowledge. However, these settings are not well aligned with real-life
+scenarios, where a learning agent has access to a vast amount of unlabeled data
+encompassing both novel (entirely unlabeled) classes and examples from known
+classes. Drawing inspiration from Generalized Category Discovery (GCD), we
+introduce a novel framework that relaxes this assumption. Precisely, in any
+task, we allow for the existence of novel and known classes, and one must use
+continual version of unsupervised learning methods to discover them. We call
+this setting Generalized Continual Category Discovery (GCCD). It unifies CL and
+GCD, bridging the gap between synthetic benchmarks and real-life scenarios.
+With a series of experiments, we present that existing methods fail to
+accumulate knowledge from subsequent tasks in which unlabeled samples of novel
+classes are present. In light of these limitations, we propose a method that
+incorporates both supervised and unsupervised signals and mitigates the
+forgetting through the use of centroid adaptation. Our method surpasses strong
+CL methods adopted for GCD techniques and presents a superior representation
+learning performance.",cs.LG,"['cs.LG', 'cs.CV']"
+Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning,Dipam Goswami · Albin Soutif · Yuyang Liu · Sandesh Kamath · Bartłomiej Twardowski · Joost van de Weijer,https://github.com/dipamgoswami/ADC,https://arxiv.org/abs/2405.19074,,2405.19074.pdf,Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning,"Continual learning methods are known to suffer from catastrophic forgetting,
+a phenomenon that is particularly hard to counter for methods that do not store
+exemplars of previous tasks. Therefore, to reduce potential drift in the
+feature extractor, existing exemplar-free methods are typically evaluated in
+settings where the first task is significantly larger than subsequent tasks.
+Their performance drops drastically in more challenging settings starting with
+a smaller first task. To address this problem of feature drift estimation for
+exemplar-free methods, we propose to adversarially perturb the current samples
+such that their embeddings are close to the old class prototypes in the old
+model embedding space. We then estimate the drift in the embedding space from
+the old to the new model using the perturbed images and compensate the
+prototypes accordingly. We exploit the fact that adversarial samples are
+transferable from the old to the new feature space in a continual learning
+setting. The generation of these images is simple and computationally cheap. We
+demonstrate in our experiments that the proposed approach better tracks the
+movement of prototypes in embedding space and outperforms existing methods on
+several standard continual learning benchmarks as well as on fine-grained
+datasets. Code is available at https://github.com/dipamgoswami/ADC.",cs.CV,"['cs.CV', 'cs.AI']"
+Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?,Zhengyue Zhao · Jinhao Duan · Kaidi Xu · Chenan Wang · Rui Zhang · Zidong Du · Qi Guo · Xing Hu, ,https://arxiv.org/abs/2312.00084,,2312.00084.pdf,Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?,"Stable Diffusion has established itself as a foundation model in generative
+AI artistic applications, receiving widespread research and application. Some
+recent fine-tuning methods have made it feasible for individuals to implant
+personalized concepts onto the basic Stable Diffusion model with minimal
+computational costs on small datasets. However, these innovations have also
+given rise to issues like facial privacy forgery and artistic copyright
+infringement. In recent studies, researchers have explored the addition of
+imperceptible adversarial perturbations to images to prevent potential
+unauthorized exploitation and infringements when personal data is used for
+fine-tuning Stable Diffusion. Although these studies have demonstrated the
+ability to protect images, it is essential to consider that these methods may
+not be entirely applicable in real-world scenarios. In this paper, we
+systematically evaluate the use of perturbations to protect images within a
+practical threat model. The results suggest that these approaches may not be
+sufficient to safeguard image privacy and copyright effectively. Furthermore,
+we introduce a purification method capable of removing protected perturbations
+while preserving the original image structure to the greatest extent possible.
+Experiments reveal that Stable Diffusion can effectively learn from purified
+images over all protective methods.",cs.CV,['cs.CV']
+MoST: Multi-modality Scene Tokenization for Motion Prediction,Norman Mu · Jingwei Ji · Zhenpei Yang · Nathan Harada · Haotian Tang · Kan Chen · Charles R. Qi · Runzhou Ge · Kratarth Goel · Zoey Yang · Scott Ettinger · Rami Al-Rfou · Dragomir Anguelov · Yin Zhou, ,http://export.arxiv.org/abs/2404.19531,,2404.19531.pdf,MoST: Multi-modality Scene Tokenization for Motion Prediction,"Many existing motion prediction approaches rely on symbolic perception
+outputs to generate agent trajectories, such as bounding boxes, road graph
+information and traffic lights. This symbolic representation is a high-level
+abstraction of the real world, which may render the motion prediction model
+vulnerable to perception errors (e.g., failures in detecting open-vocabulary
+obstacles) while missing salient information from the scene context (e.g., poor
+road conditions). An alternative paradigm is end-to-end learning from raw
+sensors. However, this approach suffers from the lack of interpretability and
+requires significantly more training resources. In this work, we propose
+tokenizing the visual world into a compact set of scene elements and then
+leveraging pre-trained image foundation models and LiDAR neural networks to
+encode all the scene elements in an open-vocabulary manner. The image
+foundation model enables our scene tokens to encode the general knowledge of
+the open world while the LiDAR neural network encodes geometry information. Our
+proposed representation can efficiently encode the multi-frame multi-modality
+observations with a few hundred tokens and is compatible with most
+transformer-based architectures. To evaluate our method, we have augmented
+Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open
+Motion Dataset show that our approach leads to significant performance
+improvements over the state-of-the-art.",cs.CV,['cs.CV']
+Task-Driven Wavelets using Constrained Empirical Risk Minimization,Eric Marcus · Ray Sheombarsing · Jan-Jakob Sonke · Jonas Teuwen,https://github.com/NKI-AI/CERM,,https://aiforoncology.nl/news/2024-02-27/two-papers-accepted-at-cvpr-2024/,,,,,nan
+Insights from the Use of Previously Unseen Neural Architecture Search Datasets,Rob Geada · David Towers · Matthew Forshaw · Amir Atapour-Abarghouei · Stephen McGough,https://github.com/Towers-D/NAS-Unseen-Datasets,https://arxiv.org/abs/2404.02189,,2404.02189.pdf,Insights from the Use of Previously Unseen Neural Architecture Search Datasets,"The boundless possibility of neural networks which can be used to solve a
+problem -- each with different performance -- leads to a situation where a Deep
+Learning expert is required to identify the best neural network. This goes
+against the hope of removing the need for experts. Neural Architecture Search
+(NAS) offers a solution to this by automatically identifying the best
+architecture. However, to date, NAS work has focused on a small set of datasets
+which we argue are not representative of real-world problems. We introduce
+eight new datasets created for a series of NAS Challenges: AddNIST, Language,
+MultNIST, CIFARTile, Gutenberg, Isabella, GeoClassing, and Chesseract. These
+datasets and challenges are developed to direct attention to issues in NAS
+development and to encourage authors to consider how their models will perform
+on datasets unknown to them at development time. We present experimentation
+using standard Deep Learning methods as well as the best results from challenge
+participants.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach,Wei Dong · Xing Zhang · Bihui Chen · Dawei Yan · Zhijun Lin · Qingsen Yan · Peng Wang · Yang Yang, ,https://arxiv.org/abs/2403.19067,,2403.19067.pdf,Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach,"Parameter-efficient fine-tuning for pre-trained Vision Transformers aims to
+adeptly tailor a model to downstream tasks by learning a minimal set of new
+adaptation parameters while preserving the frozen majority of pre-trained
+parameters. Striking a balance between retaining the generalizable
+representation capacity of the pre-trained model and acquiring task-specific
+features poses a key challenge. Currently, there is a lack of focus on guiding
+this delicate trade-off. In this study, we approach the problem from the
+perspective of Singular Value Decomposition (SVD) of pre-trained parameter
+matrices, providing insights into the tuning dynamics of existing methods.
+Building upon this understanding, we propose a Residual-based Low-Rank
+Rescaling (RLRR) fine-tuning strategy. This strategy not only enhances
+flexibility in parameter tuning but also ensures that new parameters do not
+deviate excessively from the pre-trained model through a residual design.
+Extensive experiments demonstrate that our method achieves competitive
+performance across various downstream image classification tasks, all while
+maintaining comparable new parameters. We believe this work takes a step
+forward in offering a unified perspective for interpreting existing methods and
+serves as motivation for the development of new approaches that move closer to
+effectively considering the crucial trade-off mentioned above. Our code is
+available at
+\href{https://github.com/zstarN70/RLRR.git}{https://github.com/zstarN70/RLRR.git}.",cs.CV,['cs.CV']
+Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithms,Joren Brunekreef · Eric Marcus · Ray Sheombarsing · Jan-Jakob Sonke · Jonas Teuwen,https://github.com/NKI-AI/kandinsky-calibration,https://arxiv.org/abs/2311.11837v1,,2311.11837v1.pdf,Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithms,"Image segmentation algorithms can be understood as a collection of pixel
+classifiers, for which the outcomes of nearby pixels are correlated. Classifier
+models can be calibrated using Inductive Conformal Prediction, but this
+requires holding back a sufficiently large calibration dataset for computing
+the distribution of non-conformity scores of the model's predictions. If one
+only requires only marginal calibration on the image level, this calibration
+set consists of all individual pixels in the images available for calibration.
+However, if the goal is to attain proper calibration for each individual pixel
+classifier, the calibration set consists of individual images. In a scenario
+where data are scarce (such as the medical domain), it may not always be
+possible to set aside sufficiently many images for this pixel-level
+calibration. The method we propose, dubbed ``Kandinsky calibration'', makes use
+of the spatial structure present in the distribution of natural images to
+simultaneously calibrate the classifiers of ``similar'' pixels. This can be
+seen as an intermediate approach between marginal (imagewise) and conditional
+(pixelwise) calibration, where non-conformity scores are aggregated over
+similar image regions, thereby making more efficient use of the images
+available for calibration. We run experiments on segmentation algorithms
+trained and calibrated on subsets of the public MS-COCO and Medical Decathlon
+datasets, demonstrating that Kandinsky calibration method can significantly
+improve the coverage. When compared to both pixelwise and imagewise calibration
+on little data, the Kandinsky method achieves much lower coverage errors,
+indicating the data efficiency of the Kandinsky calibration.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion,Xinyu Zhan · Lixin Yang · Yifei Zhao · Kangrui Mao · Hanlin Xu · Zenan Lin · Kailin Li · Cewu Lu, ,,https://paperswithcode.com/paper/oakink2-a-dataset-of-bimanual-hands-object,,,,,nan
+What Moves Together Belongs Together,Jenny Seidenschwarz · Aljoša Ošep · Francesco Ferroni · Simon Lucey · Laura Leal-Taixe,https://research.nvidia.com/labs/dvl/projects/semoli/,https://arxiv.org/abs/2402.19463,,2402.19463.pdf,SeMoLi: What Moves Together Belongs Together,"We tackle semi-supervised object detection based on motion cues. Recent
+results suggest that heuristic-based clustering methods in conjunction with
+object trackers can be used to pseudo-label instances of moving objects and use
+these as supervisory signals to train 3D object detectors in Lidar data without
+manual supervision. We re-think this approach and suggest that both, object
+detection, as well as motion-inspired pseudo-labeling, can be tackled in a
+data-driven manner. We leverage recent advances in scene flow estimation to
+obtain point trajectories from which we extract long-term, class-agnostic
+motion patterns. Revisiting correlation clustering in the context of message
+passing networks, we learn to group those motion patterns to cluster points to
+object instances. By estimating the full extent of the objects, we obtain
+per-scan 3D bounding boxes that we use to supervise a Lidar object detection
+network. Our method not only outperforms prior heuristic-based approaches (57.5
+AP, +14 improvement over prior work), more importantly, we show we can
+pseudo-label and train object detectors across datasets.",cs.CV,['cs.CV']
+Stratified Avatar Generation from Sparse Observations,Han Feng · Wenchao Ma · Quankai Gao · Xianwei Zheng · Nan Xue · Huijuan Xu, ,https://arxiv.org/abs/2405.20786,,2405.20786.pdf,Stratified Avatar Generation from Sparse Observations,"Estimating 3D full-body avatars from AR/VR devices is essential for creating
+immersive experiences in AR/VR applications. This task is challenging due to
+the limited input from Head Mounted Devices, which capture only sparse
+observations from the head and hands. Predicting the full-body avatars,
+particularly the lower body, from these sparse observations presents
+significant difficulties. In this paper, we are inspired by the inherent
+property of the kinematic tree defined in the Skinned Multi-Person Linear
+(SMPL) model, where the upper body and lower body share only one common
+ancestor node, bringing the potential of decoupled reconstruction. We propose a
+stratified approach to decouple the conventional full-body avatar
+reconstruction pipeline into two stages, with the reconstruction of the upper
+body first and a subsequent reconstruction of the lower body conditioned on the
+previous stage. To implement this straightforward idea, we leverage the latent
+diffusion model as a powerful probabilistic generator, and train it to follow
+the latent distribution of decoupled motions explored by a VQ-VAE
+encoder-decoder model. Extensive experiments on AMASS mocap dataset demonstrate
+our state-of-the-art performance in the reconstruction of full-body motions.",cs.CV,"['cs.CV', 'cs.HC']"
+VLP: Vision Language Planning for Autonomous Driving,Chenbin Pan · Burhaneddin Yaman · Tommaso Nesti · Abhirup Mallik · Alessandro G Allievi · Senem Velipasalar · Liu Ren, ,https://arxiv.org/abs/2401.05577,,2401.05577.pdf,VLP: Vision Language Planning for Autonomous Driving,"Autonomous driving is a complex and challenging task that aims at safe motion
+planning through scene understanding and reasoning. While vision-only
+autonomous driving methods have recently achieved notable performance, through
+enhanced scene understanding, several key issues, including lack of reasoning,
+low generalization performance and long-tail scenarios, still need to be
+addressed. In this paper, we present VLP, a novel Vision-Language-Planning
+framework that exploits language models to bridge the gap between linguistic
+understanding and autonomous driving. VLP enhances autonomous driving systems
+by strengthening both the source memory foundation and the self-driving car's
+contextual understanding. VLP achieves state-of-the-art end-to-end planning
+performance on the challenging NuScenes dataset by achieving 35.9\% and 60.5\%
+reduction in terms of average L2 error and collision rates, respectively,
+compared to the previous best method. Moreover, VLP shows improved performance
+in challenging long-tail scenarios and strong generalization capabilities when
+faced with new urban environments.",cs.CV,['cs.CV']
+Omni-Q: Omni-Directional Scene Understanding for Unsupervised Visual Grounding,Sai Wang · Yutian Lin · Yu Wu, ,https://ar5iv.labs.arxiv.org/html/2312.09625,,2312.09625.pdf,Weakly-Supervised 3D Visual Grounding based on Visual Linguistic Alignment,"Learning to ground natural language queries to target objects or regions in
+3D point clouds is quite essential for 3D scene understanding. Nevertheless,
+existing 3D visual grounding approaches require a substantial number of
+bounding box annotations for text queries, which is time-consuming and
+labor-intensive to obtain. In this paper, we propose \textbf{3D-VLA}, a weakly
+supervised approach for \textbf{3D} visual grounding based on \textbf{V}isual
+\textbf{L}inguistic \textbf{A}lignment. Our 3D-VLA exploits the superior
+ability of current large-scale vision-language models (VLMs) on aligning the
+semantics between texts and 2D images, as well as the naturally existing
+correspondences between 2D images and 3D point clouds, and thus implicitly
+constructs correspondences between texts and 3D point clouds with no need for
+fine-grained box annotations in the training procedure. During the inference
+stage, the learned text-3D correspondence will help us ground the text queries
+to the 3D target objects even without 2D images. To the best of our knowledge,
+this is the first work to investigate 3D visual grounding in a weakly
+supervised manner by involving large scale vision-language models, and
+extensive experiments on ReferIt3D and ScanRefer datasets demonstrate that our
+3D-VLA achieves comparable and even superior results over the fully supervised
+methods.",cs.CV,"['cs.CV', 'cs.CL']"
+3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions,Weijia Li · Haote Yang · Zhenghao Hu · Juepeng Zheng · Gui-Song Xia · Conghui He, ,https://arxiv.org/abs/2404.04823,,2404.04823.pdf,3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions,"3D building reconstruction from monocular remote sensing images is an
+important and challenging research problem that has received increasing
+attention in recent years, owing to its low cost of data acquisition and
+availability for large-scale applications. However, existing methods rely on
+expensive 3D-annotated samples for fully-supervised training, restricting their
+application to large-scale cross-city scenarios. In this work, we propose
+MLS-BRN, a multi-level supervised building reconstruction network that can
+flexibly utilize training samples with different annotation levels to achieve
+better reconstruction results in an end-to-end manner. To alleviate the demand
+on full 3D supervision, we design two new modules, Pseudo Building Bbox
+Calculator and Roof-Offset guided Footprint Extractor, as well as new tasks and
+training strategies for different types of samples. Experimental results on
+several public and new datasets demonstrate that our proposed MLS-BRN achieves
+competitive performance using much fewer 3D-annotated samples, and
+significantly improves the footprint extraction and 3D reconstruction
+performance compared with current state-of-the-art. The code and datasets of
+this work will be released at https://github.com/opendatalab/MLS-BRN.git.",cs.CV,['cs.CV']
+SeaBird: Segmentation in Bird’s View with Dice Loss Improves Monocular 3D Detection of Large Objects,Abhinav Kumar · Yuliang Guo · Xinyu Huang · Liu Ren · Xiaoming Liu,https://github.com/abhi1kumar/SeaBird,https://arxiv.org/abs/2403.20318,,2403.20318.pdf,SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects,"Monocular 3D detectors achieve remarkable performance on cars and smaller
+objects. However, their performance drops on larger objects, leading to fatal
+accidents. Some attribute the failures to training data scarcity or their
+receptive field requirements of large objects. In this paper, we highlight this
+understudied problem of generalization to large objects. We find that modern
+frontal detectors struggle to generalize to large objects even on nearly
+balanced datasets. We argue that the cause of failure is the sensitivity of
+depth regression losses to noise of larger objects. To bridge this gap, we
+comprehensively investigate regression and dice losses, examining their
+robustness under varying error levels and object sizes. We mathematically prove
+that the dice loss leads to superior noise-robustness and model convergence for
+large objects compared to regression losses for a simplified case. Leveraging
+our theoretical insights, we propose SeaBird (Segmentation in Bird's View) as
+the first step towards generalizing to large objects. SeaBird effectively
+integrates BEV segmentation on foreground objects for 3D detection, with the
+segmentation head trained with the dice loss. SeaBird achieves SoTA results on
+the KITTI-360 leaderboard and improves existing detectors on the nuScenes
+leaderboard, particularly for large objects. Code and models at
+https://github.com/abhi1kumar/SeaBird",cs.CV,"['cs.CV', 'cs.AI']"
+Learning to Count without Annotations,Lukas Knobel · Tengda Han · Yuki Asano,https://github.com/lukasknobel/SelfCollages,https://web3.arxiv.org/abs/2307.08727,,2307.08727.pdf,Learning to Count without Annotations,"While recent supervised methods for reference-based object counting continue
+to improve the performance on benchmark datasets, they have to rely on small
+datasets due to the cost associated with manually annotating dozens of objects
+in images. We propose UnCounTR, a model that can learn this task without
+requiring any manual annotations. To this end, we construct ""Self-Collages"",
+images with various pasted objects as training samples, that provide a rich
+learning signal covering arbitrary object types and counts. Our method builds
+on existing unsupervised representations and segmentation techniques to
+successfully demonstrate for the first time the ability of reference-based
+counting without manual supervision. Our experiments show that our method not
+only outperforms simple baselines and generic models such as FasterRCNN and
+DETR, but also matches the performance of supervised counting models in some
+domains.",cs.CV,['cs.CV']
+AM-RADIO: Agglomerative Models - Reduce All Domains Into One,Mike Ranzinger · Greg Heinrich · Jan Kautz · Pavlo Molchanov,https://github.com/NVlabs/RADIO,https://arxiv.org/abs/2312.06709,,2312.06709.pdf,AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One,"A handful of visual foundation models (VFMs) have recently emerged as the
+backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are
+trained with distinct objectives, exhibiting unique characteristics for various
+downstream tasks. We find that despite their conceptual differences, these
+models can be effectively merged into a unified model through multi-teacher
+distillation. We name this approach AM-RADIO (Agglomerative Model -- Reduce All
+Domains Into One). This integrative approach not only surpasses the performance
+of individual teacher models but also amalgamates their distinctive features,
+such as zero-shot vision-language comprehension, detailed pixel-level
+understanding, and open vocabulary segmentation capabilities. In pursuit of the
+most hardware-efficient backbone, we evaluated numerous architectures in our
+multi-teacher distillation pipeline using the same training recipe. This led to
+the development of a novel architecture (E-RADIO) that exceeds the performance
+of its predecessors and is at least 7x faster than the teacher models. Our
+comprehensive benchmarking process covers downstream tasks including ImageNet
+classification, ADE20k semantic segmentation, COCO object detection and
+LLaVa-1.5 framework.
+  Code: https://github.com/NVlabs/RADIO",cs.CV,['cs.CV']
+Activity-Biometrics: Person Identification from Daily Activities,Shehreen Azad · Yogesh S. Rawat, ,https://arxiv.org/abs/2403.17360,,2403.17360.pdf,Activity-Biometrics: Person Identification from Daily Activities,"In this work, we study a novel problem which focuses on person identification
+while performing daily activities. Learning biometric features from RGB videos
+is challenging due to spatio-temporal complexity and presence of appearance
+biases such as clothing color and background. We propose ABNet, a novel
+framework which leverages disentanglement of biometric and non-biometric
+features to perform effective person identification from daily activities.
+ABNet relies on a bias-less teacher to learn biometric features from RGB videos
+and explicitly disentangle non-biometric features with the help of biometric
+distortion. In addition, ABNet also exploits activity prior for biometrics
+which is enabled by joint biometric and activity learning. We perform
+comprehensive evaluation of the proposed approach across five different
+datasets which are derived from existing activity recognition benchmarks.
+Furthermore, we extensively compare ABNet with existing works in person
+identification and demonstrate its effectiveness for activity-based biometrics
+across all five datasets. The code and dataset can be accessed at:
+\url{https://github.com/sacrcv/Activity-Biometrics/}",cs.CV,['cs.CV']
+Unsupervised Template-assisted  Point Cloud Shape Correspondence Network,Jiacheng Deng · Jiahao Lu · Tianzhu Zhang, ,https://arxiv.org/abs/2403.16412,,2403.16412.pdf,Unsupervised Template-assisted Point Cloud Shape Correspondence Network,"Unsupervised point cloud shape correspondence aims to establish point-wise
+correspondences between source and target point clouds. Existing methods obtain
+correspondences directly by computing point-wise feature similarity between
+point clouds. However, non-rigid objects possess strong deformability and
+unusual shapes, making it a longstanding challenge to directly establish
+correspondences between point clouds with unconventional shapes. To address
+this challenge, we propose an unsupervised Template-Assisted point cloud shape
+correspondence Network, termed TANet, including a template generation module
+and a template assistance module. The proposed TANet enjoys several merits.
+Firstly, the template generation module establishes a set of learnable
+templates with explicit structures. Secondly, we introduce a template
+assistance module that extensively leverages the generated templates to
+establish more accurate shape correspondences from multiple perspectives.
+Extensive experiments on four human and animal datasets demonstrate that TANet
+achieves favorable performance against state-of-the-art methods.",cs.CV,['cs.CV']
+Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination,Yixin Zeng · Zoubin Bi · Yin Mingrui · Xiang Feng · Kun Zhou · Hongzhi Wu, ,https://arxiv.org/html/2404.10766v1,,2404.10766v1.pdf,RapidVol: Rapid Reconstruction of 3D Ultrasound Volumes from Sensorless 2D Scans,"Two-dimensional (2D) freehand ultrasonography is one of the most commonly
+used medical imaging modalities, particularly in obstetrics and gynaecology.
+However, it only captures 2D cross-sectional views of inherently 3D anatomies,
+losing valuable contextual information. As an alternative to requiring costly
+and complex 3D ultrasound scanners, 3D volumes can be constructed from 2D scans
+using machine learning. However this usually requires long computational time.
+Here, we propose RapidVol: a neural representation framework to speed up
+slice-to-volume ultrasound reconstruction. We use tensor-rank decomposition, to
+decompose the typical 3D volume into sets of tri-planes, and store those
+instead, as well as a small neural network. A set of 2D ultrasound scans, with
+their ground truth (or estimated) 3D position and orientation (pose) is all
+that is required to form a complete 3D reconstruction. Reconstructions are
+formed from real fetal brain scans, and then evaluated by requesting novel
+cross-sectional views. When compared to prior approaches based on fully
+implicit representation (e.g. neural radiance fields), our method is over 3x
+quicker, 46% more accurate, and if given inaccurate poses is more robust.
+Further speed-up is also possible by reconstructing from a structural prior
+rather than from scratch.",eess.IV,"['eess.IV', 'cs.CV']"
+FC-GNN: Recovering Reliable and Accurate Correspondences from Interferences,Haobo Xu · Jun Zhou · Hua Yang · Renjie Pan · Cunyan Li, ,,https://www.researchgate.net/publication/376796777_Matching-to-Detecting_Establishing_Dense_and_Reliable_Correspondences_Between_Images,,,,,nan
+CFAT: Unleashing Triangular Windows for Image Super-resolution,Abhisek Ray · Gaurav Kumar · Maheshkumar Kolekar, ,https://arxiv.org/abs/2403.16143,,2403.16143.pdf,CFAT: Unleashing TriangularWindows for Image Super-resolution,"Transformer-based models have revolutionized the field of image
+super-resolution (SR) by harnessing their inherent ability to capture complex
+contextual features. The overlapping rectangular shifted window technique used
+in transformer architecture nowadays is a common practice in super-resolution
+models to improve the quality and robustness of image upscaling. However, it
+suffers from distortion at the boundaries and has limited unique shifting
+modes. To overcome these weaknesses, we propose a non-overlapping triangular
+window technique that synchronously works with the rectangular one to mitigate
+boundary-level distortion and allows the model to access more unique sifting
+modes. In this paper, we propose a Composite Fusion Attention Transformer
+(CFAT) that incorporates triangular-rectangular window-based local attention
+with a channel-based global attention technique in image super-resolution. As a
+result, CFAT enables attention mechanisms to be activated on more image pixels
+and captures long-range, multi-scale features to improve SR performance. The
+extensive experimental results and ablation study demonstrate the effectiveness
+of CFAT in the SR domain. Our proposed model shows a significant 0.7 dB
+performance improvement over other state-of-the-art SR architectures.",eess.IV,"['eess.IV', 'cs.CV', 'cs.LG', 'cs.MM']"
+Deciphering ‘What’ and ‘Where’ Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations,Xiao Zhang · David Yunis · Michael Maire, ,https://arxiv.org/abs/2312.06716,,2312.06716.pdf,Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations,"We present an approach for analyzing grouping information contained within a
+neural network's activations, permitting extraction of spatial layout and
+semantic segmentation from the behavior of large pre-trained vision models.
+Unlike prior work, our method conducts a wholistic analysis of a network's
+activation state, leveraging features from all layers and obviating the need to
+guess which part of the model contains relevant information. Motivated by
+classic spectral clustering, we formulate this analysis in terms of an
+optimization objective involving a set of affinity matrices, each formed by
+comparing features within a different layer. Solving this optimization problem
+using gradient descent allows our technique to scale from single images to
+dataset-level analysis, including, in the latter, both intra- and inter-image
+relationships. Analyzing a pre-trained generative transformer provides insight
+into the computational strategy learned by such models. Equating affinity with
+key-query similarity across attention layers yields eigenvectors encoding scene
+spatial layout, whereas defining affinity by value vector similarity yields
+eigenvectors encoding object identity. This result suggests that key and query
+vectors coordinate attentional information flow according to spatial proximity
+(a `where' pathway), while value vectors refine a semantic category
+representation (a `what' pathway).",cs.CV,['cs.CV']
+DART: Implicit Doppler Tomography for Radar Novel View Synthesis,Tianshu Huang · John Miller · Akarsh Prabhakara · Tao Jin · Tarana Laroia · Zico Kolter · Anthony Rowe,https://wiselabcmu.github.io/dart/,https://arxiv.org/abs/2403.03896v1,,2403.03896v1.pdf,DART: Implicit Doppler Tomography for Radar Novel View Synthesis,"Simulation is an invaluable tool for radio-frequency system designers that
+enables rapid prototyping of various algorithms for imaging, target detection,
+classification, and tracking. However, simulating realistic radar scans is a
+challenging task that requires an accurate model of the scene, radio frequency
+material properties, and a corresponding radar synthesis function. Rather than
+specifying these models explicitly, we propose DART - Doppler Aided Radar
+Tomography, a Neural Radiance Field-inspired method which uses radar-specific
+physics to create a reflectance and transmittance-based rendering pipeline for
+range-Doppler images. We then evaluate DART by constructing a custom data
+collection platform and collecting a novel radar dataset together with accurate
+position and instantaneous velocity measurements from lidar-based localization.
+In comparison to state-of-the-art baselines, DART synthesizes superior radar
+range-Doppler images from novel views across all datasets and additionally can
+be used to generate high quality tomographic images.",cs.CV,"['cs.CV', 'cs.LG']"
+Don’t drop your samples! Coherence-aware training benefits Conditional diffusion,Nicolas Dufour · Victor Besnier · Vicky Kalogeiton · David Picard,https://nicolas-dufour.github.io/cad,https://arxiv.org/abs/2405.20324,,2405.20324.pdf,Don't drop your samples! Coherence-aware training benefits Conditional diffusion,"Conditional diffusion models are powerful generative models that can leverage
+various types of conditional information, such as class labels, segmentation
+masks, or text captions. However, in many real-world scenarios, conditional
+information may be noisy or unreliable due to human annotation errors or weak
+alignment. In this paper, we propose the Coherence-Aware Diffusion (CAD), a
+novel method that integrates coherence in conditional information into
+diffusion models, allowing them to learn from noisy annotations without
+discarding data. We assume that each data point has an associated coherence
+score that reflects the quality of the conditional information. We then
+condition the diffusion model on both the conditional information and the
+coherence score. In this way, the model learns to ignore or discount the
+conditioning when the coherence is low. We show that CAD is theoretically sound
+and empirically effective on various conditional generation tasks. Moreover, we
+show that leveraging coherence generates realistic and diverse samples that
+respect conditional information better than models trained on cleaned datasets
+where samples with low coherence have been discarded.",cs.CV,"['cs.CV', 'cs.LG']"
+"Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation",Biao Gong · Siteng Huang · Yutong Feng · Shiwei Zhang · Yuyuan Li · Yu Liu, ,https://arxiv.org/abs/2311.15773,,2311.15773.pdf,"Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation","Diffusion models have recently achieved remarkable progress in generating
+realistic images. However, challenges remain in accurately understanding and
+synthesizing the layout requirements in the textual prompts. To align the
+generated image with layout instructions, we present a training-free layout
+calibration system SimM that intervenes in the generative process on the fly
+during inference time. Specifically, following a ""check-locate-rectify""
+pipeline, the system first analyses the prompt to generate the target layout
+and compares it with the intermediate outputs to automatically detect errors.
+Then, by moving the located activations and making intra- and inter-map
+adjustments, the rectification process can be performed with negligible
+computational overhead. To evaluate SimM over a range of layout requirements,
+we present a benchmark SimMBench that compensates for the lack of superlative
+spatial relations in existing datasets. And both quantitative and qualitative
+results demonstrate the effectiveness of the proposed SimM in calibrating the
+layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM.",cs.CV,['cs.CV']
+GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos,Tomas Soucek · Dima Damen · Michael Wray · Ivan Laptev · Josef Sivic,https://soczech.github.io/genhowto/,https://arxiv.org/abs/2312.07322,,2312.07322.pdf,GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos,"We address the task of generating temporally consistent and physically
+plausible images of actions and object state transformations. Given an input
+image and a text prompt describing the targeted transformation, our generated
+images preserve the environment and transform objects in the initial image. Our
+contributions are threefold. First, we leverage a large body of instructional
+videos and automatically mine a dataset of triplets of consecutive frames
+corresponding to initial object states, actions, and resulting object
+transformations. Second, equipped with this data, we develop and train a
+conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a
+variety of objects and actions and show superior performance compared to
+existing methods. In particular, we introduce a quantitative evaluation where
+GenHowTo achieves 88% and 74% on seen and unseen interaction categories,
+respectively, outperforming prior work by a large margin.",cs.CV,['cs.CV']
+From Coarse to Fine-Grained Open-Set Recognition,Nico Lang · Vésteinn Snæbjarnarson · Elijah Cole · Oisin Mac Aodha · Christian Igel · Serge Belongie, ,https://arxiv.org/abs/2307.07214,,2307.07214.pdf,Complementary Frequency-Varying Awareness Network for Open-Set Fine-Grained Image Recognition,"Open-set image recognition is a challenging topic in computer vision. Most of
+the existing works in literature focus on learning more discriminative features
+from the input images, however, they are usually insensitive to the high- or
+low-frequency components in features, resulting in a decreasing performance on
+fine-grained image recognition. To address this problem, we propose a
+Complementary Frequency-varying Awareness Network that could better capture
+both high-frequency and low-frequency information, called CFAN. The proposed
+CFAN consists of three sequential modules: (i) a feature extraction module is
+introduced for learning preliminary features from the input images; (ii) a
+frequency-varying filtering module is designed to separate out both high- and
+low-frequency components from the preliminary features in the frequency domain
+via a frequency-adjustable filter; (iii) a complementary temporal aggregation
+module is designed for aggregating the high- and low-frequency components via
+two Long Short-Term Memory networks into discriminative features. Based on
+CFAN, we further propose an open-set fine-grained image recognition method,
+called CFAN-OSFGR, which learns image features via CFAN and classifies them via
+a linear classifier. Experimental results on 3 fine-grained datasets and 2
+coarse-grained datasets demonstrate that CFAN-OSFGR performs significantly
+better than 9 state-of-the-art methods in most cases.",cs.CV,['cs.CV']
+FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models,Jinglin Xu · Yijie Guo · Yuxin Peng,https://pku-icst-mipl.github.io/FinePOSE_ProjectPage/,https://arxiv.org/abs/2405.05216,,,FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models,"The 3D Human Pose Estimation (3D HPE) task uses 2D images or videos to
+predict human joint coordinates in 3D space. Despite recent advancements in
+deep learning-based methods, they mostly ignore the capability of coupling
+accessible texts and naturally feasible knowledge of humans, missing out on
+valuable implicit supervision to guide the 3D HPE task. Moreover, previous
+efforts often study this task from the perspective of the whole human body,
+neglecting fine-grained guidance hidden in different body parts. To this end,
+we present a new Fine-Grained Prompt-Driven Denoiser based on a diffusion model
+for 3D HPE, named \textbf{FinePOSE}. It consists of three core blocks enhancing
+the reverse process of the diffusion model: (1) Fine-grained Part-aware Prompt
+learning (FPP) block constructs fine-grained part-aware prompts via coupling
+accessible texts and naturally feasible knowledge of body parts with learnable
+prompts to model implicit guidance. (2) Fine-grained Prompt-pose Communication
+(FPC) block establishes fine-grained communications between learned part-aware
+prompts and poses to improve the denoising quality. (3) Prompt-driven Timestamp
+Stylization (PTS) block integrates learned prompt embedding and temporal
+information related to the noise level to enable adaptive adjustment at each
+denoising step. Extensive experiments on public single-human pose estimation
+datasets show that FinePOSE outperforms state-of-the-art methods. We further
+extend FinePOSE to multi-human pose estimation. Achieving 34.3mm average MPJPE
+on the EgoHumans dataset demonstrates the potential of FinePOSE to deal with
+complex multi-human scenarios. Code is available at
+https://github.com/PKU-ICST-MIPL/FinePOSE_CVPR2024.",cs.CV,['cs.CV']
+FedMef: Towards Memory-efficient Federated Dynamic Pruning,Hong Huang · Weiming Zhuang · Chen Chen · Lingjuan Lyu, ,https://arxiv.org/abs/2403.14737,,2403.14737.pdf,FedMef: Towards Memory-efficient Federated Dynamic Pruning,"Federated learning (FL) promotes decentralized training while prioritizing
+data confidentiality. However, its application on resource-constrained devices
+is challenging due to the high demand for computation and memory resources to
+train deep learning models. Neural network pruning techniques, such as dynamic
+pruning, could enhance model efficiency, but directly adopting them in FL still
+poses substantial challenges, including post-pruning performance degradation,
+high activation memory usage, etc. To address these challenges, we propose
+FedMef, a novel and memory-efficient federated dynamic pruning framework.
+FedMef comprises two key components. First, we introduce the budget-aware
+extrusion that maintains pruning efficiency while preserving post-pruning
+performance by salvaging crucial information from parameters marked for pruning
+within a given budget. Second, we propose scaled activation pruning to
+effectively reduce activation memory footprints, which is particularly
+beneficial for deploying FL to memory-limited devices. Extensive experiments
+demonstrate the effectiveness of our proposed FedMef. In particular, it
+achieves a significant reduction of 28.5% in memory footprint compared to
+state-of-the-art methods while obtaining superior accuracy.",cs.LG,"['cs.LG', 'cs.DC']"
+FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition,Sicheng Mo · Fangzhou Mu · Kuan Heng Lin · Yanli Liu · Bochen Guan · Yin Li · Bolei Zhou, ,https://arxiv.org/abs/2312.07536,,2312.07536.pdf,FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition,"Recent approaches such as ControlNet offer users fine-grained spatial control
+over text-to-image (T2I) diffusion models. However, auxiliary modules have to
+be trained for each type of spatial condition, model architecture, and
+checkpoint, putting them at odds with the diverse intents and preferences a
+human designer would like to convey to the AI models during the content
+creation process. In this work, we present FreeControl, a training-free
+approach for controllable T2I generation that supports multiple conditions,
+architectures, and checkpoints simultaneously. FreeControl designs structure
+guidance to facilitate the structure alignment with a guidance image, and
+appearance guidance to enable the appearance sharing between images generated
+using the same seed. Extensive qualitative and quantitative experiments
+demonstrate the superior performance of FreeControl across a variety of
+pre-trained T2I models. In particular, FreeControl facilitates convenient
+training-free control over many different architectures and checkpoints, allows
+the challenging input conditions on which most of the existing training-free
+methods fail, and achieves competitive synthesis quality with training-based
+approaches.",cs.CV,['cs.CV']
+Revamping Federated Learning Security from a Defender's Perspective: A Unified Defense with Homomorphic Encrypted Data Space,Naveen Kumar Kummari · Reshmi Mitra · Krishna Mohan Chalavadi,https://github.com/NaveenKumar-1311/FCD,https://arxiv.org/abs/2307.08672,,2307.08672.pdf,FedDefender: Backdoor Attack Defense in Federated Learning,"Federated Learning (FL) is a privacy-preserving distributed machine learning
+technique that enables individual clients (e.g., user participants, edge
+devices, or organizations) to train a model on their local data in a secure
+environment and then share the trained model with an aggregator to build a
+global model collaboratively. In this work, we propose FedDefender, a defense
+mechanism against targeted poisoning attacks in FL by leveraging differential
+testing. Our proposed method fingerprints the neuron activations of clients'
+models on the same input and uses differential testing to identify a
+potentially malicious client containing a backdoor. We evaluate FedDefender
+using MNIST and FashionMNIST datasets with 20 and 30 clients, and our results
+demonstrate that FedDefender effectively mitigates such attacks, reducing the
+attack success rate (ASR) to 10\% without deteriorating the global model
+performance.",cs.CR,"['cs.CR', 'cs.AI', 'cs.CV', 'cs.LG']"
+DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions,Yunxiao Shi · Manish Singh · Hong Cai · Fatih Porikli, ,https://arxiv.org/abs/2403.12202,,2403.12202.pdf,DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions,"In this paper, we introduce a novel approach that harnesses both 2D and 3D
+attentions to enable highly accurate depth completion without requiring
+iterative spatial propagations. Specifically, we first enhance a baseline
+convolutional depth completion model by applying attention to 2D features in
+the bottleneck and skip connections. This effectively improves the performance
+of this simple network and sets it on par with the latest, complex
+transformer-based models. Leveraging the initial depths and features from this
+network, we uplift the 2D features to form a 3D point cloud and construct a 3D
+point transformer to process it, allowing the model to explicitly learn and
+exploit 3D geometric features. In addition, we propose normalization techniques
+to process the point cloud, which improves learning and leads to better
+accuracy than directly using point transformers off the shelf. Furthermore, we
+incorporate global attention on downsampled point cloud features, which enables
+long-range context while still being computationally feasible. We evaluate our
+method, DeCoTR, on established depth completion benchmarks, including NYU Depth
+V2 and KITTI, showcasing that it sets new state-of-the-art performance. We
+further conduct zero-shot evaluations on ScanNet and DDAD benchmarks and
+demonstrate that DeCoTR has superior generalizability compared to existing
+approaches.",cs.CV,['cs.CV']
+TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing,Sherry X. Chen · Yaron Vaxman · Elad Ben Baruch · David Asulin · Aviad Moreshet · Kuo-Chin Lien · Misha Sra · Pradeep Sen,https://github.com/SherryXTChen/TiNO-Edit,https://arxiv.org/abs/2404.11120,,2404.11120.pdf,TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing,"Despite many attempts to leverage pre-trained text-to-image models (T2I) like
+Stable Diffusion (SD) for controllable image editing, producing good
+predictable results remains a challenge. Previous approaches have focused on
+either fine-tuning pre-trained T2I models on specific datasets to generate
+certain kinds of images (e.g., with a specific object or person), or on
+optimizing the weights, text prompts, and/or learning features for each input
+image in an attempt to coax the image generator to produce the desired result.
+However, these approaches all have shortcomings and fail to produce good
+results in a predictable and controllable manner. To address this problem, we
+present TiNO-Edit, an SD-based method that focuses on optimizing the noise
+patterns and diffusion timesteps during editing, something previously
+unexplored in the literature. With this simple change, we are able to generate
+results that both better align with the original images and reflect the desired
+result. Furthermore, we propose a set of new loss functions that operate in the
+latent domain of SD, greatly speeding up the optimization when compared to
+prior approaches, which operate in the pixel domain. Our method can be easily
+applied to variations of SD including Textual Inversion and DreamBooth that
+encode new concepts and incorporate them into the edited results. We present a
+host of image-editing capabilities enabled by our approach. Our code is
+publicly available at https://github.com/SherryXTChen/TiNO-Edit.",cs.CV,['cs.CV']
+Memory-Scalable and Simplified Functional Map Learning,Robin Magnet · Maks Ovsjanikov, ,https://arxiv.org/abs/2404.00330,,2404.00330.pdf,Memory-Scalable and Simplified Functional Map Learning,"Deep functional maps have emerged in recent years as a prominent
+learning-based framework for non-rigid shape matching problems. While early
+methods in this domain only focused on learning in the functional domain, the
+latest techniques have demonstrated that by promoting consistency between
+functional and pointwise maps leads to significant improvements in accuracy.
+Unfortunately, existing approaches rely heavily on the computation of large
+dense matrices arising from soft pointwise maps, which compromises their
+efficiency and scalability. To address this limitation, we introduce a novel
+memory-scalable and efficient functional map learning pipeline. By leveraging
+the specific structure of functional maps, we offer the possibility to achieve
+identical results without ever storing the pointwise map in memory.
+Furthermore, based on the same approach, we present a differentiable map
+refinement layer adapted from an existing axiomatic refinement algorithm.
+Unlike many functional map learning methods, which use this algorithm at a
+post-processing step, ours can be easily used at train time, enabling to
+enforce consistency between the refined and initial versions of the map. Our
+resulting approach is both simpler, more efficient and more numerically stable,
+by avoiding differentiation through a linear system, while achieving close to
+state-of-the-art results in challenging scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment,Jinglin Xu · Sibo Yin · Guohao Zhao · Zishuo Wang · Yuxin Peng, ,https://arxiv.org/abs/2405.06887,,2405.06887.pdf,FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment,"Existing action quality assessment (AQA) methods mainly learn deep
+representations at the video level for scoring diverse actions. Due to the lack
+of a fine-grained understanding of actions in videos, they harshly suffer from
+low credibility and interpretability, thus insufficient for stringent
+applications, such as Olympic diving events. We argue that a fine-grained
+understanding of actions requires the model to perceive and parse actions in
+both time and space, which is also the key to the credibility and
+interpretability of the AQA technique. Based on this insight, we propose a new
+fine-grained spatial-temporal action parser named \textbf{FineParser}. It
+learns human-centric foreground action representations by focusing on target
+action regions within each frame and exploiting their fine-grained alignments
+in time and space to minimize the impact of invalid backgrounds during the
+assessment. In addition, we construct fine-grained annotations of human-centric
+foreground action masks for the FineDiving dataset, called
+\textbf{FineDiving-HM}. With refined annotations on diverse target action
+procedures, FineDiving-HM can promote the development of real-world AQA
+systems. Through extensive experiments, we demonstrate the effectiveness of
+FineParser, which outperforms state-of-the-art methods while supporting more
+tasks of fine-grained action understanding. Data and code are available at
+\url{https://github.com/PKU-ICST-MIPL/FineParser_CVPR2024}.",cs.CV,['cs.CV']
+Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment,Jiyuan Zhang · Shiyan Chen · Yajing Zheng · Zhaofei Yu · Tiejun Huang, ,https://arxiv.org/abs/2403.09486,,2403.09486.pdf,SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams,"Reconstructing a sequence of sharp images from the blurry input is crucial
+for enhancing our insights into the captured scene and poses a significant
+challenge due to the limited temporal features embedded in the image. Spike
+cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing
+motion features and beneficial for solving this ill-posed problem. Nonetheless,
+existing methods fall into the supervised learning paradigm, which suffers from
+notable performance degradation when applied to real-world scenarios that
+diverge from the synthetic training data domain. Moreover, the quality of
+reconstructed images is capped by the generated images based on motion analysis
+interpolation, which inherently differs from the actual scene, affecting the
+generalization ability of these methods in real high-speed scenarios. To
+address these challenges, we propose the first self-supervised framework for
+the task of spike-guided motion deblurring. Our approach begins with the
+formulation of a spike-guided deblurring model that explores the theoretical
+relationships among spike streams, blurry images, and their corresponding sharp
+sequences. We subsequently develop a self-supervised cascaded framework to
+alleviate the issues of spike noise and spatial-resolution mismatching
+encountered in the deblurring model. With knowledge distillation and
+re-blurring loss, we further design a lightweight deblur network to generate
+high-quality sequences with brightness and texture consistency with the
+original input. Quantitative and qualitative experiments conducted on our
+real-world and synthetic datasets with spikes validate the superior
+generalization of the proposed framework. Our code, data and trained models
+will be available at \url{https://github.com/chenkang455/S-SDM}.",cs.CV,['cs.CV']
+Weakly Supervised Point Cloud Semantic Segmentation via Artificial Oracle,Hyeokjun Kweon · Jihun Kim · Kuk-Jin Yoon, ,,https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/cit2.12239,,,,,nan
+From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation,Hyeokjun Kweon · Kuk-Jin Yoon, ,https://arxiv.org/abs/2312.03585,,2312.03585.pdf,Foundation Model Assisted Weakly Supervised Semantic Segmentation,"This work aims to leverage pre-trained foundation models, such as contrastive
+language-image pre-training (CLIP) and segment anything model (SAM), to address
+weakly supervised semantic segmentation (WSSS) using image-level labels. To
+this end, we propose a coarse-to-fine framework based on CLIP and SAM for
+generating high-quality segmentation seeds. Specifically, we construct an image
+classification task and a seed segmentation task, which are jointly performed
+by CLIP with frozen weights and two sets of learnable task-specific prompts. A
+SAM-based seeding (SAMS) module is designed and applied to each task to produce
+either coarse or fine seed maps. Moreover, we design a multi-label contrastive
+loss supervised by image-level labels and a CAM activation loss supervised by
+the generated coarse seed map. These losses are used to learn the prompts,
+which are the only parts need to be learned in our framework. Once the prompts
+are learned, we input each image along with the learned segmentation-specific
+prompts into CLIP and the SAMS module to produce high-quality segmentation
+seeds. These seeds serve as pseudo labels to train an off-the-shelf
+segmentation network like other two-stage WSSS methods. Experiments show that
+our method achieves the state-of-the-art performance on PASCAL VOC 2012 and
+competitive results on MS COCO 2014. Code is available at
+https://github.com/HAL-42/FMA-WSSS.git.",cs.CV,"['cs.CV', 'cs.AI']"
+SchurVINS: Schur Complement-Based Lightweight Visual Inertial Navigation System,Yunfei Fan · Tianyu Zhao · Guidong Wang,https://github.com/bytedance/SchurVINS,https://arxiv.org/abs/2312.01616,,2312.01616.pdf,SchurVINS: Schur Complement-Based Lightweight Visual Inertial Navigation System,"Accuracy and computational efficiency are the most important metrics to
+Visual Inertial Navigation System (VINS). The existing VINS algorithms with
+either high accuracy or low computational complexity, are difficult to provide
+the high precision localization in resource-constrained devices. To this end,
+we propose a novel filter-based VINS framework named SchurVINS, which could
+guarantee both high accuracy by building a complete residual model and low
+computational complexity with Schur complement. Technically, we first formulate
+the full residual model where Gradient, Hessian and observation covariance are
+explicitly modeled. Then Schur complement is employed to decompose the full
+model into ego-motion residual model and landmark residual model. Finally,
+Extended Kalman Filter (EKF) update is implemented in these two models with
+high efficiency. Experiments on EuRoC and TUM-VI datasets show that our method
+notably outperforms state-of-the-art (SOTA) methods in both accuracy and
+computational complexity. The experimental code of SchurVINS is available at
+https://github.com/bytedance/SchurVINS.",cs.CV,"['cs.CV', 'cs.RO']"
+CAGE: Controllable Articulation GEneration,Jiayi Liu · Hou In Ivan Tam · Ali Mahdavi Amiri · Manolis Savva, ,https://arxiv.org/abs/2312.09570,,2312.09570.pdf,CAGE: Controllable Articulation GEneration,"We address the challenge of generating 3D articulated objects in a
+controllable fashion. Currently, modeling articulated 3D objects is either
+achieved through laborious manual authoring, or using methods from prior work
+that are hard to scale and control directly. We leverage the interplay between
+part shape, connectivity, and motion using a denoising diffusion-based method
+with attention modules designed to extract correlations between part
+attributes. Our method takes an object category label and a part connectivity
+graph as input and generates an object's geometry and motion parameters. The
+generated objects conform to user-specified constraints on the object category,
+part shape, and part articulation. Our experiments show that our method
+outperforms the state-of-the-art in articulated object generation, producing
+more realistic objects while conforming better to user constraints.
+  Video Summary at: http://youtu.be/cH_rbKbyTpE",cs.CV,['cs.CV']
+Neural Modes: Self-supervised Learning of Nonlinear Modal Subspaces,Jiahong Wang · Yinwei DU · Stelian Coros · Bernhard Thomaszewski, ,https://arxiv.org/abs/2404.17620,,2404.17620.pdf,Neural Modes: Self-supervised Learning of Nonlinear Modal Subspaces,"We propose a self-supervised approach for learning physics-based subspaces
+for real-time simulation. Existing learning-based methods construct subspaces
+by approximating pre-defined simulation data in a purely geometric way.
+However, this approach tends to produce high-energy configurations, leads to
+entangled latent space dimensions, and generalizes poorly beyond the training
+set. To overcome these limitations, we propose a self-supervised approach that
+directly minimizes the system's mechanical energy during training. We show that
+our method leads to learned subspaces that reflect physical equilibrium
+constraints, resolve overfitting issues of previous methods, and offer
+interpretable latent space parameters.",cs.LG,"['cs.LG', 'cs.CV', 'cs.GR']"
+Beyond Average: Individualized Visual Scanpath Prediction,Xianyu Chen · Ming Jiang · Qi Zhao, ,https://arxiv.org/abs/2404.12235,,2404.12235.pdf,Beyond Average: Individualized Visual Scanpath Prediction,"Understanding how attention varies across individuals has significant
+scientific and societal impacts. However, existing visual scanpath models treat
+attention uniformly, neglecting individual differences. To bridge this gap,
+this paper focuses on individualized scanpath prediction (ISP), a new attention
+modeling task that aims to accurately predict how different individuals shift
+their attention in diverse visual tasks. It proposes an ISP method featuring
+three novel technical components: (1) an observer encoder to characterize and
+integrate an observer's unique attention traits, (2) an observer-centric
+feature integration approach that holistically combines visual features, task
+guidance, and observer-specific characteristics, and (3) an adaptive fixation
+prioritization mechanism that refines scanpath predictions by dynamically
+prioritizing semantic feature maps based on individual observers' attention
+traits. These novel components allow scanpath models to effectively address the
+attention variations across different observers. Our method is generally
+applicable to different datasets, model architectures, and visual tasks,
+offering a comprehensive tool for transforming general scanpath models into
+individualized ones. Comprehensive evaluations using value-based and
+ranking-based metrics verify the method's effectiveness and generalizability.",cs.CV,['cs.CV']
+CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow,Chenbin Pan · Burhaneddin Yaman · Senem Velipasalar · Liu Ren, ,https://arxiv.org/abs/2403.08919,,2403.08919.pdf,CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow,"Autonomous driving stands as a pivotal domain in computer vision, shaping the
+future of transportation. Within this paradigm, the backbone of the system
+plays a crucial role in interpreting the complex environment. However, a
+notable challenge has been the loss of clear supervision when it comes to
+Bird's Eye View elements. To address this limitation, we introduce
+CLIP-BEVFormer, a novel approach that leverages the power of contrastive
+learning techniques to enhance the multi-view image-derived BEV backbones with
+ground truth information flow. We conduct extensive experiments on the
+challenging nuScenes dataset and showcase significant and consistent
+improvements over the SOTA. Specifically, CLIP-BEVFormer achieves an impressive
+8.5\% and 9.2\% enhancement in terms of NDS and mAP, respectively, over the
+previous best BEV model on the 3D object detection task.",cs.CV,['cs.CV']
+"OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition",Jianqiang Wan · Sibo Song · Wenwen Yu · Yuliang Liu · Wenqing Cheng · Fei Huang · Xiang Bai · Cong Yao · Zhibo Yang, ,https://arxiv.org/abs/2403.19128,,2403.19128.pdf,"OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition","Recently, visually-situated text parsing (VsTP) has experienced notable
+advancements, driven by the increasing demand for automated document
+understanding and the emergence of Generative Large Language Models (LLMs)
+capable of processing document-based questions. Various methods have been
+proposed to address the challenging problem of VsTP. However, due to the
+diversified targets and heterogeneous schemas, previous works usually design
+task-specific architectures and objectives for individual tasks, which
+inadvertently leads to modal isolation and complex workflow. In this paper, we
+propose a unified paradigm for parsing visually-situated text across diverse
+scenarios. Specifically, we devise a universal model, called OmniParser, which
+can simultaneously handle three typical visually-situated text parsing tasks:
+text spotting, key information extraction, and table recognition. In
+OmniParser, all tasks share the unified encoder-decoder architecture, the
+unified objective: point-conditioned text generation, and the unified input &
+output representation: prompt & structured sequences. Extensive experiments
+demonstrate that the proposed OmniParser achieves state-of-the-art (SOTA) or
+highly competitive performances on 7 datasets for the three visually-situated
+text parsing tasks, despite its unified, concise design. The code is available
+at https://github.com/AlibabaResearch/AdvancedLiterateMachinery.",cs.CV,['cs.CV']
+ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image,Kyle Sargent · Zizhang Li · Tanmay Shah · Charles Herrmann · Hong-Xing Yu · Yunzhi Zhang · Eric Ryan Chan · Dmitry Lagun · Li Fei-Fei · Deqing Sun · Jiajun Wu,kylesargent.github.io/zeronvs,https://arxiv.org/abs/2310.17994,,2310.17994.pdf,ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image,"We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
+synthesis for in-the-wild scenes. While existing methods are designed for
+single objects with masked backgrounds, we propose new techniques to address
+challenges introduced by in-the-wild multi-object scenes with complex
+backgrounds. Specifically, we train a generative prior on a mixture of data
+sources that capture object-centric, indoor, and outdoor scenes. To address
+issues from data mixture such as depth-scale ambiguity, we propose a novel
+camera conditioning parameterization and normalization scheme. Further, we
+observe that Score Distillation Sampling (SDS) tends to truncate the
+distribution of complex backgrounds during distillation of 360-degree scenes,
+and propose ""SDS anchoring"" to improve the diversity of synthesized novel
+views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
+in the zero-shot setting, even outperforming methods specifically trained on
+DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
+for single-image novel view synthesis, and demonstrate strong performance in
+this setting. Our code and data are at http://kylesargent.github.io/zeronvs/",cs.CV,"['cs.CV', 'cs.GR']"
+Efficient Privacy-Preserving Visual Localization Using 3D Ray Clouds,Heejoon Moon · Chunghwan Lee · Je Hyeong Hong,https://github.com/PHANTOM0122/Ray-cloud,,https://ieeexplore.ieee.org/abstract/document/10203590,,,,,nan
+BigGait: Learning Gait Representation You Want by Large Vision Models,Dingqiang Ye · Chao Fan · Jingzhe Ma · Xiaoming Liu · Shiqi Yu,https://github.com/ShiqiYu/OpenGait,https://arxiv.org/abs/2402.19122,,,BigGait: Learning Gait Representation You Want by Large Vision Models,"Gait recognition stands as one of the most pivotal remote identification
+technologies and progressively expands across research and industry
+communities. However, existing gait recognition methods heavily rely on
+task-specific upstream driven by supervised learning to provide explicit gait
+representations like silhouette sequences, which inevitably introduce expensive
+annotation costs and potential error accumulation. Escaping from this trend,
+this work explores effective gait representations based on the all-purpose
+knowledge produced by task-agnostic Large Vision Models (LVMs) and proposes a
+simple yet efficient gait framework, termed BigGait. Specifically, the Gait
+Representation Extractor (GRE) within BigGait draws upon design principles from
+established gait representations, effectively transforming all-purpose
+knowledge into implicit gait representations without requiring third-party
+supervision signals. Experiments on CCPG, CAISA-B* and SUSTech1K indicate that
+BigGait significantly outperforms the previous methods in both within-domain
+and cross-domain tasks in most cases, and provides a more practical paradigm
+for learning the next-generation gait representation. Finally, we delve into
+prospective challenges and promising directions in LVMs-based gait recognition,
+aiming to inspire future work in this emerging topic. The source code is
+available at https://github.com/ShiqiYu/OpenGait.",cs.CV,['cs.CV']
+ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding,Le Xue · Ning Yu · Shu Zhang · Artemis Panagopoulou · Junnan Li · Roberto Martín-Martín · Jiajun Wu · Caiming Xiong · Ran Xu · Juan Carlos Niebles · Silvio Savarese, ,https://ar5iv.labs.arxiv.org/html/2305.08275,,2305.08275.pdf,ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding,"Recent advancements in multimodal pre-training have shown promising efficacy
+in 3D representation learning by aligning multimodal features across 3D shapes,
+their 2D counterparts, and language descriptions. However, the methods used by
+existing frameworks to curate such multimodal data, in particular language
+descriptions for 3D shapes, are not scalable, and the collected language
+descriptions are not diverse. To address this, we introduce ULIP-2, a simple
+yet effective tri-modal pre-training framework that leverages large multimodal
+models to automatically generate holistic language descriptions for 3D shapes.
+It only needs 3D data as input, eliminating the need for any manual 3D
+annotations, and is therefore scalable to large datasets. ULIP-2 is also
+equipped with scaled-up backbones for better multimodal representation
+learning. We conduct experiments on two large-scale 3D datasets, Objaverse and
+ShapeNet, and augment them with tri-modal datasets of 3D point clouds, images,
+and language for training ULIP-2. Experiments show that ULIP-2 demonstrates
+substantial benefits in three downstream tasks: zero-shot 3D classification,
+standard 3D classification with fine-tuning, and 3D captioning (3D-to-language
+generation). It achieves a new SOTA of 50.6% (top-1) on Objaverse-LVIS and
+84.7% (top-1) on ModelNet40 in zero-shot classification. In the ScanObjectNN
+benchmark for standard fine-tuning, ULIP-2 reaches an overall accuracy of 91.5%
+with a compact model of only 1.4 million parameters. ULIP-2 sheds light on a
+new paradigm for scalable multimodal 3D representation learning without human
+annotations and shows significant improvements over existing baselines. The
+code and datasets are released at https://github.com/salesforce/ULIP.",cs.CV,['cs.CV']
+On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?,Maxime Zanella · Ismail Ben Ayed,https://github.com/MaxZanella/MTA,https://arxiv.org/abs/2405.02266,,2405.02266.pdf,On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?,"The development of large vision-language models, notably CLIP, has catalyzed
+research into effective adaptation techniques, with a particular focus on soft
+prompt tuning. Conjointly, test-time augmentation, which utilizes multiple
+augmented views of a single image to enhance zero-shot generalization, is
+emerging as a significant area of interest. This has predominantly directed
+research efforts toward test-time prompt tuning. In contrast, we introduce a
+robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based
+methods without requiring this intensive training procedure. This positions MTA
+as an ideal solution for both standalone and API-based applications.
+Additionally, our method does not rely on ad hoc rules (e.g., confidence
+threshold) used in some previous test-time augmentation techniques to filter
+the augmented views. Instead, MTA incorporates a quality assessment variable
+for each view directly into its optimization process, termed as the inlierness
+score. This score is jointly optimized with a density mode seeking process,
+leading to an efficient training- and hyperparameter-free approach. We
+extensively benchmark our method on 15 datasets and demonstrate MTA's
+superiority and computational efficiency. Deployed easily as plug-and-play
+module on top of zero-shot models and state-of-the-art few-shot methods, MTA
+shows systematic and consistent improvements.",cs.CV,['cs.CV']
+Context-Guided Spatio-Temporal Video Grounding,Xin Gu · Heng Fan · Yan Huang · Tiejian Luo · Libo Zhang, ,https://arxiv.org/abs/2401.01578,,2401.01578.pdf,Context-Guided Spatio-Temporal Video Grounding,"Spatio-temporal video grounding (or STVG) task aims at locating a
+spatio-temporal tube for a specific instance given a text query. Despite
+advancements, current methods easily suffer the distractors or heavy object
+appearance variations in videos due to insufficient object information from the
+text, leading to degradation. Addressing this, we propose a novel framework,
+context-guided STVG (CG-STVG), which mines discriminative instance context for
+object in videos and applies it as a supplementary guidance for target
+localization. The key of CG-STVG lies in two specially designed modules,
+including instance context generation (ICG), which focuses on discovering
+visual context information (in both appearance and motion) of the instance, and
+instance context refinement (ICR), which aims to improve the instance context
+from ICG by eliminating irrelevant or even harmful information from the
+context. During grounding, ICG, together with ICR, are deployed at each
+decoding stage of a Transformer architecture for instance context learning.
+Particularly, instance context learned from one decoding stage is fed to the
+next stage, and leveraged as a guidance containing rich and discriminative
+object feature to enhance the target-awareness in decoding feature, which
+conversely benefits generating better new instance context for improving
+localization finally. Compared to existing methods, CG-STVG enjoys object
+information in text query and guidance from mined instance visual context for
+more accurate target localization. In our experiments on three benchmarks,
+including HCSTVG-v1/-v2 and VidSTG, CG-STVG sets new state-of-the-arts in
+m_tIoU and m_vIoU on all of them, showing its efficacy. The code will be
+released at https://github.com/HengLan/CGSTVG.",cs.CV,['cs.CV']
+GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions,Junjie Wang · Jiemin Fang · Xiaopeng Zhang · Lingxi Xie · Qi Tian, ,https://arxiv.org/abs/2311.16037,,2311.16037.pdf,GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions,"Recently, impressive results have been achieved in 3D scene editing with text
+instructions based on a 2D diffusion model. However, current diffusion models
+primarily generate images by predicting noise in the latent space, and the
+editing is usually applied to the whole image, which makes it challenging to
+perform delicate, especially localized, editing for 3D scenes. Inspired by
+recent 3D Gaussian splatting, we propose a systematic framework, named
+GaussianEditor, to edit 3D scenes delicately via 3D Gaussians with text
+instructions. Benefiting from the explicit property of 3D Gaussians, we design
+a series of techniques to achieve delicate editing. Specifically, we first
+extract the region of interest (RoI) corresponding to the text instruction,
+aligning it to 3D Gaussians. The Gaussian RoI is further used to control the
+editing process. Our framework can achieve more delicate and precise editing of
+3D scenes than previous methods while enjoying much faster training speed, i.e.
+within 20 minutes on a single V100 GPU, more than twice as fast as
+Instruct-NeRF2NeRF (45 minutes -- 2 hours).",cs.CV,"['cs.CV', 'cs.GR']"
+GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis,Shunyuan Zheng · Boyao ZHOU · Ruizhi Shao · Boning Liu · Shengping Zhang · Liqiang Nie · Yebin Liu,https://shunyuanzheng.github.io/GPS-Gaussian,https://arxiv.org/abs/2312.02155,,2312.02155.pdf,GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis,"We present a new approach, termed GPS-Gaussian, for synthesizing novel views
+of a character in a real-time manner. The proposed method enables 2K-resolution
+rendering under a sparse-view camera setting. Unlike the original Gaussian
+Splatting or neural implicit rendering methods that necessitate per-subject
+optimizations, we introduce Gaussian parameter maps defined on the source views
+and regress directly Gaussian Splatting properties for instant novel view
+synthesis without any fine-tuning or optimization. To this end, we train our
+Gaussian parameter regression module on a large amount of human scan data,
+jointly with a depth estimation module to lift 2D parameter maps to 3D space.
+The proposed framework is fully differentiable and experiments on several
+datasets demonstrate that our method outperforms state-of-the-art methods while
+achieving an exceeding rendering speed.",cs.CV,['cs.CV']
+OpenStreetView-5M: The Many Roads to Global Visual Geolocation,Guillaume Astruc · Nicolas Dufour · Ioannis Siglidis · Constantin Aronssohn · Nacim Bouia · Stephanie Fu · Romain Loiseau · Van Nguyen Nguyen · Charles Raude · Elliot Vincent · Lintao XU · Hongyu Zhou · Loic Landrieu,https://imagine.enpc.fr/~ioannis.siglidis/osv5m/,https://arxiv.org/abs/2404.18873v1,,2404.18873v1.pdf,OpenStreetView-5M: The Many Roads to Global Visual Geolocation,"Determining the location of an image anywhere on Earth is a complex visual
+task, which makes it particularly relevant for evaluating computer vision
+algorithms. Yet, the absence of standard, large-scale, open-access datasets
+with reliably localizable images has limited its potential. To address this
+issue, we introduce OpenStreetView-5M, a large-scale, open-access dataset
+comprising over 5.1 million geo-referenced street view images, covering 225
+countries and territories. In contrast to existing benchmarks, we enforce a
+strict train/test separation, allowing us to evaluate the relevance of learned
+geographical features beyond mere memorization. To demonstrate the utility of
+our dataset, we conduct an extensive benchmark of various state-of-the-art
+image encoders, spatial representations, and training strategies. All
+associated codes and models can be found at https://github.com/gastruc/osv5m.",cs.CV,"['cs.CV', 'cs.AI']"
+Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion,Zuoyue Li · Zhenqiang Li · Zhaopeng Cui · Marc Pollefeys · Martin R. Oswald, ,https://arxiv.org/abs/2401.10786,,2401.10786.pdf,Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion,"Directly generating scenes from satellite imagery offers exciting
+possibilities for integration into applications like games and map services.
+However, challenges arise from significant view changes and scene scale.
+Previous efforts mainly focused on image or video generation, lacking
+exploration into the adaptability of scene generation for arbitrary views.
+Existing 3D generation works either operate at the object level or are
+difficult to utilize the geometry obtained from satellite imagery. To overcome
+these limitations, we propose a novel architecture for direct 3D scene
+generation by introducing diffusion models into 3D sparse representations and
+combining them with neural rendering techniques. Specifically, our approach
+generates texture colors at the point level for a given geometry using a 3D
+diffusion model first, which is then transformed into a scene representation in
+a feed-forward manner. The representation can be utilized to render arbitrary
+views which would excel in both single-frame quality and inter-frame
+consistency. Experiments in two city-scale datasets show that our model
+demonstrates proficiency in generating photo-realistic street-view image
+sequences and cross-view urban scenes from satellite imagery.",cs.CV,['cs.CV']
+CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs,Haocheng Yuan · Jing Xu · Hao Pan · Adrien Bousseau · Niloy J. Mitra · Changjian Li,https://enigma-li.github.io/CADTalk/,https://arxiv.org/abs/2311.16703,,2311.16703.pdf,CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs,"CAD programs are a popular way to compactly encode shapes as a sequence of
+operations that are easy to parametrically modify. However, without sufficient
+semantic comments and structure, such programs can be challenging to
+understand, let alone modify. We introduce the problem of semantic commenting
+CAD programs, wherein the goal is to segment the input program into code blocks
+corresponding to semantically meaningful shape parts and assign a semantic
+label to each block. We solve the problem by combining program parsing with
+visual-semantic analysis afforded by recent advances in foundational language
+and vision models. Specifically, by executing the input programs, we create
+shapes, which we use to generate conditional photorealistic images to make use
+of semantic annotators for such images. We then distill the information across
+the images and link back to the original programs to semantically comment on
+them. Additionally, we collected and annotated a benchmark dataset, CADTalk,
+consisting of 5,288 machine-made programs and 45 human-made programs with
+ground truth semantic comments. We extensively evaluated our approach, compared
+it to a GPT-based baseline, and an open-set shape segmentation baseline, and
+reported an 83.24% accuracy on the new CADTalk dataset. Code and data:
+https://enigma-li.github.io/CADTalk/.",cs.CV,"['cs.CV', 'cs.GR']"
+MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning,Ahmed Agiza · Marina Neseem · Sherief Reda,https://github.com/scale-lab/MTLoRA,https://arxiv.org/abs/2403.20320,,2403.20320.pdf,MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning,"Adapting models pre-trained on large-scale datasets to a variety of
+downstream tasks is a common strategy in deep learning. Consequently,
+parameter-efficient fine-tuning methods have emerged as a promising way to
+adapt pre-trained models to different tasks while training only a minimal
+number of parameters. While most of these methods are designed for single-task
+adaptation, parameter-efficient training in Multi-Task Learning (MTL)
+architectures is still unexplored. In this paper, we introduce MTLoRA, a novel
+framework for parameter-efficient training of MTL models. MTLoRA employs
+Task-Agnostic and Task-Specific Low-Rank Adaptation modules, which effectively
+disentangle the parameter space in MTL fine-tuning, thereby enabling the model
+to adeptly handle both task specialization and interaction within MTL contexts.
+We applied MTLoRA to hierarchical-transformer-based MTL architectures, adapting
+them to multiple downstream dense prediction tasks. Our extensive experiments
+on the PASCAL dataset show that MTLoRA achieves higher accuracy on downstream
+tasks compared to fully fine-tuning the MTL model while reducing the number of
+trainable parameters by 3.6x. Furthermore, MTLoRA establishes a Pareto-optimal
+trade-off between the number of trainable parameters and the accuracy of the
+downstream tasks, outperforming current state-of-the-art parameter-efficient
+training methods in both accuracy and efficiency. Our code is publicly
+available.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+View-Category Interactive Sharing Transformer for Incomplete Multi-View Multi-Label Learning,Shilong Ou · Zhe Xue · Yawen Li · Meiyu Liang · Yuanqiang Cai · junjiang wu, ,https://arxiv.org/abs/2404.17340,,2404.17340.pdf,Masked Two-channel Decoupling Framework for Incomplete Multi-view Weak Multi-label Learning,"Multi-view learning has become a popular research topic in recent years, but
+research on the cross-application of classic multi-label classification and
+multi-view learning is still in its early stages. In this paper, we focus on
+the complex yet highly realistic task of incomplete multi-view weak multi-label
+learning and propose a masked two-channel decoupling framework based on deep
+neural networks to solve this problem. The core innovation of our method lies
+in decoupling the single-channel view-level representation, which is common in
+deep multi-view learning methods, into a shared representation and a
+view-proprietary representation. We also design a cross-channel contrastive
+loss to enhance the semantic property of the two channels. Additionally, we
+exploit supervised information to design a label-guided graph regularization
+loss, helping the extracted embedding features preserve the geometric structure
+among samples. Inspired by the success of masking mechanisms in image and text
+analysis, we develop a random fragment masking strategy for vector features to
+improve the learning ability of encoders. Finally, it is important to emphasize
+that our model is fully adaptable to arbitrary view and label absences while
+also performing well on the ideal full data. We have conducted sufficient and
+convincing experiments to confirm the effectiveness and advancement of our
+model.",cs.CV,['cs.CV']
+FineSports: A Multi-person Hierarchical Sports Video Dataset for Fine-grained Action Understanding,Jinglin Xu · Guohao Zhao · Sibo Yin · Wenhao Zhou · Yuxin Peng, ,,,,,,,nan
+Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation,Mohammad Amin Shabani · Zhaowen Wang · Difan Liu · Nanxuan Zhao · Jimei Yang · Yasutaka Furukawa,https://aminshabani.github.io/visual_layout_composer/index.html,https://web3.arxiv.org/abs/2402.04754,,2402.04754.pdf,Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints,"Controllable layout generation refers to the process of creating a plausible
+visual arrangement of elements within a graphic design (e.g., document and web
+designs) with constraints representing design intentions. Although recent
+diffusion-based models have achieved state-of-the-art FID scores, they tend to
+exhibit more pronounced misalignment compared to earlier transformer-based
+models. In this work, we propose the $\textbf{LA}$yout $\textbf{C}$onstraint
+diffusion mod$\textbf{E}$l (LACE), a unified model to handle a broad range of
+layout generation tasks, such as arranging elements with specified attributes
+and refining or completing a coarse layout design. The model is based on
+continuous diffusion models. Compared with existing methods that use discrete
+diffusion models, continuous state-space design can enable the incorporation of
+differentiable aesthetic constraint functions in training. For conditional
+generation, we introduce conditions via masked input. Extensive experiment
+results show that LACE produces high-quality layouts and outperforms existing
+state-of-the-art baselines.",cs.CV,"['cs.CV', 'cs.LG']"
+AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift Factor,Sudong Cai,https://github.com/SudongCAI/AdaShift,,https://www.nature.com/articles/s41598-024-60598-2,,,,,nan
+Generalized Event Cameras,Varun Sundar · Matthew Dutson · Andrei Ardelean · Claudio Bruschini · Edoardo Charbon · Mohit Gupta,https://wisionlab.com/project/generalized-event-cameras/,,https://aim.autm.net/public/project/73780/,,,,,nan
+DiLiGenRT: A Photometric Stereo Dataset with Quantified Roughness and Translucency,Heng Guo · Jieji Ren · Feishi Wang · Boxin Shi · Mingjun Ren · Yasuyuki Matsushita, ,,,,,,,nan
+Instantaneous Perception of Moving Objects in 3D,Di Liu · Bingbing Zhuang · Dimitris N. Metaxas · Manmohan Chandraker, ,https://arxiv.org/abs/2405.02781,,2405.02781.pdf,Instantaneous Perception of Moving Objects in 3D,"The perception of 3D motion of surrounding traffic participants is crucial
+for driving safety. While existing works primarily focus on general large
+motions, we contend that the instantaneous detection and quantification of
+subtle motions is equally important as they indicate the nuances in driving
+behavior that may be safety critical, such as behaviors near a stop sign of
+parking positions. We delve into this under-explored task, examining its unique
+challenges and developing our solution, accompanied by a carefully designed
+benchmark. Specifically, due to the lack of correspondences between consecutive
+frames of sparse Lidar point clouds, static objects might appear to be moving -
+the so-called swimming effect. This intertwines with the true object motion,
+thereby posing ambiguity in accurate estimation, especially for subtle motions.
+To address this, we propose to leverage local occupancy completion of object
+point clouds to densify the shape cue, and mitigate the impact of swimming
+artifacts. The occupancy completion is learned in an end-to-end fashion
+together with the detection of moving objects and the estimation of their
+motion, instantaneously as soon as objects start to move. Extensive experiments
+demonstrate superior performance compared to standard 3D motion estimation
+approaches, particularly highlighting our method's specialized treatment of
+subtle motions.",cs.CV,['cs.CV']
+eTraM: Event-based Traffic Monitoring Dataset,Aayush Atul Verma · Bharatesh Chakravarthi · Arpitsinh Vaghela · Hua Wei · 'YZ' Yezhou Yang,https://eventbasedvision.github.io/eTraM/,https://arxiv.org/abs/2403.19976,,2403.19976.pdf,eTraM: Event-based Traffic Monitoring Dataset,"Event cameras, with their high temporal and dynamic range and minimal memory
+usage, have found applications in various fields. However, their potential in
+static traffic monitoring remains largely unexplored. To facilitate this
+exploration, we present eTraM - a first-of-its-kind, fully event-based traffic
+monitoring dataset. eTraM offers 10 hr of data from different traffic scenarios
+in various lighting and weather conditions, providing a comprehensive overview
+of real-world situations. Providing 2M bounding box annotations, it covers
+eight distinct classes of traffic participants, ranging from vehicles to
+pedestrians and micro-mobility. eTraM's utility has been assessed using
+state-of-the-art methods for traffic participant detection, including RVT, RED,
+and YOLOv8. We quantitatively evaluate the ability of event-based models to
+generalize on nighttime and unseen scenes. Our findings substantiate the
+compelling potential of leveraging event cameras for traffic monitoring,
+opening new avenues for research and application. eTraM is available at
+https://eventbasedvision.github.io/eTraM",cs.CV,['cs.CV']
+Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation,Hang Li · Chengzhi Shen · Philip H.S. Torr · Volker Tresp · Jindong Gu, ,https://arxiv.org/abs/2311.17216,,2311.17216.pdf,Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation,"Diffusion-based models have gained significant popularity for text-to-image
+generation due to their exceptional image-generation capabilities. A risk with
+these models is the potential generation of inappropriate content, such as
+biased or harmful images. However, the underlying reasons for generating such
+undesired content from the perspective of the diffusion model's internal
+representation remain unclear. Previous work interprets vectors in an
+interpretable latent space of diffusion models as semantic concepts. However,
+existing approaches cannot discover directions for arbitrary concepts, such as
+those related to inappropriate concepts. In this work, we propose a novel
+self-supervised approach to find interpretable latent directions for a given
+concept. With the discovered vectors, we further propose a simple approach to
+mitigate inappropriate generation. Extensive experiments have been conducted to
+verify the effectiveness of our mitigation approach, namely, for fair
+generation, safe generation, and responsible text-enhancing generation. Project
+page: \url{https://interpretdiffusion.github.io}.",cs.CV,['cs.CV']
+GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction,Xiao Chen · Quanyi Li · Tai Wang · Tianfan Xue · Jiangmiao Pang, ,https://arxiv.org/abs/2402.16174,,2402.16174.pdf,GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction,"While recent advances in neural radiance field enable realistic digitization
+for large-scale scenes, the image-capturing process is still time-consuming and
+labor-intensive. Previous works attempt to automate this process using the
+Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing
+NBV policies heavily rely on hand-crafted criteria, limited action space, or
+per-scene optimized representations. These constraints limit their
+cross-dataset generalizability. To overcome them, we propose GenNBV, an
+end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning
+(RL)-based framework and extends typical limited action space to 5D free space.
+It empowers our agent drone to scan from any viewpoint, and even interact with
+unseen geometries during training. To boost the cross-dataset generalizability,
+we also propose a novel multi-source state embedding, including geometric,
+semantic, and action representations. We establish a benchmark using the Isaac
+Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV
+policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12%
+coverage ratio on unseen building-scale objects from these datasets,
+respectively, outperforming prior solutions.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation,Agneet Chatterjee · Tejas Gokhale · Chitta Baral · 'YZ' Yezhou Yang,https://agneetchatterjee.com/robustness_depth_lang/,https://arxiv.org/abs/2404.08540,,2404.08540.pdf,On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation,"Recent advances in monocular depth estimation have been made by incorporating
+natural language as additional guidance. Although yielding impressive results,
+the impact of the language prior, particularly in terms of generalization and
+robustness, remains unexplored. In this paper, we address this gap by
+quantifying the impact of this prior and introduce methods to benchmark its
+effectiveness across various settings. We generate ""low-level"" sentences that
+convey object-centric, three-dimensional spatial relationships, incorporate
+them as additional language priors and evaluate their downstream impact on
+depth estimation. Our key finding is that current language-guided depth
+estimators perform optimally only with scene-level descriptions and
+counter-intuitively fare worse with low level descriptions. Despite leveraging
+additional data, these methods are not robust to directed adversarial attacks
+and decline in performance with an increase in distribution shift. Finally, to
+provide a foundation for future research, we identify points of failures and
+offer insights to better understand these shortcomings. With an increasing
+number of methods using language for depth estimation, our findings highlight
+the opportunities and pitfalls that require careful consideration for effective
+deployment in real-world settings",cs.CV,['cs.CV']
+Towards a Perceptual Evaluation Framework for Lighting Estimation,Justine Giroux · Mohammad Reza Karimi Dastjerdi · Yannick Hold-Geoffroy · Javier Vazquez-Corral · Jean-François Lalonde, ,https://arxiv.org/abs/2312.04334,,2312.04334.pdf,Towards a Perceptual Evaluation Framework for Lighting Estimation,"Progress in lighting estimation is tracked by computing existing image
+quality assessment (IQA) metrics on images from standard datasets. While this
+may appear to be a reasonable approach, we demonstrate that doing so does not
+correlate to human preference when the estimated lighting is used to relight a
+virtual scene into a real photograph. To study this, we design a controlled
+psychophysical experiment where human observers must choose their preference
+amongst rendered scenes lit using a set of lighting estimation algorithms
+selected from the recent literature, and use it to analyse how these algorithms
+perform according to human perception. Then, we demonstrate that none of the
+most popular IQA metrics from the literature, taken individually, correctly
+represent human perception. Finally, we show that by learning a combination of
+existing IQA metrics, we can more accurately represent human preference. This
+provides a new perceptual framework to help evaluate future lighting estimation
+algorithms.",cs.CV,['cs.CV']
+HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video,Zicong Fan · Maria Parelli · Maria Kadoglou · Xu Chen · Muhammed Kocabas · Michael J. Black · Otmar Hilliges,https://zc-alexfan.github.io/hold,https://arxiv.org/abs/2311.18448v1,,2311.18448v1.pdf,HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video,"Since humans interact with diverse objects every day, the holistic 3D capture
+of these interactions is important to understand and model human behaviour.
+However, most existing methods for hand-object reconstruction from RGB either
+assume pre-scanned object templates or heavily rely on limited 3D hand-object
+data, restricting their ability to scale and generalize to more unconstrained
+interaction settings. To this end, we introduce HOLD -- the first
+category-agnostic method that reconstructs an articulated hand and object
+jointly from a monocular interaction video. We develop a compositional
+articulated implicit model that can reconstruct disentangled 3D hand and object
+from 2D images. We also further incorporate hand-object constraints to improve
+hand-object poses and consequently the reconstruction quality. Our method does
+not rely on 3D hand-object annotations while outperforming fully-supervised
+baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we
+qualitatively show its robustness in reconstructing from in-the-wild videos.
+Code: https://github.com/zc-alexfan/hold",cs.CV,['cs.CV']
+Language-driven All-in-one Adverse Weather Removal,Hao Yang · Liyuan Pan · Yan Yang · Wei Liang, ,https://arxiv.org/abs/2312.01381,,2312.01381.pdf,Language-driven All-in-one Adverse Weather Removal,"All-in-one (AiO) frameworks restore various adverse weather degradations with
+a single set of networks jointly. To handle various weather conditions, an AiO
+framework is expected to adaptively learn weather-specific knowledge for
+different degradations and shared knowledge for common patterns. However,
+existing methods: 1) rely on extra supervision signals, which are usually
+unknown in real-world applications; 2) employ fixed network structures, which
+restrict the diversity of weather-specific knowledge. In this paper, we propose
+a Language-driven Restoration framework (LDR) to alleviate the aforementioned
+issues. First, we leverage the power of pre-trained vision-language (PVL)
+models to enrich the diversity of weather-specific knowledge by reasoning about
+the occurrence, type, and severity of degradation, generating description-based
+degradation priors. Then, with the guidance of degradation prior, we sparsely
+select restoration experts from a candidate list dynamically based on a
+Mixture-of-Experts (MoE) structure. This enables us to adaptively learn the
+weather-specific and shared knowledge to handle various weather conditions
+(e.g., unknown or mixed weather). Experiments on extensive restoration
+scenarios show our superior performance (see Fig. 1). The source code will be
+made available.",cs.CV,['cs.CV']
+Gaussian Splatting SLAM,Hidenobu Matsuki · Riku Murai · Paul Kelly · Andrew J. Davison,https://rmurai.co.uk/projects/GaussianSplattingSLAM/,https://arxiv.org/abs/2312.06741,,2312.06741.pdf,Gaussian Splatting SLAM,"We present the first application of 3D Gaussian Splatting in monocular SLAM,
+the most fundamental but the hardest setup for Visual SLAM. Our method, which
+runs live at 3fps, utilises Gaussians as the only 3D representation, unifying
+the required representation for accurate, efficient tracking, mapping, and
+high-quality rendering. Designed for challenging monocular settings, our
+approach is seamlessly extendable to RGB-D SLAM when an external depth sensor
+is available. Several innovations are required to continuously reconstruct 3D
+scenes with high fidelity from a live camera. First, to move beyond the
+original 3DGS algorithm, which requires accurate poses from an offline
+Structure from Motion (SfM) system, we formulate camera tracking for 3DGS using
+direct optimisation against the 3D Gaussians, and show that this enables fast
+and robust tracking with a wide basin of convergence. Second, by utilising the
+explicit nature of the Gaussians, we introduce geometric verification and
+regularisation to handle the ambiguities occurring in incremental 3D dense
+reconstruction. Finally, we introduce a full SLAM system which not only
+achieves state-of-the-art results in novel view synthesis and trajectory
+estimation but also reconstruction of tiny and even transparent objects.",cs.CV,"['cs.CV', 'cs.RO']"
+Backdoor Defense via Test-Time Detecting and Repairing,Jiyang Guan · Jian Liang · Ran He, ,https://arxiv.org/abs/2308.06107,,2308.06107.pdf,Test-Time Backdoor Defense via Detecting and Repairing,"Deep neural networks have played a crucial part in many critical domains,
+such as autonomous driving, face recognition, and medical diagnosis. However,
+deep neural networks are facing security threats from backdoor attacks and can
+be manipulated into attacker-decided behaviors by the backdoor attacker. To
+defend the backdoor, prior research has focused on using clean data to remove
+backdoor attacks before model deployment. In this paper, we investigate the
+possibility of defending against backdoor attacks at test time by utilizing
+partially poisoned data to remove the backdoor from the model. To address the
+problem, a two-stage method Test-Time Backdoor Defense (TTBD) is proposed. In
+the first stage, we propose a backdoor sample detection method DDP to identify
+poisoned samples from a batch of mixed, partially poisoned samples. Once the
+poisoned samples are detected, we employ Shapley estimation to calculate the
+contribution of each neuron's significance in the network, locate the poisoned
+neurons, and prune them to remove backdoor in the models. Our experiments
+demonstrate that TTBD removes the backdoor successfully with only a batch of
+partially poisoned data across different model architectures and datasets
+against different types of backdoor attacks.",cs.CR,['cs.CR']
+XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies,Xuanchi Ren · Jiahui Huang · Xiaohui Zeng · Ken Museth · Sanja Fidler · Francis Williams, ,https://arxiv.org/abs/2312.03806,,2312.03806.pdf,XCube ($\mathcal{X}^3$): Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies,"We present $\mathcal{X}^3$ (pronounced XCube), a novel generative model for
+high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can
+generate millions of voxels with a finest effective resolution of up to
+$1024^3$ in a feed-forward fashion without time-consuming test-time
+optimization. To achieve this, we employ a hierarchical voxel latent diffusion
+model which generates progressively higher resolution grids in a coarse-to-fine
+manner using a custom framework built on the highly efficient VDB data
+structure. Apart from generating high-resolution objects, we demonstrate the
+effectiveness of XCube on large outdoor scenes at scales of 100m$\times$100m
+with a voxel size as small as 10cm. We observe clear qualitative and
+quantitative improvements over past approaches. In addition to unconditional
+generation, we show that our model can be used to solve a variety of tasks such
+as user-guided editing, scene completion from a single scan, and text-to-3D.
+More results and details can be found at
+https://research.nvidia.com/labs/toronto-ai/xcube/.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+MultiPhys: Multi-Person Physics-aware 3D Motion Estimation,Nicolás Ugrinovic · Boxiao Pan · Georgios Pavlakos · Despoina Paschalidou · Bokui Shen · Jordi Sanchez-Riera · Francesc Moreno-Noguer · Leonidas Guibas, ,https://arxiv.org/abs/2404.11987,,2404.11987.pdf,MultiPhys: Multi-Person Physics-aware 3D Motion Estimation,"We introduce MultiPhys, a method designed for recovering multi-person motion
+from monocular videos. Our focus lies in capturing coherent spatial placement
+between pairs of individuals across varying degrees of engagement. MultiPhys,
+being physically aware, exhibits robustness to jittering and occlusions, and
+effectively eliminates penetration issues between the two individuals. We
+devise a pipeline in which the motion estimated by a kinematic-based method is
+fed into a physics simulator in an autoregressive manner. We introduce distinct
+components that enable our model to harness the simulator's properties without
+compromising the accuracy of the kinematic estimates. This results in final
+motion estimates that are both kinematically coherent and physically compliant.
+Extensive evaluations on three challenging datasets characterized by
+substantial inter-person interaction show that our method significantly reduces
+errors associated with penetration and foot skating, while performing
+competitively with the state-of-the-art on motion accuracy and smoothness.
+Results and code can be found on our project page
+(http://www.iri.upc.edu/people/nugrinovic/multiphys/).",cs.CV,['cs.CV']
+Implicit Motion Function,Yue Gao · Jiahao Li · Lei Chu · Yan Lu, ,,https://ieeexplore.ieee.org/document/10378136/citations?tabFilter=papers,,,,,nan
+Improving Generalized Zero-Shot Learning by Exploring the Diverse Semantics from  External Class Names,Yapeng Li · Yong Luo · Zengmao Wang · Bo Du, ,,https://ieeexplore.ieee.org/document/10283906,,,,,nan
+Unsupervised 3D Structure Inference from Category-Specific Image Collections,Weikang Wang · Dongliang Cao · Florian Bernard,https://wei-kang-wang.github.io/unsuper3Dstructure/,,,,,,,nan
+Text-Driven Image Editing via Learnable Regions,Yuanze Lin · Yi-Wen Chen · Yi-Hsuan Tsai · Lu Jiang · Ming-Hsuan Yang,https://yuanze-lin.me/LearnableRegions_page/,https://arxiv.org/abs/2311.16432,,2311.16432.pdf,Text-Driven Image Editing via Learnable Regions,"Language has emerged as a natural interface for image editing. In this paper,
+we introduce a method for region-based image editing driven by textual prompts,
+without the need for user-provided masks or sketches. Specifically, our
+approach leverages an existing pre-trained text-to-image model and introduces a
+bounding box generator to identify the editing regions that are aligned with
+the textual prompts. We show that this simple approach enables flexible editing
+that is compatible with current image generation models, and is able to handle
+complex prompts featuring multiple objects, complex sentences, or lengthy
+paragraphs. We conduct an extensive user study to compare our method against
+state-of-the-art methods. The experiments demonstrate the competitive
+performance of our method in manipulating images with high fidelity and realism
+that correspond to the provided language descriptions. Our project webpage can
+be found at: https://yuanze-lin.me/LearnableRegions_page.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Balancing Act: Distribution-Guided Debiasing in Diffusion Models,Rishubh Parihar · Abhijnya Bhat · Abhipsa Basu · Saswat Mallick · Jogendra Kundu Kundu · R. Venkatesh Babu, ,https://arxiv.org/abs/2402.18206,,2402.18206.pdf,Balancing Act: Distribution-Guided Debiasing in Diffusion Models,"Diffusion Models (DMs) have emerged as powerful generative models with
+unprecedented image generation capability. These models are widely used for
+data augmentation and creative applications. However, DMs reflect the biases
+present in the training datasets. This is especially concerning in the context
+of faces, where the DM prefers one demographic subgroup vs others (eg. female
+vs male). In this work, we present a method for debiasing DMs without relying
+on additional data or model retraining. Specifically, we propose Distribution
+Guidance, which enforces the generated images to follow the prescribed
+attribute distribution. To realize this, we build on the key insight that the
+latent features of denoising UNet hold rich demographic semantics, and the same
+can be leveraged to guide debiased generation. We train Attribute Distribution
+Predictor (ADP) - a small mlp that maps the latent features to the distribution
+of attributes. ADP is trained with pseudo labels generated from existing
+attribute classifiers. The proposed Distribution Guidance with ADP enables us
+to do fair generation. Our method reduces bias across single/multiple
+attributes and outperforms the baseline by a significant margin for
+unconditional and text-conditional diffusion models. Further, we present a
+downstream task of training a fair attribute classifier by rebalancing the
+training set with our generated data.",cs.CV,['cs.CV']
+Close Imitation of Expert Retouching for Black-and-White Photography,Seunghyun Shin · Jisu Shin · Jihwan Bae · Inwook Shim · Hae-Gon Jeon,https://github.com/seunghyuns98/Decolorization,,https://retouchinglabs.com/retouching-black-and-white-photos/,,,,,nan
+Generative Image Dynamics,Zhengqi Li · Richard Tucker · Noah Snavely · Aleksander Holynski, ,https://arxiv.org/abs/2309.07906,,2309.07906.pdf,Generative Image Dynamics,"We present an approach to modeling an image-space prior on scene motion. Our
+prior is learned from a collection of motion trajectories extracted from real
+video sequences depicting natural, oscillatory dynamics such as trees, flowers,
+candles, and clothes swaying in the wind. We model this dense, long-term motion
+prior in the Fourier domain:given a single image, our trained model uses a
+frequency-coordinated diffusion sampling process to predict a spectral volume,
+which can be converted into a motion texture that spans an entire video. Along
+with an image-based rendering module, these trajectories can be used for a
+number of downstream applications, such as turning still images into seamlessly
+looping videos, or allowing users to realistically interact with objects in
+real pictures by interpreting the spectral volumes as image-space modal bases,
+which approximate object dynamics.",cs.CV,['cs.CV']
+RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control,xiang deng · Zerong Zheng · Yuxiang Zhang · Jingxiang Sun · Chao Xu · Xiaodong Yang · Lizhen Wang · Yebin Liu, ,https://arxiv.org/html/2303.10275v2,,2303.10275v2.pdf,MoRF: Mobile Realistic Fullbody Avatars from a Monocular Video,"We present a system to create Mobile Realistic Fullbody (MoRF) avatars. MoRF
+avatars are rendered in real-time on mobile devices, learned from monocular
+videos, and have high realism. We use SMPL-X as a proxy geometry and render it
+with DNR (neural texture and image-2-image network). We improve on prior work,
+by overfitting per-frame warping fields in the neural texture space, allowing
+to better align the training signal between different frames. We also refine
+SMPL-X mesh fitting procedure to improve the overall avatar quality. In the
+comparisons to other monocular video-based avatar systems, MoRF avatars achieve
+higher image sharpness and temporal consistency. Participants of our user study
+also preferred avatars generated by MoRF.",cs.CV,['cs.CV']
+SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation,Zhixuan Liu · Peter Schaldenbrand · Beverley-Claire Okogwu · Wenxuan Peng · Youngsik Yun · Andrew Hundt · Jihie Kim · Jean Oh,ariannaliu.github.io/SCoFT/,https://arxiv.org/abs/2401.08053,,2401.08053.pdf,SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation,"Accurate representation in media is known to improve the well-being of the
+people who consume it. Generative image models trained on large web-crawled
+datasets such as LAION are known to produce images with harmful stereotypes and
+misrepresentations of cultures. We improve inclusive representation in
+generated images by (1) engaging with communities to collect a culturally
+representative dataset that we call the Cross-Cultural Understanding Benchmark
+(CCUB) and (2) proposing a novel Self-Contrastive Fine-Tuning (SCoFT) method
+that leverages the model's known biases to self-improve. SCoFT is designed to
+prevent overfitting on small datasets, encode only high-level information from
+the data, and shift the generated distribution away from misrepresentations
+encoded in a pretrained model. Our user study conducted on 51 participants from
+5 different countries based on their self-selected national cultural
+affiliation shows that fine-tuning on CCUB consistently generates images with
+higher cultural relevance and fewer stereotypes when compared to the Stable
+Diffusion baseline, which is further improved with our SCoFT technique.",cs.CV,['cs.CV']
+Rendering Every Pixel for High-Fidelity Geometry in 3D GANs,Alex Trevithick · Matthew Chan · Towaki Takikawa · Umar Iqbal · Shalini De Mello · Manmohan Chandraker · Ravi Ramamoorthi · Koki Nagano, ,https://arxiv.org/abs/2401.02411,,2401.02411.pdf,What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs,"3D-aware Generative Adversarial Networks (GANs) have shown remarkable
+progress in learning to generate multi-view-consistent images and 3D geometries
+of scenes from collections of 2D images via neural volume rendering. Yet, the
+significant memory and computational costs of dense sampling in volume
+rendering have forced 3D GANs to adopt patch-based training or employ
+low-resolution rendering with post-processing 2D super resolution, which
+sacrifices multiview consistency and the quality of resolved geometry.
+Consequently, 3D GANs have not yet been able to fully resolve the rich 3D
+geometry present in 2D images. In this work, we propose techniques to scale
+neural volume rendering to the much higher resolution of native 2D images,
+thereby resolving fine-grained 3D geometry with unprecedented detail. Our
+approach employs learning-based samplers for accelerating neural rendering for
+3D GAN training using up to 5 times fewer depth samples. This enables us to
+explicitly ""render every pixel"" of the full-resolution image during training
+and inference without post-processing superresolution in 2D. Together with our
+strategy to learn high-quality surface geometry, our method synthesizes
+high-resolution 3D geometry and strictly view-consistent images while
+maintaining image quality on par with baselines relying on post-processing
+super resolution. We demonstrate state-of-the-art 3D gemetric quality on FFHQ
+and AFHQ, setting a new standard for unsupervised learning of 3D shapes in 3D
+GANs.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG']"
+An Interactive Navigation Method with Effect-oriented Affordance,Xiaohan Wang · Yuehu LIU · Xinhang Song · Yuyi Liu · Sixian Zhang · Shuqiang Jiang, ,https://arxiv.org/abs/2310.08873,,2310.08873.pdf,Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Models,"This paper proposes an interactive navigation framework by using large
+language and vision-language models, allowing robots to navigate in
+environments with traversable obstacles. We utilize the large language model
+(GPT-3.5) and the open-set Vision-language Model (Grounding DINO) to create an
+action-aware costmap to perform effective path planning without fine-tuning.
+With the large models, we can achieve an end-to-end system from textual
+instructions like ""Can you pass through the curtains to deliver medicines to
+me?"", to bounding boxes (e.g., curtains) with action-aware attributes. They can
+be used to segment LiDAR point clouds into two parts: traversable and
+untraversable parts, and then an action-aware costmap is constructed for
+generating a feasible path. The pre-trained large models have great
+generalization ability and do not require additional annotated data for
+training, allowing fast deployment in the interactive navigation tasks. We
+choose to use multiple traversable objects such as curtains and grasses for
+verification by instructing the robot to traverse them. Besides, traversing
+curtains in a medical scenario was tested. All experimental results
+demonstrated the proposed framework's effectiveness and adaptability to diverse
+environments.",cs.RO,"['cs.RO', 'cs.AI']"
+Communication-Efficient Federated Learning with Accelerated Client Gradient,Geeho Kim · Jinkyu Kim · Bohyung Han, ,,https://openreview.net/forum?id=qwymfs6cKe,,,,,nan
+InceptionNeXt: When Inception Meets ConvNeXt,Weihao Yu · Pan Zhou · Shuicheng Yan · Xinchao Wang,https://github.com/sail-sg/inceptionnext,,https://dblp.org/rec/journals/corr/abs-2303-16900,,,,,nan
+MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation,Petru-Daniel Tudosiu · Yongxin Yang · Shifeng Zhang · Fei Chen · Steven McDonagh · Gerasimos Lampouras · Ignacio Iacobacci · Sarah Parisot,https://mulan-dataset.github.io/,https://arxiv.org/abs/2404.02790,,2404.02790.pdf,MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation,"Text-to-image generation has achieved astonishing results, yet precise
+spatial controllability and prompt fidelity remain highly challenging. This
+limitation is typically addressed through cumbersome prompt engineering, scene
+layout conditioning, or image editing techniques which often require hand drawn
+masks. Nonetheless, pre-existing works struggle to take advantage of the
+natural instance-level compositionality of scenes due to the typically flat
+nature of rasterized RGB output images. Towards adressing this challenge, we
+introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of
+RGB images as multilayer, instance-wise RGBA decompositions, and over 100K
+instance images. To build MuLAn, we developed a training free pipeline which
+decomposes a monocular RGB image into a stack of RGBA layers comprising of
+background and isolated instances. We achieve this through the use of
+pretrained general-purpose models, and by developing three modules: image
+decomposition for instance discovery and extraction, instance completion to
+reconstruct occluded areas, and image re-assembly. We use our pipeline to
+create MuLAn-COCO and MuLAn-LAION datasets, which contain a variety of image
+decompositions in terms of style, composition and complexity. With MuLAn, we
+provide the first photorealistic resource providing instance decomposition and
+occlusion information for high quality images, opening up new avenues for
+text-to-image generative AI research. With this, we aim to encourage the
+development of novel generation and editing technology, in particular
+layer-wise solutions. MuLAn data resources are available at
+https://MuLAn-dataset.github.io/.",cs.CV,['cs.CV']
+Ink Dot-Oriented Differentiable Optimization for Neural Image Halftoning,Hao Jiang · Bingfeng Zhou · Yadong Mu, ,,https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/ipr2.12998,,,,,nan
+On the Scalability of Diffusion-based Text-to-Image Generation,Hao Li · Yang Zou · Ying Wang · Orchid Majumder · Yusheng Xie · R. Manmatha · Ashwin Swaminathan · Zhuowen Tu · Stefano Ermon · Stefano Soatto, ,https://arxiv.org/abs/2404.02883,,2404.02883.pdf,On the Scalability of Diffusion-based Text-to-Image Generation,"Scaling up model and data size has been quite successful for the evolution of
+LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
+models is not fully explored. It is also unclear how to efficiently scale the
+model for better performance at reduced cost. The different training settings
+and expensive training cost make a fair model comparison extremely difficult.
+In this work, we empirically study the scaling properties of diffusion based
+T2I models by performing extensive and rigours ablations on scaling both
+denoising backbones and training set, including training scaled UNet and
+Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
+images. For model scaling, we find the location and amount of cross attention
+distinguishes the performance of existing UNet designs. And increasing the
+transformer blocks is more parameter-efficient for improving text-image
+alignment than increasing channel numbers. We then identify an efficient UNet
+variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
+scaling side, we show the quality and diversity of the training set matters
+more than simply dataset size. Increasing caption density and diversity
+improves text-image alignment performance and the learning efficiency. Finally,
+we provide scaling functions to predict the text-image alignment performance as
+functions of the scale of model size, compute and dataset size.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+G-FARS: Gradient-Field-based Auto-Regressive Sampling for 3D Part Grouping,Junfeng Cheng · Tania Stathaki,https://github.com/J-F-Cheng/G-FARS-3DPartGrouping,https://arxiv.org/abs/2405.06828,,2405.06828.pdf,G-FARS: Gradient-Field-based Auto-Regressive Sampling for 3D Part Grouping,"This paper proposes a novel task named ""3D part grouping"". Suppose there is a
+mixed set containing scattered parts from various shapes. This task requires
+algorithms to find out every possible combination among all the parts. To
+address this challenge, we propose the so called Gradient Field-based
+Auto-Regressive Sampling framework (G-FARS) tailored specifically for the 3D
+part grouping task. In our framework, we design a gradient-field-based
+selection graph neural network (GNN) to learn the gradients of a log
+conditional probability density in terms of part selection, where the condition
+is the given mixed part set. This innovative approach, implemented through the
+gradient-field-based selection GNN, effectively captures complex relationships
+among all the parts in the input. Upon completion of the training process, our
+framework becomes capable of autonomously grouping 3D parts by iteratively
+selecting them from the mixed part set, leveraging the knowledge acquired by
+the trained gradient-field-based selection GNN. Our code is available at:
+https://github.com/J-F-Cheng/G-FARS-3DPartGrouping.",cs.CV,['cs.CV']
+Unsupervised Salient Instance Detection,Xin Tian · Ke Xu · Rynson W.H. Lau, ,https://arxiv.org/abs/2404.14759,,2404.14759.pdf,Unified Unsupervised Salient Object Detection via Knowledge Transfer,"Recently, unsupervised salient object detection (USOD) has gained increasing
+attention due to its annotation-free nature. However, current methods mainly
+focus on specific tasks such as RGB and RGB-D, neglecting the potential for
+task migration. In this paper, we propose a unified USOD framework for generic
+USOD tasks. Firstly, we propose a Progressive Curriculum Learning-based
+Saliency Distilling (PCL-SD) mechanism to extract saliency cues from a
+pre-trained deep network. This mechanism starts with easy samples and
+progressively moves towards harder ones, to avoid initial interference caused
+by hard samples. Afterwards, the obtained saliency cues are utilized to train a
+saliency detector, and we employ a Self-rectify Pseudo-label Refinement (SPR)
+mechanism to improve the quality of pseudo-labels. Finally, an adapter-tuning
+method is devised to transfer the acquired saliency knowledge, leveraging
+shared knowledge to attain superior transferring performance on the target
+tasks. Extensive experiments on five representative SOD tasks confirm the
+effectiveness and feasibility of our proposed method. Code and supplement
+materials are available at https://github.com/I2-Multimedia-Lab/A2S-v3.",cs.CV,['cs.CV']
+TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion,Yu-Ying Yeh · Jia-Bin Huang · Changil Kim · Lei Xiao · Thu Nguyen-Phuoc · Numair Khan · Cheng Zhang · Manmohan Chandraker · Carl Marshall · Zhao Dong · Zhengqin Li, ,,https://huggingface.co/papers/2401.09416,,,,,nan
+RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models,Ozgur Kara · Bariscan Kurtkaya · Hidir Yesiltepe · James Rehg · Pinar Yanardag,https://rave-video.github.io/,https://arxiv.org/abs/2312.04524,,2312.04524.pdf,RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models,"Recent advancements in diffusion-based models have demonstrated significant
+success in generating images from text. However, video editing models have not
+yet reached the same level of visual quality and user control. To address this,
+we introduce RAVE, a zero-shot video editing method that leverages pre-trained
+text-to-image diffusion models without additional training. RAVE takes an input
+video and a text prompt to produce high-quality videos while preserving the
+original motion and semantic structure. It employs a novel noise shuffling
+strategy, leveraging spatio-temporal interactions between frames, to produce
+temporally consistent videos faster than existing methods. It is also efficient
+in terms of memory requirements, allowing it to handle longer videos. RAVE is
+capable of a wide range of edits, from local attribute modifications to shape
+transformations. In order to demonstrate the versatility of RAVE, we create a
+comprehensive video evaluation dataset ranging from object-focused scenes to
+complex human activities like dancing and typing, and dynamic scenes featuring
+swimming fish and boats. Our qualitative and quantitative experiments highlight
+the effectiveness of RAVE in diverse video editing scenarios compared to
+existing methods. Our code, dataset and videos can be found in
+https://rave-video.github.io.",cs.CV,['cs.CV']
+CosmicMan: A Text-to-Image Foundation Model for Humans,Shikai Li · Jianglin Fu · Kaiyuan Liu · Wentao Wang · Kwan-Yee Lin · Wayne Wu, ,http://export.arxiv.org/abs/2404.01294,,2404.01294.pdf,CosmicMan: A Text-to-Image Foundation Model for Humans,"We present CosmicMan, a text-to-image foundation model specialized for
+generating high-fidelity human images. Unlike current general-purpose
+foundation models that are stuck in the dilemma of inferior quality and
+text-image misalignment for humans, CosmicMan enables generating
+photo-realistic human images with meticulous appearance, reasonable structure,
+and precise text-image alignment with detailed dense descriptions. At the heart
+of CosmicMan's success are the new reflections and perspectives on data and
+models: (1) We found that data quality and a scalable data production flow are
+essential for the final results from trained models. Hence, we propose a new
+data production paradigm, Annotate Anyone, which serves as a perpetual data
+flywheel to produce high-quality data with accurate yet cost-effective
+annotations over time. Based on this, we constructed a large-scale dataset,
+CosmicMan-HQ 1.0, with 6 Million high-quality real-world human images in a mean
+resolution of 1488x1255, and attached with precise text annotations deriving
+from 115 Million attributes in diverse granularities. (2) We argue that a
+text-to-image foundation model specialized for humans must be pragmatic -- easy
+to integrate into down-streaming tasks while effective in producing
+high-quality human images. Hence, we propose to model the relationship between
+dense text descriptions and image pixels in a decomposed manner, and present
+Decomposed-Attention-Refocusing (Daring) training framework. It seamlessly
+decomposes the cross-attention features in existing text-to-image diffusion
+model, and enforces attention refocusing without adding extra modules. Through
+Daring, we show that explicitly discretizing continuous text space into several
+basic groups that align with human body structure is the key to tackling the
+misalignment problem in a breeze.",cs.CV,['cs.CV']
+Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts,Jiayi Chen · Benteng Ma · Hengfei Cui · Kwang-Ting Cheng · Yong Xia, ,https://arxiv.org/abs/2312.02567,,2312.02567.pdf,Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts,"Federated learning facilitates the collaborative learning of a global model
+across multiple distributed medical institutions without centralizing data.
+Nevertheless, the expensive cost of annotation on local clients remains an
+obstacle to effectively utilizing local data. To mitigate this issue, federated
+active learning methods suggest leveraging local and global model predictions
+to select a relatively small amount of informative local data for annotation.
+However, existing methods mainly focus on all local data sampled from the same
+domain, making them unreliable in realistic medical scenarios with domain
+shifts among different clients. In this paper, we make the first attempt to
+assess the informativeness of local data derived from diverse domains and
+propose a novel methodology termed Federated Evidential Active Learning (FEAL)
+to calibrate the data evaluation under domain shift. Specifically, we introduce
+a Dirichlet prior distribution in both local and global models to treat the
+prediction as a distribution over the probability simplex and capture both
+aleatoric and epistemic uncertainties by using the Dirichlet-based evidential
+model. Then we employ the epistemic uncertainty to calibrate the aleatoric
+uncertainty. Afterward, we design a diversity relaxation strategy to reduce
+data redundancy and maintain data diversity. Extensive experiments and analysis
+on five real multi-center medical image datasets demonstrate the superiority of
+FEAL over the state-of-the-art active learning methods in federated scenarios
+with domain shifts. The code will be available at
+https://github.com/JiayiChen815/FEAL.",cs.CV,['cs.CV']
+Riemannian Multinomial Logistics Regression for SPD Neural Networks,Ziheng Chen · Yue Song · Gaowen Liu · Ramana Kompella · Xiaojun Wu · Nicu Sebe,https://github.com/GitZH-Chen/SPDMLR.git,,https://openreview.net/forum?id=S0DUtGgkTM,,,,,nan
+Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval,Young Kyun Jang · Donghyun Kim · Zihang Meng · Dat Huynh · Ser-Nam Lim,https://youngkyunjang.github.io/VDG_project/,https://arxiv.org/abs/2404.15516,,2404.15516.pdf,Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval,"Composed Image Retrieval (CIR) is a task that retrieves images similar to a
+query, based on a provided textual modification. Current techniques rely on
+supervised learning for CIR models using labeled triplets of the reference
+image, text, target image. These specific triplets are not as commonly
+available as simple image-text pairs, limiting the widespread use of CIR and
+its scalability. On the other hand, zero-shot CIR can be relatively easily
+trained with image-caption pairs without considering the image-to-image
+relation, but this approach tends to yield lower accuracy. We propose a new
+semi-supervised CIR approach where we search for a reference and its related
+target images in auxiliary data and learn our large language model-based Visual
+Delta Generator (VDG) to generate text describing the visual difference (i.e.,
+visual delta) between the two. VDG, equipped with fluent language knowledge and
+being model agnostic, can generate pseudo triplets to boost the performance of
+CIR models. Our approach significantly improves the existing supervised
+learning approaches and achieves state-of-the-art results on the CIR
+benchmarks.",cs.CV,"['cs.CV', 'cs.AI']"
+LiDAR-based Person Re-identification,Wenxuan Guo · Zhiyu Pan · Yingping Liang · Ziheng Xi · Zhi Chen Zhong · Jianjiang Feng · Jie Zhou,https://github.com/GWxuan/ReID3D,https://arxiv.org/abs/2312.03033,,2312.03033.pdf,LiDAR-based Person Re-identification,"Camera-based person re-identification (ReID) systems have been widely applied
+in the field of public security. However, cameras often lack the perception of
+3D morphological information of human and are susceptible to various
+limitations, such as inadequate illumination, complex background, and personal
+privacy. In this paper, we propose a LiDAR-based ReID framework, ReID3D, that
+utilizes pre-training strategy to retrieve features of 3D body shape and
+introduces Graph-based Complementary Enhancement Encoder for extracting
+comprehensive features. Due to the lack of LiDAR datasets, we build LReID, the
+first LiDAR-based person ReID dataset, which is collected in several outdoor
+scenes with variations in natural conditions. Additionally, we introduce
+LReID-sync, a simulated pedestrian dataset designed for pre-training encoders
+with tasks of point cloud completion and shape parameter learning. Extensive
+experiments on LReID show that ReID3D achieves exceptional performance with a
+rank-1 accuracy of 94.0, highlighting the significant potential of LiDAR in
+addressing person ReID tasks. To the best of our knowledge, we are the first to
+propose a solution for LiDAR-based ReID. The code and datasets will be released
+soon.",cs.CV,['cs.CV']
+Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition,Zihan Wang · Siyang Song · Cheng Luo · Songhe Deng · Weicheng Xie · Linlin Shen,https://github.com/CVI-SZU/MDHR,https://arxiv.org/abs/2404.06443,,2404.06443.pdf,Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition,"Human facial action units (AUs) are mutually related in a hierarchical
+manner, as not only they are associated with each other in both spatial and
+temporal domains but also AUs located in the same/close facial regions show
+stronger relationships than those of different facial regions. While none of
+existing approach thoroughly model such hierarchical inter-dependencies among
+AUs, this paper proposes to comprehensively model multi-scale AU-related
+dynamic and hierarchical spatio-temporal relationship among AUs for their
+occurrences recognition. Specifically, we first propose a novel multi-scale
+temporal differencing network with an adaptive weighting block to explicitly
+capture facial dynamics across frames at different spatial scales, which
+specifically considers the heterogeneity of range and magnitude in different
+AUs' activation. Then, a two-stage strategy is introduced to hierarchically
+model the relationship among AUs based on their spatial distribution (i.e.,
+local and cross-region AU relationship modelling). Experimental results
+achieved on BP4D and DISFA show that our approach is the new state-of-the-art
+in the field of AU occurrence recognition. Our code is publicly available at
+https://github.com/CVI-SZU/MDHR.",cs.CV,['cs.CV']
+Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration,Shihao Zhou · Duosheng Chen · Jinshan Pan · Jinglei Shi · Jufeng Yang,https://github.com/joshyZhou/AST,https://arxiv.org/abs/2312.06874,,2312.06874.pdf,Dozerformer: Sequence Adaptive Sparse Transformer for Multivariate Time Series Forecasting,"Transformers have achieved remarkable performance in multivariate time
+series(MTS) forecasting due to their capability to capture long-term
+dependencies. However, the canonical attention mechanism has two key
+limitations: (1) its quadratic time complexity limits the sequence length, and
+(2) it generates future values from the entire historical sequence. To address
+this, we propose a Dozer Attention mechanism consisting of three sparse
+components: (1) Local, each query exclusively attends to keys within a
+localized window of neighboring time steps. (2) Stride, enables each query to
+attend to keys at predefined intervals. (3) Vary, allows queries to selectively
+attend to keys from a subset of the historical sequence. Notably, the size of
+this subset dynamically expands as forecasting horizons extend. Those three
+components are designed to capture essential attributes of MTS data, including
+locality, seasonality, and global temporal dependencies. Additionally, we
+present the Dozerformer Framework, incorporating the Dozer Attention mechanism
+for the MTS forecasting task. We evaluated the proposed Dozerformer framework
+with recent state-of-the-art methods on nine benchmark datasets and confirmed
+its superior performance. The code will be released after the manuscript is
+accepted.",cs.LG,"['cs.LG', 'cs.CL']"
+Circuit Design and Efficient Simulation of Quantum Inner Product and Empirical Studies of Its Effect on Near-Term Hybrid Quantum-Classic Machine Learning,Hao Xiong · Yehui Tang · Xinyu Ye · Junchi Yan,https://github.com/ShawXh/qip_cvpr24,https://arxiv.org/abs/2310.03978,,2310.03978.pdf,Efficient Quantum Circuit Simulation by Tensor Network Methods on Modern GPUs,"Efficient simulation of quantum circuits has become indispensable with the
+rapid development of quantum hardware. The primary simulation methods are based
+on state vectors and tensor networks. As the number of qubits and quantum gates
+grows larger in current quantum devices, traditional state-vector based quantum
+circuit simulation methods prove inadequate due to the overwhelming size of the
+Hilbert space and extensive entanglement. Consequently, brutal force tensor
+network simulation algorithms become the only viable solution in such
+scenarios. The two main challenges faced in tensor network simulation
+algorithms are optimal contraction path finding and efficient execution on
+modern computing devices, with the latter determines the actual efficiency. In
+this study, we investigate the optimization of such tensor network simulations
+on modern GPUs and propose general optimization strategies from two aspects:
+computational efficiency and accuracy. Firstly, we propose to transform
+critical Einstein summation operations into GEMM operations, leveraging the
+specific features of tensor network simulations to amplify the efficiency of
+GPUs. Secondly, by analyzing the data characteristics of quantum circuits, we
+employ extended precision to ensure the accuracy of simulation results and
+mixed precision to fully exploit the potential of GPUs, resulting in faster and
+more precise simulations. Our numerical experiments demonstrate that our
+approach can achieve a 3.96x reduction in verification time for random quantum
+circuit samples in the 18-cycle case of Sycamore, with sustained performance
+exceeding 21 TFLOPS on one A100. This method can be easily extended to the
+20-cycle case, maintaining the same performance, accelerating by 12.5x compared
+to the state-of-the-art CPU-based results and 4.48-6.78x compared to the
+state-of-the-art GPU-based results reported in the literature.",quant-ph,"['quant-ph', 'cs.DC', 'physics.comp-ph']"
+Image Sculpting: Precise Object Editing with 3D Geometry Control,Jiraphon Yenphraphai · Xichen Pan · Sainan Liu · Daniele Panozzo · Saining Xie,https://image-sculpting.github.io/,https://arxiv.org/abs/2401.01702,,2401.01702.pdf,Image Sculpting: Precise Object Editing with 3D Geometry Control,"We present Image Sculpting, a new framework for editing 2D images by
+incorporating tools from 3D geometry and graphics. This approach differs
+markedly from existing methods, which are confined to 2D spaces and typically
+rely on textual instructions, leading to ambiguity and limited control. Image
+Sculpting converts 2D objects into 3D, enabling direct interaction with their
+3D geometry. Post-editing, these objects are re-rendered into 2D, merging into
+the original image to produce high-fidelity results through a coarse-to-fine
+enhancement process. The framework supports precise, quantifiable, and
+physically-plausible editing options such as pose editing, rotation,
+translation, 3D composition, carving, and serial addition. It marks an initial
+step towards combining the creative freedom of generative models with the
+precision of graphics pipelines.",cs.GR,"['cs.GR', 'cs.CV']"
+Test-Time Domain Generalization for Face Anti-Spoofing,Qianyu Zhou · Ke-Yue Zhang · Taiping Yao · Xuequan Lu · Shouhong Ding · Lizhuang Ma, ,https://arxiv.org/abs/2403.19334,,2403.19334.pdf,Test-Time Domain Generalization for Face Anti-Spoofing,"Face Anti-Spoofing (FAS) is pivotal in safeguarding facial recognition
+systems against presentation attacks. While domain generalization (DG) methods
+have been developed to enhance FAS performance, they predominantly focus on
+learning domain-invariant features during training, which may not guarantee
+generalizability to unseen data that differs largely from the source
+distributions. Our insight is that testing data can serve as a valuable
+resource to enhance the generalizability beyond mere evaluation for DG FAS. In
+this paper, we introduce a novel Test-Time Domain Generalization (TTDG)
+framework for FAS, which leverages the testing data to boost the model's
+generalizability. Our method, consisting of Test-Time Style Projection (TTSP)
+and Diverse Style Shifts Simulation (DSSS), effectively projects the unseen
+data to the seen domain space. In particular, we first introduce the innovative
+TTSP to project the styles of the arbitrarily unseen samples of the testing
+distribution to the known source space of the training distributions. We then
+design the efficient DSSS to synthesize diverse style shifts via learnable
+style bases with two specifically designed losses in a hyperspherical feature
+space. Our method eliminates the need for model updates at the test time and
+can be seamlessly integrated into not only the CNN but also ViT backbones.
+Comprehensive experiments on widely used cross-domain FAS benchmarks
+demonstrate our method's state-of-the-art performance and effectiveness.",cs.CV,['cs.CV']
+Towards Learning a Generalist Model for Embodied Navigation,Duo Zheng · Shijia Huang · Lin Zhao · Yiwu Zhong · Liwei Wang, ,https://arxiv.org/abs/2312.02010,,2312.02010.pdf,Towards Learning a Generalist Model for Embodied Navigation,"Building a generalist agent that can interact with the world is the
+intriguing target of AI systems, thus spurring the research for embodied
+navigation, where an agent is required to navigate according to instructions or
+respond to queries. Despite the major progress attained, previous works
+primarily focus on task-specific agents and lack generalizability to unseen
+scenarios. Recently, LLMs have presented remarkable capabilities across various
+fields, and provided a promising opportunity for embodied navigation. Drawing
+on this, we propose the first generalist model for embodied navigation,
+NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based
+instruction. The schema-based instruction flexibly casts various tasks into
+generation problems, thereby unifying a wide range of tasks. This approach
+allows us to integrate diverse data sources from various datasets into the
+training, equipping NaviLLM with a wide range of capabilities required by
+embodied navigation. We conduct extensive experiments to evaluate the
+performance and generalizability of our model. The experimental results
+demonstrate that our unified model achieves state-of-the-art performance on
+CVDN, SOON, and ScanQA. Specifically, it surpasses the previous
+stats-of-the-art method by a significant margin of 29% in goal progress on
+CVDN. Moreover, our model also demonstrates strong generalizability and
+presents impressive results on unseen tasks, e.g., embodied question answering
+and 3D captioning.",cs.CV,"['cs.CV', 'cs.AI']"
+Integrating Efficient Optimal Transport and Functional Maps For Unsupervised Shape Correspondence Learning,Tung Le · Khai Nguyen · Shanlin Sun · Nhat Ho · Xiaohui Xie, ,https://arxiv.org/abs/2403.01781v1,,2403.01781v1.pdf,Integrating Efficient Optimal Transport and Functional Maps For Unsupervised Shape Correspondence Learning,"In the realm of computer vision and graphics, accurately establishing
+correspondences between geometric 3D shapes is pivotal for applications like
+object tracking, registration, texture transfer, and statistical shape
+analysis. Moving beyond traditional hand-crafted and data-driven feature
+learning methods, we incorporate spectral methods with deep learning, focusing
+on functional maps (FMs) and optimal transport (OT). Traditional OT-based
+approaches, often reliant on entropy regularization OT in learning-based
+framework, face computational challenges due to their quadratic cost. Our key
+contribution is to employ the sliced Wasserstein distance (SWD) for OT, which
+is a valid fast optimal transport metric in an unsupervised shape matching
+framework. This unsupervised framework integrates functional map regularizers
+with a novel OT-based loss derived from SWD, enhancing feature alignment
+between shapes treated as discrete probability measures. We also introduce an
+adaptive refinement process utilizing entropy regularized OT, further refining
+feature alignments for accurate point-to-point correspondences. Our method
+demonstrates superior performance in non-rigid shape matching, including
+near-isometric and non-isometric scenarios, and excels in downstream tasks like
+segmentation transfer. The empirical results on diverse datasets highlight our
+framework's effectiveness and generalization capabilities, setting new
+standards in non-rigid shape matching with efficient OT metrics and an adaptive
+refinement module.",cs.CV,"['cs.CV', 'cs.AI']"
+NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images,Yufei Han · Heng Guo · Koki Fukai · Hiroaki Santo · Boxin Shi · Fumio Okura · Zhanyu Ma · Yunpeng Jia, ,,,,,,,nan
+Towards Transferable Targeted 3D Adversarial Attack in the Physical World,Yao Huang · Yinpeng Dong · Shouwei Ruan · Xiao Yang · Hang Su · Xingxing Wei, ,https://arxiv.org/abs/2312.09558,,2312.09558.pdf,Towards Transferable Targeted 3D Adversarial Attack in the Physical World,"Compared with transferable untargeted attacks, transferable targeted
+adversarial attacks could specify the misclassification categories of
+adversarial samples, posing a greater threat to security-critical tasks. In the
+meanwhile, 3D adversarial samples, due to their potential of multi-view
+robustness, can more comprehensively identify weaknesses in existing deep
+learning systems, possessing great application value. However, the field of
+transferable targeted 3D adversarial attacks remains vacant. The goal of this
+work is to develop a more effective technique that could generate transferable
+targeted 3D adversarial examples, filling the gap in this field. To achieve
+this goal, we design a novel framework named TT3D that could rapidly
+reconstruct from few multi-view images into Transferable Targeted 3D textured
+meshes. While existing mesh-based texture optimization methods compute
+gradients in the high-dimensional mesh space and easily fall into local optima,
+leading to unsatisfactory transferability and distinct distortions, TT3D
+innovatively performs dual optimization towards both feature grid and
+Multi-layer Perceptron (MLP) parameters in the grid-based NeRF space, which
+significantly enhances black-box transferability while enjoying naturalness.
+Experimental results show that TT3D not only exhibits superior cross-model
+transferability but also maintains considerable adaptability across different
+renders and vision tasks. More importantly, we produce 3D adversarial examples
+with 3D printing techniques in the real world and verify their robust
+performance under various scenarios.",cs.CV,['cs.CV']
+Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences,Axel Barroso-Laguna · Sowmya Munukutla · Victor Adrian Prisacariu · Eric Brachmann,https://nianticlabs.github.io/mickey/,https://arxiv.org/abs/2404.06337,,2404.06337.pdf,Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences,"Given two images, we can estimate the relative camera pose between them by
+establishing image-to-image correspondences. Usually, correspondences are
+2D-to-2D and the pose we estimate is defined only up to scale. Some
+applications, aiming at instant augmented reality anywhere, require
+scale-metric pose estimates, and hence, they rely on external depth estimators
+to recover the scale. We present MicKey, a keypoint matching pipeline that is
+able to predict metric correspondences in 3D camera space. By learning to match
+3D coordinates across images, we are able to infer the metric relative pose
+without depth measurements. Depth measurements are also not required for
+training, nor are scene reconstructions or image overlap information. MicKey is
+supervised only by pairs of images and their relative poses. MicKey achieves
+state-of-the-art performance on the Map-Free Relocalisation benchmark while
+requiring less supervision than competing approaches.",cs.CV,['cs.CV']
+3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation,Xingguang Zhong · Yue Pan · Cyrill Stachniss · Jens Behley,https://github.com/PRBonn/4dNDF,http://export.arxiv.org/abs/2405.03388,,2405.03388.pdf,3D LiDAR Mapping in Dynamic Environments Using a 4D Implicit Neural Representation,"Building accurate maps is a key building block to enable reliable
+localization, planning, and navigation of autonomous vehicles. We propose a
+novel approach for building accurate maps of dynamic environments utilizing a
+sequence of LiDAR scans. To this end, we propose encoding the 4D scene into a
+novel spatio-temporal implicit neural map representation by fitting a
+time-dependent truncated signed distance function to each point. Using our
+representation, we extract the static map by filtering the dynamic parts. Our
+neural representation is based on sparse feature grids, a globally shared
+decoder, and time-dependent basis functions, which we jointly optimize in an
+unsupervised fashion. To learn this representation from a sequence of LiDAR
+scans, we design a simple yet efficient loss function to supervise the map
+optimization in a piecewise way. We evaluate our approach on various scenes
+containing moving objects in terms of the reconstruction quality of static maps
+and the segmentation of dynamic point clouds. The experimental results
+demonstrate that our method is capable of removing the dynamic part of the
+input point clouds while reconstructing accurate and complete 3D maps,
+outperforming several state-of-the-art methods. Codes are available at:
+https://github.com/PRBonn/4dNDF",cs.CV,"['cs.CV', 'cs.RO']"
+Robust Emotion Recognition in Context Debiasing,Dingkang Yang · Kun Yang · Mingcheng Li · Shunli Wang · Shuaibing Wang · Lihua Zhang, ,https://arxiv.org/abs/2403.05963,,2403.05963.pdf,Robust Emotion Recognition in Context Debiasing,"Context-aware emotion recognition (CAER) has recently boosted the practical
+applications of affective computing techniques in unconstrained environments.
+Mainstream CAER methods invariably extract ensemble representations from
+diverse contexts and subject-centred characteristics to perceive the target
+person's emotional state. Despite advancements, the biggest challenge remains
+due to context bias interference. The harmful bias forces the models to rely on
+spurious correlations between background contexts and emotion labels in
+likelihood estimation, causing severe performance bottlenecks and confounding
+valuable context priors. In this paper, we propose a counterfactual emotion
+inference (CLEF) framework to address the above issue. Specifically, we first
+formulate a generalized causal graph to decouple the causal relationships among
+the variables in CAER. Following the causal graph, CLEF introduces a
+non-invasive context branch to capture the adverse direct effect caused by the
+context bias. During the inference, we eliminate the direct context effect from
+the total causal effect by comparing factual and counterfactual outcomes,
+resulting in bias mitigation and robust prediction. As a model-agnostic
+framework, CLEF can be readily integrated into existing methods, bringing
+consistent performance gains.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning to Produce Semi-dense Correspondences for Visual Localization,Khang Truong Giang · Soohwan Song · Sungho Jo,https://github.com/TruongKhang/DeViLoc,https://arxiv.org/abs/2402.08359,,2402.08359.pdf,Learning to Produce Semi-dense Correspondences for Visual Localization,"This study addresses the challenge of performing visual localization in
+demanding conditions such as night-time scenarios, adverse weather, and
+seasonal changes. While many prior studies have focused on improving
+image-matching performance to facilitate reliable dense keypoint matching
+between images, existing methods often heavily rely on predefined feature
+points on a reconstructed 3D model. Consequently, they tend to overlook
+unobserved keypoints during the matching process. Therefore, dense keypoint
+matches are not fully exploited, leading to a notable reduction in accuracy,
+particularly in noisy scenes. To tackle this issue, we propose a novel
+localization method that extracts reliable semi-dense 2D-3D matching points
+based on dense keypoint matches. This approach involves regressing semi-dense
+2D keypoints into 3D scene coordinates using a point inference network. The
+network utilizes both geometric and visual cues to effectively infer 3D
+coordinates for unobserved keypoints from the observed ones. The abundance of
+matching information significantly enhances the accuracy of camera pose
+estimation, even in scenarios involving noisy or sparse 3D models.
+Comprehensive evaluations demonstrate that the proposed method outperforms
+other methods in challenging scenes and achieves competitive results in
+large-scale visual localization benchmarks. The code will be available.",cs.CV,['cs.CV']
+Distilling CLIP with Dual Guidance for  Learning Discriminative Human Body Shape Representation,Feng Liu · Minchul Kim · Zhiyuan Ren · Xiaoming Liu, ,https://arxiv.org/abs/2307.12732,,,CLIP-KD: An Empirical Study of CLIP Model Distillation,"Contrastive Language-Image Pre-training (CLIP) has become a promising
+language-supervised visual pre-training framework. This paper aims to distill
+small CLIP models supervised by a large teacher CLIP model. We propose several
+distillation strategies, including relation, feature, gradient and contrastive
+paradigms, to examine the effectiveness of CLIP-Knowledge Distillation (KD). We
+show that a simple feature mimicry with Mean Squared Error loss works
+surprisingly well. Moreover, interactive contrastive learning across teacher
+and student encoders is also effective in performance improvement. We explain
+that the success of CLIP-KD can be attributed to maximizing the feature
+similarity between teacher and student. The unified method is applied to
+distill several student models trained on CC3M+12M. CLIP-KD improves student
+CLIP models consistently over zero-shot ImageNet classification and cross-modal
+retrieval benchmarks. When using ViT-L/14 pretrained on Laion-400M as the
+teacher, CLIP-KD achieves 57.5\% and 55.4\% zero-shot top-1 ImageNet accuracy
+over ViT-B/16 and ResNet-50, surpassing the original CLIP without KD by 20.5\%
+and 20.1\% margins, respectively. Our code is released on
+https://github.com/winycg/CLIP-KD.",cs.CV,['cs.CV']
+From Feature to Gaze: A Generalizable Replacement of Linear Layer for Gaze Estimation,Yiwei Bao · Feng Lu, ,https://arxiv.org/abs/2309.02165,,2309.02165.pdf,PCFGaze: Physics-Consistent Feature for Appearance-based Gaze Estimation,"Although recent deep learning based gaze estimation approaches have achieved
+much improvement, we still know little about how gaze features are connected to
+the physics of gaze. In this paper, we try to answer this question by analyzing
+the gaze feature manifold. Our analysis revealed the insight that the geodesic
+distance between gaze features is consistent with the gaze differences between
+samples. According to this finding, we construct the Physics- Consistent
+Feature (PCF) in an analytical way, which connects gaze feature to the physical
+definition of gaze. We further propose the PCFGaze framework that directly
+optimizes gaze feature space by the guidance of PCF. Experimental results
+demonstrate that the proposed framework alleviates the overfitting problem and
+significantly improves cross-domain gaze estimation accuracy without extra
+training data. The insight of gaze feature has the potential to benefit other
+regression tasks with physical meanings.",cs.CV,['cs.CV']
+Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception,Haoming Chen · Zhizhong Zhang · Yanyun Qu · Ruixin Zhang · Xin Tan · Yuan Xie, ,https://arxiv.org/abs/2405.07201,,2405.07201.pdf,Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception,"An effective pre-training framework with universal 3D representations is
+extremely desired in perceiving large-scale dynamic scenes. However,
+establishing such an ideal framework that is both task-generic and
+label-efficient poses a challenge in unifying the representation of the same
+primitive across diverse scenes. The current contrastive 3D pre-training
+methods typically follow a frame-level consistency, which focuses on the 2D-3D
+relationships in each detached image. Such inconsiderate consistency greatly
+hampers the promising path of reaching an universal pre-training framework: (1)
+The cross-scene semantic self-conflict, i.e., the intense collision between
+primitive segments of the same semantics from different scenes; (2) Lacking a
+globally unified bond that pushes the cross-scene semantic consistency into 3D
+representation learning. To address above challenges, we propose a CSC
+framework that puts a scene-level semantic consistency in the heart, bridging
+the connection of the similar semantic segments across various scenes. To
+achieve this goal, we combine the coherent semantic cues provided by the vision
+foundation model and the knowledge-rich cross-scene prototypes derived from the
+complementary multi-modality information. These allow us to train a universal
+3D pre-training model that facilitates various downstream tasks with less
+fine-tuning efforts. Empirically, we achieve consistent improvements over SOTA
+pre-training approaches in semantic segmentation (+1.4% mIoU), object detection
+(+1.0% mAP), and panoptic segmentation (+3.0% PQ) using their task-specific 3D
+network on nuScenes. Code is released at https://github.com/chenhaomingbob/CSC,
+hoping to inspire future research.",cs.CV,['cs.CV']
+Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation,Dongliang Cao · Marvin Eisenberger · Nafie El Amrani · Daniel Cremers · Florian Bernard, ,https://web3.arxiv.org/abs/2402.18920,,2402.18920.pdf,Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation,"Although 3D shape matching and interpolation are highly interrelated, they
+are often studied separately and applied sequentially to relate different 3D
+shapes, thus resulting in sub-optimal performance. In this work we present a
+unified framework to predict both point-wise correspondences and shape
+interpolation between 3D shapes. To this end, we combine the deep functional
+map framework with classical surface deformation models to map shapes in both
+spectral and spatial domains. On the one hand, by incorporating spatial maps,
+our method obtains more accurate and smooth point-wise correspondences compared
+to previous functional map methods for shape matching. On the other hand, by
+introducing spectral maps, our method gets rid of commonly used but
+computationally expensive geodesic distance constraints that are only valid for
+near-isometric shape deformations. Furthermore, we propose a novel test-time
+adaptation scheme to capture both pose-dominant and shape-dominant
+deformations. Using different challenging datasets, we demonstrate that our
+method outperforms previous state-of-the-art methods for both shape matching
+and interpolation, even compared to supervised approaches.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CG']"
+SinSR: Diffusion-Based Image Super-Resolution in a Single Step,Yufei Wang · Wenhan Yang · Xinyuan Chen · Yaohui Wang · Lanqing Guo · Lap-Pui Chau · Ziwei Liu · Yu Qiao · Alex C. Kot · Bihan Wen, ,https://arxiv.org/abs/2311.14760,,2311.14760.pdf,SinSR: Diffusion-Based Image Super-Resolution in a Single Step,"While super-resolution (SR) methods based on diffusion models exhibit
+promising results, their practical application is hindered by the substantial
+number of required inference steps. Recent methods utilize degraded images in
+the initial state, thereby shortening the Markov chain. Nevertheless, these
+solutions either rely on a precise formulation of the degradation process or
+still necessitate a relatively lengthy generation path (e.g., 15 iterations).
+To enhance inference speed, we propose a simple yet effective method for
+achieving single-step SR generation, named SinSR. Specifically, we first derive
+a deterministic sampling process from the most recent state-of-the-art (SOTA)
+method for accelerating diffusion-based SR. This allows the mapping between the
+input random noise and the generated high-resolution image to be obtained in a
+reduced and acceptable number of inference steps during training. We show that
+this deterministic mapping can be distilled into a student model that performs
+SR within only one inference step. Additionally, we propose a novel
+consistency-preserving loss to simultaneously leverage the ground-truth image
+during the distillation process, ensuring that the performance of the student
+model is not solely bound by the feature manifold of the teacher model,
+resulting in further performance improvement. Extensive experiments conducted
+on synthetic and real-world datasets demonstrate that the proposed method can
+achieve comparable or even superior performance compared to both previous SOTA
+methods and the teacher model, in just one sampling step, resulting in a
+remarkable up to x10 speedup for inference. Our code will be released at
+https://github.com/wyf0912/SinSR",cs.CV,['cs.CV']
+SRTube: Video-Language Pre-Training with Action-Centric Video Tube Features and Semantic Role Labeling,Juhee Lee · Jewon Kang, ,https://arxiv.org/abs/2402.03161,,2402.03161.pdf,Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization,"In light of recent advances in multimodal Large Language Models (LLMs), there
+is increasing attention to scaling them from image-text data to more
+informative real-world videos. Compared to static images, video poses unique
+challenges for effective large-scale pre-training due to the modeling of its
+spatiotemporal dynamics. In this paper, we address such limitations in
+video-language pre-training with an efficient video decomposition that
+represents each video as keyframes and temporal motions. These are then adapted
+to an LLM using well-designed tokenizers that discretize visual and temporal
+information as a few tokens, thus enabling unified generative pre-training of
+videos, images, and text. At inference, the generated tokens from the LLM are
+carefully recovered to the original continuous pixel space to create various
+video content. Our proposed framework is both capable of comprehending and
+generating image and video content, as demonstrated by its competitive
+performance across 13 multimodal benchmarks in image and video understanding
+and generation. Our code and models are available at
+https://video-lavit.github.io.",cs.CV,"['cs.CV', 'cs.CL']"
+Quantifying Task Priority for Multi-Task Optimization,Wooseong Jeong · Kuk-Jin Yoon, ,https://arxiv.org/abs/2403.16162,,2403.16162.pdf,Multi-Task Learning with Multi-Task Optimization,"Multi-task learning solves multiple correlated tasks. However, conflicts may
+exist between them. In such circumstances, a single solution can rarely
+optimize all the tasks, leading to performance trade-offs. To arrive at a set
+of optimized yet well-distributed models that collectively embody different
+trade-offs in one algorithmic pass, this paper proposes to view Pareto
+multi-task learning through the lens of multi-task optimization. Multi-task
+learning is first cast as a multi-objective optimization problem, which is then
+decomposed into a diverse set of unconstrained scalar-valued subproblems. These
+subproblems are solved jointly using a novel multi-task gradient descent
+method, whose uniqueness lies in the iterative transfer of model parameters
+among the subproblems during the course of optimization. A theorem proving
+faster convergence through the inclusion of such transfers is presented. We
+investigate the proposed multi-task learning with multi-task optimization for
+solving various problem settings including image classification, scene
+understanding, and multi-target regression. Comprehensive experiments confirm
+that the proposed method significantly advances the state-of-the-art in
+discovering sets of Pareto-optimized models. Notably, on the large image
+dataset we tested on, namely NYUv2, the hypervolume convergence achieved by our
+method was found to be nearly two times faster than the next-best among the
+state-of-the-art.",cs.AI,['cs.AI']
+Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding,Le Zhang · Rabiul Awal · Aishwarya Agrawal,https://github.com/lezhang7/Enhance-FineGrained,https://arxiv.org/abs/2306.08832,,2306.08832.pdf,Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding,"Vision-Language Models (VLMs), such as CLIP, exhibit strong image-text
+comprehension abilities, facilitating advances in several downstream tasks such
+as zero-shot image classification, image-text retrieval, and text-to-image
+generation. However, the compositional reasoning abilities of existing VLMs
+remains subpar. The root of this limitation lies in the inadequate alignment
+between the images and captions in the pretraining datasets. Additionally, the
+current contrastive learning objective fails to focus on fine-grained grounding
+components like relations, actions, and attributes, resulting in ""bag-of-words""
+representations. We introduce a simple and effective method to improve
+compositional reasoning in VLMs. Our method better leverages available datasets
+by refining and expanding the standard image-text contrastive learning
+framework. Our approach does not require specific annotations and does not
+incur extra parameters. When integrated with CLIP, our technique yields notable
+improvement over state-of-the-art baselines across five vision-language
+compositional benchmarks. We open-source our code at
+https://github.com/lezhang7/Enhance-FineGrained.",cs.CV,['cs.CV']
+CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition,Feng Lu · Xiangyuan Lan · Lijun Zhang · Dongmei Jiang · Yaowei Wang · Chun Yuan, ,https://arxiv.org/abs/2402.19231,,2402.19231.pdf,CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition,"Over the past decade, most methods in visual place recognition (VPR) have
+used neural networks to produce feature representations. These networks
+typically produce a global representation of a place image using only this
+image itself and neglect the cross-image variations (e.g. viewpoint and
+illumination), which limits their robustness in challenging scenes. In this
+paper, we propose a robust global representation method with cross-image
+correlation awareness for VPR, named CricaVPR. Our method uses the attention
+mechanism to correlate multiple images within a batch. These images can be
+taken in the same place with different conditions or viewpoints, or even
+captured from different places. Therefore, our method can utilize the
+cross-image variations as a cue to guide the representation learning, which
+ensures more robust features are produced. To further facilitate the
+robustness, we propose a multi-scale convolution-enhanced adaptation method to
+adapt pre-trained visual foundation models to the VPR task, which introduces
+the multi-scale local information to further enhance the cross-image
+correlation-aware representation. Experimental results show that our method
+outperforms state-of-the-art methods by a large margin with significantly less
+training time. The code is released at https://github.com/Lu-Feng/CricaVPR.",cs.CV,"['cs.CV', 'cs.RO']"
+Dual Prior Unfolding for Snapshot Compressive Imaging,Jiancheng Zhang · Haijin Zeng · Jiezhang Cao · Yongyong Chen · Dengxiu Yu · Yinping Zhao,https://github.com/ZhangJC-2k/DPU,,https://link.springer.com/article/10.1007/s11263-023-01844-4,,,,,nan
+Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships,Rangel Daroya · Aaron Sun · Subhransu Maji,https://github.com/cvl-umass/task2box,https://arxiv.org/abs/2403.17173,,2403.17173.pdf,Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships,"Modeling and visualizing relationships between tasks or datasets is an
+important step towards solving various meta-tasks such as dataset discovery,
+multi-tasking, and transfer learning. However, many relationships, such as
+containment and transferability, are naturally asymmetric and current
+approaches for representation and visualization (e.g., t-SNE) do not readily
+support this. We propose Task2Box, an approach to represent tasks using box
+embeddings -- axis-aligned hyperrectangles in low dimensional spaces -- that
+can capture asymmetric relationships between them through volumetric overlaps.
+We show that Task2Box accurately predicts unseen hierarchical relationships
+between nodes in ImageNet and iNaturalist datasets, as well as transferability
+between tasks in the Taskonomy benchmark. We also show that box embeddings
+estimated from task representations (e.g., CLIP, Task2Vec, or attribute based)
+can be used to predict relationships between unseen tasks more accurately than
+classifiers trained on the same representations, as well as handcrafted
+asymmetric distances (e.g., KL divergence). This suggests that low-dimensional
+box embeddings can effectively capture these task relationships and have the
+added advantage of being interpretable. We use the approach to visualize
+relationships among publicly available image classification datasets on popular
+dataset hosting platform called Hugging Face.",cs.CV,['cs.CV']
+Shadow Generation for Composite Image Using Diffusion Model,Qingyang Liu · Junqi You · Jian-Ting Wang · Xinhao Tao · Bo Zhang · Li Niu, ,https://arxiv.org/abs/2403.15234,,2403.15234.pdf,Shadow Generation for Composite Image Using Diffusion model,"In the realm of image composition, generating realistic shadow for the
+inserted foreground remains a formidable challenge. Previous works have
+developed image-to-image translation models which are trained on paired
+training data. However, they are struggling to generate shadows with accurate
+shapes and intensities, hindered by data scarcity and inherent task complexity.
+In this paper, we resort to foundation model with rich prior knowledge of
+natural shadow images. Specifically, we first adapt ControlNet to our task and
+then propose intensity modulation modules to improve the shadow intensity.
+Moreover, we extend the small-scale DESOBA dataset to DESOBAv2 using a novel
+data acquisition pipeline. Experimental results on both DESOBA and DESOBAv2
+datasets as well as real composite images demonstrate the superior capability
+of our model for shadow generation task. The dataset, code, and model are
+released at https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2.",cs.CV,['cs.CV']
+NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild,Weining Ren · Zihan Zhu · Boyang Sun · Jiaqi Chen · Marc Pollefeys · Songyou Peng,https://rwn17.github.io/nerf-on-the-go,https://arxiv.org/abs/2405.18715,,2405.18715.pdf,NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild,"Neural Radiance Fields (NeRFs) have shown remarkable success in synthesizing
+photorealistic views from multi-view images of static scenes, but face
+challenges in dynamic, real-world environments with distractors like moving
+objects, shadows, and lighting changes. Existing methods manage controlled
+environments and low occlusion ratios but fall short in render quality,
+especially under high occlusion scenarios. In this paper, we introduce NeRF
+On-the-go, a simple yet effective approach that enables the robust synthesis of
+novel views in complex, in-the-wild scenes from only casually captured image
+sequences. Delving into uncertainty, our method not only efficiently eliminates
+distractors, even when they are predominant in captures, but also achieves a
+notably faster convergence speed. Through comprehensive experiments on various
+scenes, our method demonstrates a significant improvement over state-of-the-art
+techniques. This advancement opens new avenues for NeRF in diverse and dynamic
+real-world applications.",cs.CV,['cs.CV']
+Improved Baselines with Visual Instruction Tuning,Haotian Liu · Chunyuan Li · Yuheng Li · Yong Jae Lee,https://llava-vl.github.io,https://arxiv.org/abs/2310.03744,,2310.03744.pdf,Improved Baselines with Visual Instruction Tuning,"Large multimodal models (LMM) have recently shown encouraging progress with
+visual instruction tuning. In this note, we show that the fully-connected
+vision-language cross-modal connector in LLaVA is surprisingly powerful and
+data-efficient. With simple modifications to LLaVA, namely, using
+CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA
+data with simple response formatting prompts, we establish stronger baselines
+that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint
+uses merely 1.2M publicly available data, and finishes full training in ~1 day
+on a single 8-A100 node. We hope this can make state-of-the-art LMM research
+more accessible. Code and model will be publicly available.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+ParamISP: Learned Forward and Inverse ISPs using Camera Parameters,Woohyeok Kim · Geonu Kim · Junyong Lee · Seungyong Lee · Seung-Hwan Baek · Sunghyun Cho,https://woo525.github.io/ParamISP/,https://arxiv.org/abs/2312.13313,,2312.13313.pdf,ParamISP: Learned Forward and Inverse ISPs using Camera Parameters,"RAW images are rarely shared mainly due to its excessive data size compared
+to their sRGB counterparts obtained by camera ISPs. Learning the forward and
+inverse processes of camera ISPs has been recently demonstrated, enabling
+physically-meaningful RAW-level image processing on input sRGB images. However,
+existing learning-based ISP methods fail to handle the large variations in the
+ISP processes with respect to camera parameters such as ISO and exposure time,
+and have limitations when used for various applications. In this paper, we
+propose ParamISP, a learning-based method for forward and inverse conversion
+between sRGB and RAW images, that adopts a novel neural-network module to
+utilize camera parameters, which is dubbed as ParamNet. Given the camera
+parameters provided in the EXIF data, ParamNet converts them into a feature
+vector to control the ISP networks. Extensive experiments demonstrate that
+ParamISP achieve superior RAW and sRGB reconstruction results compared to
+previous methods and it can be effectively used for a variety of applications
+such as deblurring dataset synthesis, raw deblurring, HDR reconstruction, and
+camera-to-camera transfer.",eess.IV,"['eess.IV', 'cs.CV']"
+ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts,Mu Cai · Haotian Liu · Siva Mustikovela · Gregory P. Meyer · Yuning Chai · Dennis Park · Yong Jae Lee,https://vip-llava.github.io/,https://arxiv.org/abs/2312.00784,,2312.00784.pdf,ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts,"While existing large vision-language multimodal models focus on whole image
+understanding, there is a prominent gap in achieving region-specific
+comprehension. Current approaches that use textual coordinates or spatial
+encodings often fail to provide a user-friendly interface for visual prompting.
+To address this challenge, we introduce a novel multimodal model capable of
+decoding arbitrary visual prompts. This allows users to intuitively mark images
+and interact with the model using natural cues like a ""red bounding box"" or
+""pointed arrow"". Our simple design directly overlays visual markers onto the
+RGB image, eliminating the need for complex region encodings, yet achieves
+state-of-the-art performance on region-understanding tasks like Visual7W,
+PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present
+ViP-Bench, a comprehensive benchmark to assess the capability of models in
+understanding visual prompts across multiple dimensions, enabling future
+research in this domain. Code, data, and model are publicly available.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+Compact 3D Gaussian Representation for Radiance Field,Joo Chan Lee · Daniel Rho · Xiangyu Sun · Jong Hwan Ko · Eunbyung Park,https://maincold2.github.io/c3dgs/,https://arxiv.org/abs/2311.13681,,2311.13681.pdf,Compact 3D Gaussian Representation for Radiance Field,"Neural Radiance Fields (NeRFs) have demonstrated remarkable potential in
+capturing complex 3D scenes with high fidelity. However, one persistent
+challenge that hinders the widespread adoption of NeRFs is the computational
+bottleneck due to the volumetric rendering. On the other hand, 3D Gaussian
+splatting (3DGS) has recently emerged as an alternative representation that
+leverages a 3D Gaussisan-based representation and adopts the rasterization
+pipeline to render the images rather than volumetric rendering, achieving very
+fast rendering speed and promising image quality. However, a significant
+drawback arises as 3DGS entails a substantial number of 3D Gaussians to
+maintain the high fidelity of the rendered images, which requires a large
+amount of memory and storage. To address this critical issue, we place a
+specific emphasis on two key objectives: reducing the number of Gaussian points
+without sacrificing performance and compressing the Gaussian attributes, such
+as view-dependent color and covariance. To this end, we propose a learnable
+mask strategy that significantly reduces the number of Gaussians while
+preserving high performance. In addition, we propose a compact but effective
+representation of view-dependent color by employing a grid-based neural field
+rather than relying on spherical harmonics. Finally, we learn codebooks to
+compactly represent the geometric attributes of Gaussian by vector
+quantization. With model compression techniques such as quantization and
+entropy coding, we consistently show over 25$\times$ reduced storage and
+enhanced rendering speed, while maintaining the quality of the scene
+representation, compared to 3DGS. Our work provides a comprehensive framework
+for 3D scene representation, achieving high performance, fast training,
+compactness, and real-time rendering. Our project page is available at
+https://maincold2.github.io/c3dgs/.",cs.CV,"['cs.CV', 'cs.GR']"
+Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology,Oren Kraus · Kian Kenyon-Dean · Saber Saberian · Maryam Fallah · Peter McLean · Jess Leung · Vasudev Sharma · Ayla Khan · Jia Balakrishnan · Safiye Celik · Dominique Beaini · Maciej Sypetkowski · Chi Cheng · Kristen Morse · Maureen Makes · Ben Mabey · Berton Earnshaw, ,https://arxiv.org/abs/2404.10242,,2404.10242.pdf,Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology,"Featurizing microscopy images for use in biological research remains a
+significant challenge, especially for large-scale experiments spanning millions
+of images. This work explores the scaling properties of weakly supervised
+classifiers and self-supervised masked autoencoders (MAEs) when training with
+increasingly larger model backbones and microscopy datasets. Our results show
+that ViT-based MAEs outperform weakly supervised classifiers on a variety of
+tasks, achieving as much as a 11.5% relative improvement when recalling known
+biological relationships curated from public databases. Additionally, we
+develop a new channel-agnostic MAE architecture (CA-MAE) that allows for
+inputting images of different numbers and orders of channels at inference time.
+We demonstrate that CA-MAEs effectively generalize by inferring and evaluating
+on a microscopy image dataset (JUMP-CP) generated under different experimental
+conditions with a different channel structure than our pretraining data
+(RPI-93M). Our findings motivate continued research into scaling
+self-supervised learning on microscopy data in order to create powerful
+foundation models of cellular biology that have the potential to catalyze
+advancements in drug discovery and beyond.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis,Yanzuo Lu · Manlin Zhang · Jinhua Ma · Xiaohua Xie · Jianhuang Lai,https://github.com/YanzuoLu/CFLD,https://arxiv.org/abs/2402.18078,,2402.18078.pdf,Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis,"Diffusion model is a promising approach to image generation and has been
+employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive
+performance. While existing methods simply align the person appearance to the
+target pose, they are prone to overfitting due to the lack of a high-level
+semantic understanding on the source person image. In this paper, we propose a
+novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence
+of image-caption pairs and textual prompts, we develop a novel training
+paradigm purely based on images to control the generation process of a
+pre-trained text-to-image diffusion model. A perception-refined decoder is
+designed to progressively refine a set of learnable queries and extract
+semantic understanding of person images as a coarse-grained prompt. This allows
+for the decoupling of fine-grained appearance and pose information controls at
+different stages, and thus circumventing the potential overfitting problem. To
+generate more realistic texture details, a hybrid-granularity attention module
+is proposed to encode multi-scale fine-grained appearance features as bias
+terms to augment the coarse-grained prompt. Both quantitative and qualitative
+experimental results on the DeepFashion benchmark demonstrate the superiority
+of our method over the state of the arts for PGPIS. Code is available at
+https://github.com/YanzuoLu/CFLD.",cs.CV,['cs.CV']
+SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training,WU Sitong · Haoru Tan · Zhuotao Tian · Yukang Chen · Xiaojuan Qi · Jiaya Jia, ,https://arxiv.org/abs/2405.10286,,,FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models,"Despite noise and caption quality having been acknowledged as important
+factors impacting vision-language contrastive pre-training, in this paper, we
+show that the full potential of improving the training process by addressing
+such issues is yet to be realized. Specifically, we firstly study and analyze
+two issues affecting training: incorrect assignment of negative pairs, and low
+caption quality and diversity. Then, we devise effective solutions for
+addressing both problems, which essentially require training with multiple true
+positive pairs. Finally, we propose training with sigmoid loss to address such
+a requirement. We show very large gains over the current state-of-the-art for
+both image recognition ($\sim +6\%$ on average over 11 datasets) and image
+retrieval ($\sim +19\%$ on Flickr30k and $\sim +15\%$ on MSCOCO).",cs.CV,"['cs.CV', 'cs.AI']"
+Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps,Octave Mariotti · Oisin Mac Aodha · Hakan Bilen, ,https://arxiv.org/abs/2312.13216,,2312.13216.pdf,Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps,"Recent progress in self-supervised representation learning has resulted in
+models that are capable of extracting image features that are not only
+effective at encoding image level, but also pixel-level, semantics. These
+features have been shown to be effective for dense visual semantic
+correspondence estimation, even outperforming fully-supervised methods.
+Nevertheless, current self-supervised approaches still fail in the presence of
+challenging image characteristics such as symmetries and repeated parts. To
+address these limitations, we propose a new approach for semantic
+correspondence estimation that supplements discriminative self-supervised
+features with 3D understanding via a weak geometric spherical prior. Compared
+to more involved 3D pipelines, our model only requires weak viewpoint
+information, and the simplicity of our spherical representation enables us to
+inject informative geometric priors into the model during training. We propose
+a new evaluation metric that better accounts for repeated part and
+symmetry-induced mistakes. We present results on the challenging SPair-71k
+dataset, where we show that our approach demonstrates is capable of
+distinguishing between symmetric views and repeated parts across many object
+categories, and also demonstrate that we can generalize to unseen classes on
+the AwA dataset.",cs.CV,['cs.CV']
+XFeat: Accelerated Features for Lightweight Image Matching,Guilherme Potje · Felipe Cadar · André Araujo · Renato Martins · Erickson R. Nascimento,https://verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24,https://arxiv.org/abs/2404.19174,,2404.19174.pdf,XFeat: Accelerated Features for Lightweight Image Matching,"We introduce a lightweight and accurate architecture for resource-efficient
+visual correspondence. Our method, dubbed XFeat (Accelerated Features),
+revisits fundamental design choices in convolutional neural networks for
+detecting, extracting, and matching local features. Our new model satisfies a
+critical need for fast and robust algorithms suitable to resource-limited
+devices. In particular, accurate image matching requires sufficiently large
+image resolutions - for this reason, we keep the resolution as large as
+possible while limiting the number of channels in the network. Besides, our
+model is designed to offer the choice of matching at the sparse or semi-dense
+levels, each of which may be more suitable for different downstream
+applications, such as visual navigation and augmented reality. Our model is the
+first to offer semi-dense matching efficiently, leveraging a novel match
+refinement module that relies on coarse local descriptors. XFeat is versatile
+and hardware-independent, surpassing current deep learning-based local features
+in speed (up to 5x faster) with comparable or better accuracy, proven in pose
+estimation and visual localization. We showcase it running in real-time on an
+inexpensive laptop CPU without specialized hardware optimizations. Code and
+weights are available at www.verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24.",cs.CV,['cs.CV']
+Towards Realistic Scene Generation with LiDAR Diffusion Models,Haoxi Ran · Vitor Guizilini · Yue Wang,https://lidar-diffusion.github.io/,https://arxiv.org/abs/2404.00815,,2404.00815.pdf,Towards Realistic Scene Generation with LiDAR Diffusion Models,"Diffusion models (DMs) excel in photo-realistic image synthesis, but their
+adaptation to LiDAR scene generation poses a substantial hurdle. This is
+primarily because DMs operating in the point space struggle to preserve the
+curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of
+their representation power. In this paper, we propose LiDAR Diffusion Models
+(LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to
+capture the realism of LiDAR scenes by incorporating geometric priors into the
+learning pipeline. Our method targets three major desiderata: pattern realism,
+geometry realism, and object realism. Specifically, we introduce curve-wise
+compression to simulate real-world LiDAR patterns, point-wise coordinate
+supervision to learn scene geometry, and patch-wise encoding for a full 3D
+object context. With these three core designs, our method achieves competitive
+performance on unconditional LiDAR generation in 64-beam scenario and state of
+the art on conditional LiDAR generation, while maintaining high efficiency
+compared to point-based DMs (up to 107$\times$ faster). Furthermore, by
+compressing LiDAR scenes into a latent space, we enable the controllability of
+DMs with various conditions such as semantic maps, camera views, and text
+prompts.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models,Muyang Li · Tianle Cai · Jiaxin Cao · Qinsheng Zhang · Han Cai · Junjie Bai · Yangqing Jia · Kai Li · Song Han, ,https://arxiv.org/abs/2402.19481,,2402.19481.pdf,DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models,"Diffusion models have achieved great success in synthesizing high-quality
+images. However, generating high-resolution images with diffusion models is
+still challenging due to the enormous computational costs, resulting in a
+prohibitive latency for interactive applications. In this paper, we propose
+DistriFusion to tackle this problem by leveraging parallelism across multiple
+GPUs. Our method splits the model input into multiple patches and assigns each
+patch to a GPU. However, naively implementing such an algorithm breaks the
+interaction between patches and loses fidelity, while incorporating such an
+interaction will incur tremendous communication overhead. To overcome this
+dilemma, we observe the high similarity between the input from adjacent
+diffusion steps and propose displaced patch parallelism, which takes advantage
+of the sequential nature of the diffusion process by reusing the pre-computed
+feature maps from the previous timestep to provide context for the current
+step. Therefore, our method supports asynchronous communication, which can be
+pipelined by computation. Extensive experiments show that our method can be
+applied to recent Stable Diffusion XL with no quality degradation and achieve
+up to a 6.1$\times$ speedup on eight NVIDIA A100s compared to one. Our code is
+publicly available at https://github.com/mit-han-lab/distrifuser.",cs.CV,['cs.CV']
+Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use,Imad Eddine Toubal · Aditya Avinash · Neil Alldrin · Jan Dlabal · Wenlei Zhou · Enming Luo · Otilia Stretcu · Hao Xiong · Chun-Ta Lu · Howard Zhou · Ranjay Krishna · Ariel Fuxman · Tom Duerig, ,https://arxiv.org/abs/2403.02626,,2403.02626.pdf,Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use,"From content moderation to wildlife conservation, the number of applications
+that require models to recognize nuanced or subjective visual concepts is
+growing. Traditionally, developing classifiers for such concepts requires
+substantial manual effort measured in hours, days, or even months to identify
+and annotate data needed for training. Even with recently proposed Agile
+Modeling techniques, which enable rapid bootstrapping of image classifiers,
+users are still required to spend 30 minutes or more of monotonous, repetitive
+data labeling just to train a single classifier. Drawing on Fiske's Cognitive
+Miser theory, we propose a new framework that alleviates manual effort by
+replacing human labeling with natural language interactions, reducing the total
+effort required to define a concept by an order of magnitude: from labeling
+2,000 images to only 100 plus some natural language interactions. Our framework
+leverages recent advances in foundation models, both large language models and
+vision-language models, to carve out the concept space through conversation and
+by automatically labeling training data points. Most importantly, our framework
+eliminates the need for crowd-sourced annotations. Moreover, our framework
+ultimately produces lightweight classification models that are deployable in
+cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
+image classification datasets, our trained models outperform traditional Agile
+Modeling as well as state-of-the-art zero-shot classification models like
+ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.",cs.CV,"['cs.CV', 'cs.LG']"
+Multi-Task Dense Prediction via Mixture of Low-Rank Experts,Yuqi Yang · Peng-Tao Jiang · Qibin Hou · Hao Zhang · Jinwei Chen · Bo Li, ,https://arxiv.org/abs/2403.17749,,2403.17749.pdf,Multi-Task Dense Prediction via Mixture of Low-Rank Experts,"Previous multi-task dense prediction methods based on the Mixture of Experts
+(MoE) have received great performance but they neglect the importance of
+explicitly modeling the global relations among all tasks. In this paper, we
+present a novel decoder-focused method for multi-task dense prediction, called
+Mixture-of-Low-Rank-Experts (MLoRE). To model the global task relationships,
+MLoRE adds a generic convolution path to the original MoE structure, where each
+task feature can go through this path for explicit parameter sharing.
+Furthermore, to control the parameters and computational cost brought by the
+increase in the number of experts, we take inspiration from LoRA and propose to
+leverage the low-rank format of a vanilla convolution in the expert network.
+Since the low-rank experts have fewer parameters and can be dynamically
+parameterized into the generic convolution, the parameters and computational
+cost do not change much with the increase of experts. Benefiting from this
+design, we increase the number of experts and its reception field to enlarge
+the representation capacity, facilitating multiple dense tasks learning in a
+unified network. Extensive experiments on the PASCAL-Context and NYUD-v2
+benchmarks show that our MLoRE achieves superior performance compared to
+previous state-of-the-art methods on all metrics. Our code is available at
+https://github.com/YuqiYang213/MLoRE.",cs.CV,['cs.CV']
+Traffic Scene Parsing through the TSP6K Dataset,Peng-Tao Jiang · Yuqi Yang · Yang Cao · Qibin Hou · Ming-Ming Cheng · Chunhua Shen, ,https://ar5iv.labs.arxiv.org/html/2303.02835,,2303.02835.pdf,Traffic Scene Parsing through the TSP6K Dataset,"Traffic scene perception in computer vision is a critically important task to
+achieve intelligent cities. To date, most existing datasets focus on autonomous
+driving scenes. We observe that the models trained on those driving datasets
+often yield unsatisfactory results on traffic monitoring scenes. However,
+little effort has been put into improving the traffic monitoring scene
+understanding, mainly due to the lack of specific datasets. To fill this gap,
+we introduce a specialized traffic monitoring dataset, termed TSP6K, containing
+images from the traffic monitoring scenario, with high-quality pixel-level and
+instance-level annotations. The TSP6K dataset captures more crowded traffic
+scenes with several times more traffic participants than the existing driving
+scenes. We perform a detailed analysis of the dataset and comprehensively
+evaluate previous popular scene parsing methods, instance segmentation methods
+and unsupervised domain adaption methods. Furthermore, considering the vast
+difference in instance sizes, we propose a detail refining decoder for scene
+parsing, which recovers the details of different semantic regions in traffic
+scenes owing to the proposed TSP6K dataset. Experiments show its effectiveness
+in parsing the traffic monitoring scenes. Code and dataset are available at
+https://github.com/PengtaoJiang/TSP6K.",cs.CV,['cs.CV']
+"FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation",Chris Rockwell · Nilesh Kulkarni · Linyi Jin · Jeong Joon Park · Justin Johnson · David Fouhey, ,https://arxiv.org/abs/2403.03221,,,"FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation","Estimating relative camera poses between images has been a central problem in
+computer vision. Methods that find correspondences and solve for the
+fundamental matrix offer high precision in most cases. Conversely, methods
+predicting pose directly using neural networks are more robust to limited
+overlap and can infer absolute translation scale, but at the expense of reduced
+precision. We show how to combine the best of both methods; our approach yields
+results that are both precise and robust, while also accurately inferring
+translation scales. At the heart of our model lies a Transformer that (1)
+learns to balance between solved and learned pose estimations, and (2) provides
+a prior to guide a solver. A comprehensive analysis supports our design choices
+and demonstrates that our method adapts flexibly to various feature extractors
+and correspondence estimators, showing state-of-the-art performance in 6DoF
+pose estimation on Matterport3D, InteriorNet, StreetLearn, and Map-free
+Relocalization.",cs.CV,['cs.CV']
+Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing,Ling Lo · Cheng Yeo · Hong-Han Shuai · Wen-Huang Cheng, ,https://arxiv.org/abs/2402.02583,,,DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing,"Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
+generation over the last few years. Although owning diverse and high-quality
+generation capabilities, translating these abilities to fine-grained image
+editing remains challenging. In this paper, we propose DiffEditor to rectify
+two weaknesses in existing diffusion-based image editing: (1) in complex
+scenarios, editing results often lack editing accuracy and exhibit unexpected
+artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
+imagine new content. In our solution, we introduce image prompts in
+fine-grained image editing, cooperating with the text prompt to better describe
+the editing content. To increase the flexibility while maintaining content
+consistency, we locally combine stochastic differential equation (SDE) into the
+ordinary differential equation (ODE) sampling. In addition, we incorporate
+regional score-based gradient guidance and a time travel strategy into the
+diffusion sampling, further improving the editing quality. Extensive
+experiments demonstrate that our method can efficiently achieve
+state-of-the-art performance on various fine-grained image editing tasks,
+including editing within a single image (e.g., object moving, resizing, and
+content dragging) and across images (e.g., appearance replacing and object
+pasting). Our source code is released at
+https://github.com/MC-E/DragonDiffusion.",cs.CV,"['cs.CV', 'cs.LG']"
+"Point, Segment and Count: A Generalized Framework for Object Counting",Zhizhong Huang · Mingliang Dai · Yi Zhang · Junping Zhang · Hongming Shan, ,https://arxiv.org/abs/2311.12386,,2311.12386.pdf,"Point, Segment and Count: A Generalized Framework for Object Counting","Class-agnostic object counting aims to count all objects in an image with
+respect to example boxes or class names, \emph{a.k.a} few-shot and zero-shot
+counting. In this paper, we propose a generalized framework for both few-shot
+and zero-shot object counting based on detection. Our framework combines the
+superior advantages of two foundation models without compromising their
+zero-shot capability: (\textbf{i}) SAM to segment all possible objects as mask
+proposals, and (\textbf{ii}) CLIP to classify proposals to obtain accurate
+object counts. However, this strategy meets the obstacles of efficiency
+overhead and the small crowded objects that cannot be localized and
+distinguished. To address these issues, our framework, termed PseCo, follows
+three steps: point, segment, and count. Specifically, we first propose a
+class-agnostic object localization to provide accurate but least point prompts
+for SAM, which consequently not only reduces computation costs but also avoids
+missing small objects. Furthermore, we propose a generalized object
+classification that leverages CLIP image/text embeddings as the classifier,
+following a hierarchical knowledge distillation to obtain discriminative
+classifications among hierarchical mask proposals. Extensive experimental
+results on FSC-147, COCO, and LVIS demonstrate that PseCo achieves
+state-of-the-art performance in both few-shot/zero-shot object
+counting/detection. Code: https://github.com/Hzzone/PseCo",cs.CV,['cs.CV']
+On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm,Peng Sun · Bei Shi · Daiwei Yu · Tao Lin, ,https://arxiv.org/abs/2312.03526,,2312.03526.pdf,On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm,"Contemporary machine learning requires training large neural networks on
+massive datasets and thus faces the challenges of high computational demands.
+Dataset distillation, as a recent emerging strategy, aims to compress
+real-world datasets for efficient training. However, this line of research
+currently struggle with large-scale and high-resolution datasets, hindering its
+practicality and feasibility. To this end, we re-examine the existing dataset
+distillation methods and identify three properties required for large-scale
+real-world applications, namely, realism, diversity, and efficiency. As a
+remedy, we propose RDED, a novel computationally-efficient yet effective data
+distillation paradigm, to enable both diversity and realism of the distilled
+data. Extensive empirical results over various neural architectures and
+datasets demonstrate the advancement of RDED: we can distill the full
+ImageNet-1K to a small dataset comprising 10 images per class within 7 minutes,
+achieving a notable 42% top-1 accuracy with ResNet-18 on a single RTX-4090 GPU
+(while the SOTA only achieves 21% but requires 6 hours).",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+3DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surfaces,Linyi Jin · Nilesh Kulkarni · David Fouhey,https://jinlinyi.github.io/3DFIRES/,https://arxiv.org/abs/2403.08768,,2403.08768.pdf,3DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surface,"This paper introduces 3DFIRES, a novel system for scene-level 3D
+reconstruction from posed images. Designed to work with as few as one view,
+3DFIRES reconstructs the complete geometry of unseen scenes, including hidden
+surfaces. With multiple view inputs, our method produces full reconstruction
+within all camera frustums. A key feature of our approach is the fusion of
+multi-view information at the feature level, enabling the production of
+coherent and comprehensive 3D reconstruction. We train our system on
+non-watertight scans from large-scale real scene dataset. We show it matches
+the efficacy of single-view reconstruction methods with only one input and
+surpasses existing techniques in both quantitative and qualitative measures for
+sparse-view 3D reconstruction.",cs.CV,['cs.CV']
+AlignMiF: Geometry-Aligned Multimodal Implicit Field for Enhanced LiDAR-Camera Joint Synthesis,Tao Tang · Guangrun Wang · Yixing Lao · Peng Chen · Jie Liu · Liang Lin · Kaicheng Yu · Xiaodan Liang, ,https://arxiv.org/abs/2402.17483,,2402.17483.pdf,AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis,"Neural implicit fields have been a de facto standard in novel view synthesis.
+Recently, there exist some methods exploring fusing multiple modalities within
+a single field, aiming to share implicit features from different modalities to
+enhance reconstruction performance. However, these modalities often exhibit
+misaligned behaviors: optimizing for one modality, such as LiDAR, can adversely
+affect another, like camera performance, and vice versa. In this work, we
+conduct comprehensive analyses on the multimodal implicit field of LiDAR-camera
+joint synthesis, revealing the underlying issue lies in the misalignment of
+different sensors. Furthermore, we introduce AlignMiF, a geometrically aligned
+multimodal implicit field with two proposed modules: Geometry-Aware Alignment
+(GAA) and Shared Geometry Initialization (SGI). These modules effectively align
+the coarse geometry across different modalities, significantly enhancing the
+fusion process between LiDAR and camera data. Through extensive experiments
+across various datasets and scenes, we demonstrate the effectiveness of our
+approach in facilitating better interaction between LiDAR and camera modalities
+within a unified neural field. Specifically, our proposed AlignMiF, achieves
+remarkable improvement over recent implicit fusion methods (+2.01 and +3.11
+image PSNR on the KITTI-360 and Waymo datasets) and consistently surpasses
+single modality performance (13.8% and 14.2% reduction in LiDAR Chamfer
+Distance on the respective datasets).",cs.CV,['cs.CV']
+Facial Identity Anonymization via Intrinsic and Extrinsic Attention Distraction,Zhenzhong Kuang · Xiaochen Yang · Yingjie Shen · Chao Hu · Jun Yu, ,https://arxiv.org/abs/2309.04228,,2309.04228.pdf,FIVA: Facial Image and Video Anonymization and Anonymization Defense,"In this paper, we present a new approach for facial anonymization in images
+and videos, abbreviated as FIVA. Our proposed method is able to maintain the
+same face anonymization consistently over frames with our suggested
+identity-tracking and guarantees a strong difference from the original face.
+FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work
+considers the important security issue of reconstruction attacks and
+investigates adversarial noise, uniform noise, and parameter noise to disrupt
+reconstruction attacks. In this regard, we apply different defense and
+protection methods against these privacy threats to demonstrate the scalability
+of FIVA. On top of this, we also show that reconstruction attack models can be
+used for detection of deep fakes. Last but not least, we provide experimental
+results showing how FIVA can even enable face swapping, which is purely trained
+on a single target image.",cs.CV,['cs.CV']
+EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation,Md Mostafijur Rahman · Mustafa Munir · Radu Marculescu,https://github.com/SLDGroup/EMCAD,https://arxiv.org/abs/2405.06880,,2405.06880.pdf,EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation,"An efficient and effective decoding mechanism is crucial in medical image
+segmentation, especially in scenarios with limited computational resources.
+However, these decoding mechanisms usually come with high computational costs.
+To address this concern, we introduce EMCAD, a new efficient multi-scale
+convolutional attention decoder, designed to optimize both performance and
+computational efficiency. EMCAD leverages a unique multi-scale depth-wise
+convolution block, significantly enhancing feature maps through multi-scale
+convolutions. EMCAD also employs channel, spatial, and grouped (large-kernel)
+gated attention mechanisms, which are highly effective at capturing intricate
+spatial relationships while focusing on salient regions. By employing group and
+depth-wise convolution, EMCAD is very efficient and scales well (e.g., only
+1.91M parameters and 0.381G FLOPs are needed when using a standard encoder).
+Our rigorous evaluations across 12 datasets that belong to six medical image
+segmentation tasks reveal that EMCAD achieves state-of-the-art (SOTA)
+performance with 79.4% and 80.3% reduction in #Params and #FLOPs, respectively.
+Moreover, EMCAD's adaptability to different encoders and versatility across
+segmentation tasks further establish EMCAD as a promising tool, advancing the
+field towards more efficient and accurate medical image analysis. Our
+implementation is available at https://github.com/SLDGroup/EMCAD.",eess.IV,"['eess.IV', 'cs.CV']"
+UniDepth: Universal Monocular Metric Depth Estimation,Luigi Piccinelli · Yung-Hsu Yang · Christos Sakaridis · Mattia Segu · Siyuan Li · Luc Van Gool · Fisher Yu,https://github.com/lpiccinelli-eth/unidepth,https://arxiv.org/abs/2403.18913,,2403.18913.pdf,UniDepth: Universal Monocular Metric Depth Estimation,"Accurate monocular metric depth estimation (MMDE) is crucial to solving
+downstream tasks in 3D perception and modeling. However, the remarkable
+accuracy of recent MMDE methods is confined to their training domains. These
+methods fail to generalize to unseen domains even in the presence of moderate
+domain gaps, which hinders their practical applicability. We propose a new
+model, UniDepth, capable of reconstructing metric 3D scenes from solely single
+images across domains. Departing from the existing MMDE methods, UniDepth
+directly predicts metric 3D points from the input image at inference time
+without any additional information, striving for a universal and flexible MMDE
+solution. In particular, UniDepth implements a self-promptable camera module
+predicting dense camera representation to condition depth features. Our model
+exploits a pseudo-spherical output representation, which disentangles camera
+and depth representations. In addition, we propose a geometric invariance loss
+that promotes the invariance of camera-prompted depth features. Thorough
+evaluations on ten datasets in a zero-shot regime consistently demonstrate the
+superior performance of UniDepth, even when compared with methods directly
+trained on the testing domains. Code and models are available at:
+https://github.com/lpiccinelli-eth/unidepth",cs.CV,['cs.CV']
+Learning from Synthetic Human Group Activities,Che-Jui Chang · Danrui Li · Deep Patel · Parth Goel · Seonghyeon Moon · Samuel Sohn · Honglu Zhou · Sejong Yoon · Vladimir Pavlovic · Mubbasir Kapadia,https://cjerry1243.github.io/M3Act/,https://arxiv.org/abs/2306.16772,,2306.16772.pdf,M3Act: Learning from Synthetic Human Group Activities,"The study of complex human interactions and group activities has become a
+focal point in human-centric computer vision. However, progress in related
+tasks is often hindered by the challenges of obtaining large-scale labeled
+datasets from real-world scenarios. To address the limitation, we introduce
+M3Act, a synthetic data generator for multi-view multi-group multi-person human
+atomic actions and group activities. Powered by Unity Engine, M3Act features
+multiple semantic groups, highly diverse and photorealistic images, and a
+comprehensive set of annotations, which facilitates the learning of
+human-centered tasks across single-person, multi-person, and multi-group
+conditions. We demonstrate the advantages of M3Act across three core
+experiments. The results suggest our synthetic dataset can significantly
+improve the performance of several downstream methods and replace real-world
+datasets to reduce cost. Notably, M3Act improves the state-of-the-art MOTRv2 on
+DanceTrack dataset, leading to a hop on the leaderboard from 10th to 2nd place.
+Moreover, M3Act opens new research for controllable 3D group activity
+generation. We define multiple metrics and propose a competitive baseline for
+the novel task. Our code and data are available at our project page:
+http://cjerry1243.github.io/M3Act.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Online Task-Free Continual Generative and Discriminative Learning via Dynamic Cluster Memory,飞 叶 · Adrian Bors, ,,https://ojs.aaai.org/index.php/AAAI/article/view/29582,,,,,nan
+AMU-Tuning: Learning Effective Bias for CLIP-based Few-shot Classification,Yuwei Tang · ZhenYi Lin · Qilong Wang · Pengfei Zhu · Qinghua Hu, ,https://arxiv.org/abs/2404.08958,,2404.08958.pdf,AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning,"Recently, pre-trained vision-language models (e.g., CLIP) have shown great
+potential in few-shot learning and attracted a lot of research interest.
+Although efforts have been made to improve few-shot ability of CLIP, key
+factors on the effectiveness of existing methods have not been well studied,
+limiting further exploration of CLIP's potential in few-shot learning. In this
+paper, we first introduce a unified formulation to analyze CLIP-based few-shot
+learning methods from a perspective of logit bias, which encourages us to learn
+an effective logit bias for further improving performance of CLIP-based
+few-shot learning methods. To this end, we disassemble three key components
+involved in computation of logit bias (i.e., logit features, logit predictor,
+and logit fusion) and empirically analyze the effect on performance of few-shot
+classification. Based on analysis of key components, this paper proposes a
+novel AMU-Tuning method to learn effective logit bias for CLIP-based few-shot
+classification. Specifically, our AMU-Tuning predicts logit bias by exploiting
+the appropriate $\underline{\textbf{A}}$uxiliary features, which are fed into
+an efficient feature-initialized linear classifier with
+$\underline{\textbf{M}}$ulti-branch training. Finally, an
+$\underline{\textbf{U}}$ncertainty-based fusion is developed to incorporate
+logit bias into CLIP for few-shot classification. The experiments are conducted
+on several widely used benchmarks, and the results show AMU-Tuning clearly
+outperforms its counterparts while achieving state-of-the-art performance of
+CLIP-based few-shot learning without bells and whistles.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+CoGS: Controllable Gaussian Splatting,Heng Yu · Joel Julin · Zoltán Á. Milacski · Koichiro Niinuma · László A. Jeni,https://cogs2024.github.io,https://arxiv.org/abs/2312.05664,,2312.05664.pdf,CoGS: Controllable Gaussian Splatting,"Capturing and re-animating the 3D structure of articulated objects present
+significant barriers. On one hand, methods requiring extensively calibrated
+multi-view setups are prohibitively complex and resource-intensive, limiting
+their practical applicability. On the other hand, while single-camera Neural
+Radiance Fields (NeRFs) offer a more streamlined approach, they have excessive
+training and rendering costs. 3D Gaussian Splatting would be a suitable
+alternative but for two reasons. Firstly, existing methods for 3D dynamic
+Gaussians require synchronized multi-view cameras, and secondly, the lack of
+controllability in dynamic scenarios. We present CoGS, a method for
+Controllable Gaussian Splatting, that enables the direct manipulation of scene
+elements, offering real-time control of dynamic scenes without the prerequisite
+of pre-computing control signals. We evaluated CoGS using both synthetic and
+real-world datasets that include dynamic objects that differ in degree of
+difficulty. In our evaluations, CoGS consistently outperformed existing dynamic
+and controllable neural representations in terms of visual fidelity.",cs.CV,['cs.CV']
+Neural Spline Fields for Burst Image Fusion and Layer Separation,Ilya Chugunov · David Shustin · Ruyu Yan · Chenyang Lei · Felix Heide, ,https://arxiv.org/abs/2312.14235,,2312.14235.pdf,Neural Spline Fields for Burst Image Fusion and Layer Separation,"Each photo in an image burst can be considered a sample of a complex 3D
+scene: the product of parallax, diffuse and specular materials, scene motion,
+and illuminant variation. While decomposing all of these effects from a stack
+of misaligned images is a highly ill-conditioned task, the conventional
+align-and-merge burst pipeline takes the other extreme: blending them into a
+single image. In this work, we propose a versatile intermediate representation:
+a two-layer alpha-composited image plus flow model constructed with neural
+spline fields -- networks trained to map input coordinates to spline control
+points. Our method is able to, during test-time optimization, jointly fuse a
+burst image capture into one high-resolution reconstruction and decompose it
+into transmission and obstruction layers. Then, by discarding the obstruction
+layer, we can perform a range of tasks including seeing through occlusions,
+reflection suppression, and shadow removal. Validated on complex synthetic and
+in-the-wild captures we find that, with no post-processing steps or learned
+priors, our generalizable model is able to outperform existing dedicated
+single-image and multi-view obstruction removal approaches.",cs.CV,['cs.CV']
+Object Recognition as Next Token Prediction,Kaiyu Yue · Bor-Chun Chen · Jonas Geiping · Hengduo Li · Tom Goldstein · Ser-Nam Lim,https://github.com/kaiyuyue/nxtp,,https://www.semanticscholar.org/paper/Object-Recognition-as-Next-Token-Prediction-Yue-Chen/529a3164a4ef5c227b6a775f73936866cb51d72f,,,,,nan
+GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models,Taoran Yi · Jiemin Fang · Junjie Wang · Guanjun Wu · Lingxi Xie · Xiaopeng Zhang · Wenyu Liu · Qi Tian · Xinggang Wang,https://taoranyi.com/gaussiandreamer/,https://arxiv.org/abs/2310.08529v3,,2310.08529v3.pdf,GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models,"In recent times, the generation of 3D assets from text prompts has shown
+impressive results. Both 2D and 3D diffusion models can help generate decent 3D
+objects based on prompts. 3D diffusion models have good 3D consistency, but
+their quality and generalization are limited as trainable 3D data is expensive
+and hard to obtain. 2D diffusion models enjoy strong abilities of
+generalization and fine generation, but 3D consistency is hard to guarantee.
+This paper attempts to bridge the power from the two types of diffusion models
+via the recent explicit and efficient 3D Gaussian splatting representation. A
+fast 3D object generation framework, named as GaussianDreamer, is proposed,
+where the 3D diffusion model provides priors for initialization and the 2D
+diffusion model enriches the geometry and appearance. Operations of noisy point
+growing and color perturbation are introduced to enhance the initialized
+Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D
+avatar within 15 minutes on one GPU, much faster than previous methods, while
+the generated instances can be directly rendered in real time. Demos and code
+are available at https://taoranyi.com/gaussiandreamer/.",cs.CV,"['cs.CV', 'cs.GR']"
+APISR: Anime Production Inspired Real-World Anime Super-Resolution,Boyang Wang · Fengyu Yang · Xihang Yu · Chao Zhang · Hanbin Zhao, ,https://arxiv.org/abs/2403.01598,,2403.01598.pdf,APISR: Anime Production Inspired Real-World Anime Super-Resolution,"While real-world anime super-resolution (SR) has gained increasing attention
+in the SR community, existing methods still adopt techniques from the
+photorealistic domain. In this paper, we analyze the anime production workflow
+and rethink how to use characteristics of it for the sake of the real-world
+anime SR. First, we argue that video networks and datasets are not necessary
+for anime SR due to the repetition use of hand-drawing frames. Instead, we
+propose an anime image collection pipeline by choosing the least compressed and
+the most informative frames from the video sources. Based on this pipeline, we
+introduce the Anime Production-oriented Image (API) dataset. In addition, we
+identify two anime-specific challenges of distorted and faint hand-drawn lines
+and unwanted color artifacts. We address the first issue by introducing a
+prediction-oriented compression module in the image degradation model and a
+pseudo-ground truth preparation with enhanced hand-drawn lines. In addition, we
+introduce the balanced twin perceptual loss combining both anime and
+photorealistic high-level features to mitigate unwanted color artifacts and
+increase visual clarity. We evaluate our method through extensive experiments
+on the public benchmark, showing our method outperforms state-of-the-art anime
+dataset-trained approaches.",eess.IV,"['eess.IV', 'cs.AI', 'cs.CV']"
+Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation,Zhiwei Yang · Kexue Fu · Minghong Duan · Linhao Qu · Shuo Wang · Zhijian Song,https://github.com/zwyang6/SeCo,https://arxiv.org/abs/2402.18467,,2402.18467.pdf,Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation,"Weakly supervised semantic segmentation (WSSS) with image-level labels aims
+to achieve segmentation tasks without dense annotations. However, attributed to
+the frequent coupling of co-occurring objects and the limited supervision from
+image-level labels, the challenging co-occurrence problem is widely present and
+leads to false activation of objects in WSSS. In this work, we devise a
+'Separate and Conquer' scheme SeCo to tackle this issue from dimensions of
+image space and feature space. In the image space, we propose to 'separate' the
+co-occurring objects with image decomposition by subdividing images into
+patches. Importantly, we assign each patch a category tag from Class Activation
+Maps (CAMs), which spatially helps remove the co-context bias and guide the
+subsequent representation. In the feature space, we propose to 'conquer' the
+false activation by enhancing semantic representation with multi-granularity
+knowledge contrast. To this end, a dual-teacher-single-student architecture is
+designed and tag-guided contrast is conducted, which guarantee the correctness
+of knowledge and further facilitate the discrepancy among co-contexts. We
+streamline the multi-staged WSSS pipeline end-to-end and tackle this issue
+without external supervision. Extensive experiments are conducted, validating
+the efficiency of our method and the superiority over previous single-staged
+and even multi-staged competitors on PASCAL VOC and MS COCO. Code is available
+at https://github.com/zwyang6/SeCo.git.",cs.CV,['cs.CV']
+Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation,Sixian Zhang · Xinyao Yu · Xinhang Song · Xiaohan Wang · Shuqiang Jiang, ,,http://vipl.ict.ac.cn/en/news/researchevents/202403/t20240315_207762.html,,,,,nan
+MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis,Dewei Zhou · You Li · Fan Ma · Xiaoting Zhang · Yi Yang, ,https://arxiv.org/abs/2402.05408,,2402.05408.pdf,MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis,"We present a Multi-Instance Generation (MIG) task, simultaneously generating
+multiple instances with diverse controls in one image. Given a set of
+predefined coordinates and their corresponding descriptions, the task is to
+ensure that generated instances are accurately at the designated locations and
+that all instances' attributes adhere to their corresponding description. This
+broadens the scope of current research on Single-instance generation, elevating
+it to a more versatile and practical dimension. Inspired by the idea of divide
+and conquer, we introduce an innovative approach named Multi-Instance
+Generation Controller (MIGC) to address the challenges of the MIG task.
+Initially, we break down the MIG task into several subtasks, each involving the
+shading of a single instance. To ensure precise shading for each instance, we
+introduce an instance enhancement attention mechanism. Lastly, we aggregate all
+the shaded instances to provide the necessary information for accurately
+generating multiple instances in stable diffusion (SD). To evaluate how well
+generation models perform on the MIG task, we provide a COCO-MIG benchmark
+along with an evaluation pipeline. Extensive experiments were conducted on the
+proposed COCO-MIG benchmark, as well as on various commonly used benchmarks.
+The evaluation results illustrate the exceptional control capabilities of our
+model in terms of quantity, position, attribute, and interaction. Code and
+demos will be released at https://migcproject.github.io/.",cs.CV,['cs.CV']
+Transfer CLIP for Generalizable Image Denoising,Jun Cheng · Dong Liang · Shan Tan,https://github.com/alwaysuu/CLIPDenoising,https://arxiv.org/abs/2403.15132,,,Transfer CLIP for Generalizable Image Denoising,"Image denoising is a fundamental task in computer vision. While prevailing
+deep learning-based supervised and self-supervised methods have excelled in
+eliminating in-distribution noise, their susceptibility to out-of-distribution
+(OOD) noise remains a significant challenge. The recent emergence of
+contrastive language-image pre-training (CLIP) model has showcased exceptional
+capabilities in open-world image recognition and segmentation. Yet, the
+potential for leveraging CLIP to enhance the robustness of low-level tasks
+remains largely unexplored. This paper uncovers that certain dense features
+extracted from the frozen ResNet image encoder of CLIP exhibit
+distortion-invariant and content-related properties, which are highly desirable
+for generalizable denoising. Leveraging these properties, we devise an
+asymmetrical encoder-decoder denoising network, which incorporates dense
+features including the noisy image and its multi-scale features from the frozen
+ResNet encoder of CLIP into a learnable image decoder to achieve generalizable
+denoising. The progressive feature augmentation strategy is further proposed to
+mitigate feature overfitting and improve the robustness of the learnable
+decoder. Extensive experiments and comparisons conducted across diverse OOD
+noises, including synthetic noise, real-world sRGB noise, and low-dose CT image
+noise, demonstrate the superior generalization ability of our method.",cs.CV,"['cs.CV', 'eess.IV']"
+Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning,Yiwen Ye · Yutong Xie · Jianpeng Zhang · Ziyang Chen · Qi Wu · Yong Xia, ,https://arxiv.org/abs/2311.17597,,2311.17597.pdf,Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning,"Self-supervised learning is an efficient pre-training method for medical
+image analysis. However, current research is mostly confined to
+specific-modality data pre-training, consuming considerable time and resources
+without achieving universality across different modalities. A straightforward
+solution is combining all modality data for joint self-supervised pre-training,
+which poses practical challenges. Firstly, our experiments reveal conflicts in
+representation learning as the number of modalities increases. Secondly,
+multi-modal data collected in advance cannot cover all real-world scenarios. In
+this paper, we reconsider versatile self-supervised learning from the
+perspective of continual learning and propose MedCoSS, a continuous
+self-supervised learning approach for multi-modal medical data. Unlike joint
+self-supervised learning, MedCoSS assigns different modality data to different
+training stages, forming a multi-stage pre-training process. To balance modal
+conflicts and prevent catastrophic forgetting, we propose a rehearsal-based
+continual learning method. We introduce the k-means sampling strategy to retain
+data from previous modalities and rehearse it when learning new modalities.
+Instead of executing the pretext task on buffer data, a feature distillation
+strategy and an intra-modal mixup strategy are applied to these data for
+knowledge retention. We conduct continuous self-supervised pre-training on a
+large-scale multi-modal unlabeled dataset, including clinical reports, X-rays,
+CT scans, MRI scans, and pathological images. Experimental results demonstrate
+MedCoSS's exceptional generalization ability across nine downstream datasets
+and its significant scalability in integrating new modality data. Code and
+pre-trained weight are available at https://github.com/yeerwen/MedCoSS.",cs.CV,['cs.CV']
+OmniVid: A Generative Framework for Universal Video Understanding,Junke Wang · Dongdong Chen · Chong Luo · Bo He · Lu Yuan · Zuxuan Wu · Yu-Gang Jiang, ,https://arxiv.org/abs/2403.17935,,2403.17935.pdf,OmniVid: A Generative Framework for Universal Video Understanding,"The core of video understanding tasks, such as recognition, captioning, and
+tracking, is to automatically detect objects or actions in a video and analyze
+their temporal evolution. Despite sharing a common goal, different tasks often
+rely on distinct model architectures and annotation formats. In contrast,
+natural language processing benefits from a unified output space, i.e., text
+sequences, which simplifies the training of powerful foundational language
+models, such as GPT-3, with extensive training corpora. Inspired by this, we
+seek to unify the output space of video understanding tasks by using languages
+as labels and additionally introducing time and box tokens. In this way, a
+variety of video tasks could be formulated as video-grounded token generation.
+This enables us to address various types of video tasks, including
+classification (such as action recognition), captioning (covering clip
+captioning, video question answering, and dense video captioning), and
+localization tasks (such as visual object tracking) within a fully shared
+encoder-decoder architecture, following a generative framework. Through
+comprehensive experiments, we demonstrate such a simple and straightforward
+idea is quite effective and can achieve state-of-the-art or competitive results
+on seven video benchmarks, providing a novel perspective for more universal
+video understanding. Code is available at https://github.com/wangjk666/OmniVid.",cs.CV,['cs.CV']
+Learning from One Continuous Video Stream,Joao Carreira · Michael King · Viorica Patraucean · Dilara Gokay · Catalin Ionescu · Yi Yang · Daniel Zoran · Joseph Heyward · Carl Doersch · Yusuf Aytar · Dima Damen · Andrew Zisserman, ,https://arxiv.org/abs/2312.00598,,2312.00598.pdf,Learning from One Continuous Video Stream,"We introduce a framework for online learning from a single continuous video
+stream -- the way people and animals learn, without mini-batches, data
+augmentation or shuffling. This poses great challenges given the high
+correlation between consecutive video frames and there is very little prior
+work on it. Our framework allows us to do a first deep dive into the topic and
+includes a collection of streams and tasks composed from two existing video
+datasets, plus methodology for performance evaluation that considers both
+adaptation and generalization. We employ pixel-to-pixel modelling as a
+practical and flexible way to switch between pre-training and single-stream
+evaluation as well as between arbitrary tasks, without ever requiring changes
+to models and always using the same pixel loss. Equipped with this framework we
+obtained large single-stream learning gains from pre-training with a novel
+family of future prediction tasks, found that momentum hurts, and that the pace
+of weight updates matters. The combination of these insights leads to matching
+the performance of IID learning with batch size 1, when using the same
+architecture and without costly replay buffers.",cs.CV,"['cs.CV', 'cs.AI']"
+Blur2Blur: Blur Conversion for Unsupervised Image Deblurring on Unknown Domains,Bang-Dang Pham · Phong Tran · Anh Tran · Cuong Pham · Rang Nguyen · Minh Hoai,https://zero1778.github.io/blur2blur/,https://arxiv.org/abs/2403.16205,,2403.16205.pdf,Blur2Blur: Blur Conversion for Unsupervised Image Deblurring on Unknown Domains,"This paper presents an innovative framework designed to train an image
+deblurring algorithm tailored to a specific camera device. This algorithm works
+by transforming a blurry input image, which is challenging to deblur, into
+another blurry image that is more amenable to deblurring. The transformation
+process, from one blurry state to another, leverages unpaired data consisting
+of sharp and blurry images captured by the target camera device. Learning this
+blur-to-blur transformation is inherently simpler than direct blur-to-sharp
+conversion, as it primarily involves modifying blur patterns rather than the
+intricate task of reconstructing fine image details. The efficacy of the
+proposed approach has been demonstrated through comprehensive experiments on
+various benchmarks, where it significantly outperforms state-of-the-art methods
+both quantitatively and qualitatively. Our code and data are available at
+https://zero1778.github.io/blur2blur/",cs.CV,['cs.CV']
+Video Harmonization with Triplet Spatio-Temporal Variation Patterns,Zonghui Guo · XinYu Han · Jie Zhang · Shiguang Shan · Haiyong Zheng,https://github.com/zhenglab/VideoTripletTransformer,,http://vipl.ict.ac.cn/en/news/researchevents/202403/t20240315_207762.html,,,,,nan
+D3still: Decoupled Differential Distillation for Asymmetric Image Retrieval,Yi Xie · Yihong Lin · Wenjie Cai · Xuemiao Xu · Huaidong Zhang · Yong Du · Shengfeng He, ,https://arxiv.org/abs/2403.01431,,2403.01431.pdf,Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval,"The task of composed image retrieval (CIR) aims to retrieve images based on
+the query image and the text describing the users' intent. Existing methods
+have made great progress with the advanced large vision-language (VL) model in
+CIR task, however, they generally suffer from two main issues: lack of labeled
+triplets for model training and difficulty of deployment on resource-restricted
+environments when deploying the large vision-language model. To tackle the
+above problems, we propose Image2Sentence based Asymmetric zero-shot composed
+image retrieval (ISA), which takes advantage of the VL model and only relies on
+unlabeled images for composition learning. In the framework, we propose a new
+adaptive token learner that maps an image to a sentence in the word embedding
+space of VL model. The sentence adaptively captures discriminative visual
+information and is further integrated with the text modifier. An asymmetric
+structure is devised for flexible deployment, in which the lightweight model is
+adopted for the query side while the large VL model is deployed on the gallery
+side. The global contrastive distillation and the local alignment
+regularization are adopted for the alignment between the light model and the VL
+model for CIR task. Our experiments demonstrate that the proposed ISA could
+better cope with the real retrieval scenarios and further improve retrieval
+accuracy and efficiency.",cs.CV,['cs.CV']
+DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks,Jiaxin Zhang · Dezhi Peng · Chongyu Liu · Peirong Zhang · Lianwen Jin,https://github.com/ZZZHANG-jx/DocRes,https://arxiv.org/abs/2405.04408,,2405.04408.pdf,DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks,"Document image restoration is a crucial aspect of Document AI systems, as the
+quality of document images significantly influences the overall performance.
+Prevailing methods address distinct restoration tasks independently, leading to
+intricate systems and the incapability to harness the potential synergies of
+multi-task learning. To overcome this challenge, we propose DocRes, a
+generalist model that unifies five document image restoration tasks including
+dewarping, deshadowing, appearance enhancement, deblurring, and binarization.
+To instruct DocRes to perform various restoration tasks, we propose a novel
+visual prompt approach called Dynamic Task-Specific Prompt (DTSPrompt). The
+DTSPrompt for different tasks comprises distinct prior features, which are
+additional characteristics extracted from the input image. Beyond its role as a
+cue for task-specific execution, DTSPrompt can also serve as supplementary
+information to enhance the model's performance. Moreover, DTSPrompt is more
+flexible than prior visual prompt approaches as it can be seamlessly applied
+and adapted to inputs with high and variable resolutions. Experimental results
+demonstrate that DocRes achieves competitive or superior performance compared
+to existing state-of-the-art task-specific models. This underscores the
+potential of DocRes across a broader spectrum of document image restoration
+tasks. The source code is publicly available at
+https://github.com/ZZZHANG-jx/DocRes",cs.CV,['cs.CV']
+Unified Entropy Optimization for Open-Set Test-Time Adaptation,Zhengqing Gao · Xu-Yao Zhang · Cheng-Lin Liu,https://github.com/gaozhengqing/UniEnt,https://arxiv.org/abs/2404.06065,,2404.06065.pdf,Unified Entropy Optimization for Open-Set Test-Time Adaptation,"Test-time adaptation (TTA) aims at adapting a model pre-trained on the
+labeled source domain to the unlabeled target domain. Existing methods usually
+focus on improving TTA performance under covariate shifts, while neglecting
+semantic shifts. In this paper, we delve into a realistic open-set TTA setting
+where the target domain may contain samples from unknown classes. Many
+state-of-the-art closed-set TTA methods perform poorly when applied to open-set
+scenarios, which can be attributed to the inaccurate estimation of data
+distribution and model confidence. To address these issues, we propose a simple
+but effective framework called unified entropy optimization (UniEnt), which is
+capable of simultaneously adapting to covariate-shifted in-distribution (csID)
+data and detecting covariate-shifted out-of-distribution (csOOD) data.
+Specifically, UniEnt first mines pseudo-csID and pseudo-csOOD samples from test
+data, followed by entropy minimization on the pseudo-csID data and entropy
+maximization on the pseudo-csOOD data. Furthermore, we introduce UniEnt+ to
+alleviate the noise caused by hard data partition leveraging sample-level
+confidence. Extensive experiments on CIFAR benchmarks and Tiny-ImageNet-C show
+the superiority of our framework. The code is available at
+https://github.com/gaozhengqing/UniEnt",cs.CV,['cs.CV']
+Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology,Wenhao Tang · Fengtao ZHOU · Sheng Huang · Xiang Zhu · Yi Zhang · Bo Liu, ,https://arxiv.org/abs/2402.17228,,2402.17228.pdf,Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology,"Multiple instance learning (MIL) is the most widely used framework in
+computational pathology, encompassing sub-typing, diagnosis, prognosis, and
+more. However, the existing MIL paradigm typically requires an offline instance
+feature extractor, such as a pre-trained ResNet or a foundation model. This
+approach lacks the capability for feature fine-tuning within the specific
+downstream tasks, limiting its adaptability and performance. To address this
+issue, we propose a Re-embedded Regional Transformer (R$^2$T) for re-embedding
+the instance features online, which captures fine-grained local features and
+establishes connections across different regions. Unlike existing works that
+focus on pre-training powerful feature extractor or designing sophisticated
+instance aggregator, R$^2$T is tailored to re-embed instance features online.
+It serves as a portable module that can seamlessly integrate into mainstream
+MIL models. Extensive experimental results on common computational pathology
+tasks validate that: 1) feature re-embedding improves the performance of MIL
+models based on ResNet-50 features to the level of foundation model features,
+and further enhances the performance of foundation model features; 2) the
+R$^2$T can introduce more significant performance improvements to various MIL
+models; 3) R$^2$T-MIL, as an R$^2$T-enhanced AB-MIL, outperforms other latest
+methods by a large margin.The code is available at:
+https://github.com/DearCaat/RRT-MIL.",cs.CV,['cs.CV']
+Gradient-based Parameter Selection for Efficient Fine-Tuning,Zhi Zhang · Qizhe Zhang · Zijun Gao · Renrui Zhang · Ekaterina Shutova · Shiji Zhou · Shanghang Zhang, ,https://arxiv.org/abs/2312.10136,,2312.10136.pdf,Gradient-based Parameter Selection for Efficient Fine-Tuning,"With the growing size of pre-trained models, full fine-tuning and storing all
+the parameters for various downstream tasks is costly and infeasible. In this
+paper, we propose a new parameter-efficient fine-tuning method, Gradient-based
+Parameter Selection (GPS), demonstrating that only tuning a few selected
+parameters from the pre-trained model while keeping the remainder of the model
+frozen can generate similar or better performance compared with the full model
+fine-tuning method. Different from the existing popular and state-of-the-art
+parameter-efficient fine-tuning approaches, our method does not introduce any
+additional parameters and computational costs during both the training and
+inference stages. Another advantage is the model-agnostic and non-destructive
+property, which eliminates the need for any other design specific to a
+particular model. Compared with the full fine-tuning, GPS achieves 3.33%
+(91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the
+accuracy with tuning only 0.36% parameters of the pre-trained model on average
+over 24 image classification tasks; it also demonstrates a significant
+improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image
+segmentation task. Moreover, GPS achieves state-of-the-art performance compared
+with existing PEFT methods.",cs.CV,['cs.CV']
+UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model,Shuai Yuan · Lei Luo · Zhuo Hui · Can Pu · Xiaoyu Xiang · Rakesh Ranjan · Denis Demandolx, ,https://arxiv.org/abs/2405.02608,,2405.02608.pdf,UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model,"Traditional unsupervised optical flow methods are vulnerable to occlusions
+and motion boundaries due to lack of object-level information. Therefore, we
+propose UnSAMFlow, an unsupervised flow network that also leverages object
+information from the latest foundation model Segment Anything Model (SAM). We
+first include a self-supervised semantic augmentation module tailored to SAM
+masks. We also analyze the poor gradient landscapes of traditional smoothness
+losses and propose a new smoothness definition based on homography instead. A
+simple yet effective mask feature module has also been added to further
+aggregate features on the object level. With all these adaptations, our method
+produces clear optical flow estimation with sharp boundaries around objects,
+which outperforms state-of-the-art methods on both KITTI and Sintel datasets.
+Our method also generalizes well across domains and runs very efficiently.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Diversity-aware Channel Pruning for StyleGAN Compression,Jiwoo Chung · Sangeek Hyun · Sang-Heon Shim · Jae-Pil Heo,https://jiwoogit.github.io/DCP-GAN_site/,https://arxiv.org/abs/2403.13548,,2403.13548.pdf,Diversity-aware Channel Pruning for StyleGAN Compression,"StyleGAN has shown remarkable performance in unconditional image generation.
+However, its high computational cost poses a significant challenge for
+practical applications. Although recent efforts have been made to compress
+StyleGAN while preserving its performance, existing compressed models still lag
+behind the original model, particularly in terms of sample diversity. To
+overcome this, we propose a novel channel pruning method that leverages varying
+sensitivities of channels to latent vectors, which is a key factor in sample
+diversity. Specifically, by assessing channel importance based on their
+sensitivities to latent vector perturbations, our method enhances the diversity
+of samples in the compressed model. Since our method solely focuses on the
+channel pruning stage, it has complementary benefits with prior training
+schemes without additional training cost. Extensive experiments demonstrate
+that our method significantly enhances sample diversity across various
+datasets. Moreover, in terms of FID scores, our method not only surpasses
+state-of-the-art by a large margin but also achieves comparable scores with
+only half training iterations.",cs.CV,['cs.CV']
+Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning,Rashindrie Perera · Saman Halgamuge,https://github.com/rashindrie/DIPA,https://arxiv.org/abs/2403.04492,,2403.04492.pdf,Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning,"In this paper, we look at cross-domain few-shot classification which presents
+the challenging task of learning new classes in previously unseen domains with
+few labelled examples. Existing methods, though somewhat effective, encounter
+several limitations, which we alleviate through two significant improvements.
+First, we introduce a lightweight parameter-efficient adaptation strategy to
+address overfitting associated with fine-tuning a large number of parameters on
+small datasets. This strategy employs a linear transformation of pre-trained
+features, significantly reducing the trainable parameter count. Second, we
+replace the traditional nearest centroid classifier with a discriminative
+sample-aware loss function, enhancing the model's sensitivity to the inter- and
+intra-class variances within the training set for improved clustering in
+feature space. Empirical evaluations on the Meta-Dataset benchmark showcase
+that our approach not only improves accuracy up to 7.7\% and 5.3\% on
+previously seen and unseen datasets, respectively, but also achieves the above
+performance while being at least $\sim3\times$ more parameter-efficient than
+existing methods, establishing a new state-of-the-art in cross-domain few-shot
+learning. Our code is available at https://github.com/rashindrie/DIPA.",cs.CV,['cs.CV']
+FaceLift: Semi-supervised 3D Facial Landmark Localization,David Ferman · Pablo Garrido · Gaurav Bharaj,https://davidcferman.github.io/FaceLift/,https://arxiv.org/abs/2405.19646,,2405.19646.pdf,FaceLift: Semi-supervised 3D Facial Landmark Localization,"3D facial landmark localization has proven to be of particular use for
+applications, such as face tracking, 3D face modeling, and image-based 3D face
+reconstruction. In the supervised learning case, such methods usually rely on
+3D landmark datasets derived from 3DMM-based registration that often lack
+spatial definition alignment, as compared with that chosen by hand-labeled
+human consensus, e.g., how are eyebrow landmarks defined? This creates a gap
+between landmark datasets generated via high-quality 2D human labels and 3DMMs,
+and it ultimately limits their effectiveness. To address this issue, we
+introduce a novel semi-supervised learning approach that learns 3D landmarks by
+directly lifting (visible) hand-labeled 2D landmarks and ensures better
+definition alignment, without the need for 3D landmark datasets. To lift 2D
+landmarks to 3D, we leverage 3D-aware GANs for better multi-view consistency
+learning and in-the-wild multi-frame videos for robust cross-generalization.
+Empirical experiments demonstrate that our method not only achieves better
+definition alignment between 2D-3D landmarks but also outperforms other
+supervised learning 3D landmark localization methods on both 3DMM labeled and
+photogrammetric ground truth evaluation datasets. Project Page:
+https://davidcferman.github.io/FaceLift",cs.CV,['cs.CV']
+MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models,Sanjoy Chowdhury · Sayan Nag · Joseph K J · Balaji Vasan Srinivasan · Dinesh Manocha, ,https://arxiv.org/abs/2310.13772,,2310.13772.pdf,TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models,"We present TexFusion (Texture Diffusion), a new method to synthesize textures
+for given 3D geometries, using large-scale text-guided image diffusion models.
+In contrast to recent works that leverage 2D text-to-image diffusion models to
+distill 3D objects using a slow and fragile optimization process, TexFusion
+introduces a new 3D-consistent generation technique specifically designed for
+texture synthesis that employs regular diffusion model sampling on different 2D
+rendered views. Specifically, we leverage latent diffusion models, apply the
+diffusion model's denoiser on a set of 2D renders of the 3D object, and
+aggregate the different denoising predictions on a shared latent texture map.
+Final output RGB textures are produced by optimizing an intermediate neural
+color field on the decodings of 2D renders of the latent texture. We thoroughly
+validate TexFusion and show that we can efficiently generate diverse, high
+quality and globally coherent textures. We achieve state-of-the-art text-guided
+texture synthesis performance using only image diffusion models, while avoiding
+the pitfalls of previous distillation-based methods. The text-conditioning
+offers detailed control and we also do not rely on any ground truth 3D textures
+for training. This makes our method versatile and applicable to a broad range
+of geometry and texture types. We hope that TexFusion will advance AI-based
+texturing of 3D assets for applications in virtual reality, game design,
+simulation, and more.",cs.CV,"['cs.CV', 'cs.LG', 'I.3.3']"
+Intensity-Robust Autofocus for Spike Camera,Changqing Su · Zhiyuan Ye · Yongsheng Xiao · You Zhou · Zhen Cheng · Bo Xiong · Zhaofei Yu · Tiejun Huang, ,https://arxiv.org/abs/2405.16790,,2405.16790.pdf,SCSim: A Realistic Spike Cameras Simulator,"Spike cameras, with their exceptional temporal resolution, are
+revolutionizing high-speed visual applications. Large-scale synthetic datasets
+have significantly accelerated the development of these cameras, particularly
+in reconstruction and optical flow. However, current synthetic datasets for
+spike cameras lack sophistication. Addressing this gap, we introduce SCSim, a
+novel and more realistic spike camera simulator with a comprehensive noise
+model. SCSim is adept at autonomously generating driving scenarios and
+synthesizing corresponding spike streams. To enhance the fidelity of these
+streams, we've developed a comprehensive noise model tailored to the unique
+circuitry of spike cameras. Our evaluations demonstrate that SCSim outperforms
+existing simulation methods in generating authentic spike streams. Crucially,
+SCSim simplifies the creation of datasets, thereby greatly advancing
+spike-based visual tasks like reconstruction. Our project refers to
+https://github.com/Acnext/SCSim.",cs.CV,['cs.CV']
+SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields,"Quentin HERAU · Nathan Piasco · Moussab Bennehar · Luis Guiller,o Roldao Jimenez · Dzmitry Tsishkou · MigniotCyrille · Modélisation Information Systèmes · Cedric Demonceaux", ,https://arxiv.org/abs/2311.15803,,2311.15803.pdf,SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields,"In rapidly-evolving domains such as autonomous driving, the use of multiple
+sensors with different modalities is crucial to ensure high operational
+precision and stability. To correctly exploit the provided information by each
+sensor in a single common frame, it is essential for these sensors to be
+accurately calibrated. In this paper, we leverage the ability of Neural
+Radiance Fields (NeRF) to represent different sensors modalities in a common
+volumetric representation to achieve robust and accurate spatio-temporal sensor
+calibration. By designing a partitioning approach based on the visible part of
+the scene for each sensor, we formulate the calibration problem using only the
+overlapping areas. This strategy results in a more robust and accurate
+calibration that is less prone to failure. We demonstrate that our approach
+works on outdoor urban scenes by validating it on multiple established driving
+datasets. Results show that our method is able to get better accuracy and
+robustness compared to existing methods.",cs.CV,"['cs.CV', 'cs.RO']"
+Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations,Lei Fan · Jianxiong Zhou · Xiaoying Xing · Ying Wu, ,https://arxiv.org/abs/2311.17938,,2311.17938.pdf,Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations,"Active recognition, which allows intelligent agents to explore observations
+for better recognition performance, serves as a prerequisite for various
+embodied AI tasks, such as grasping, navigation and room arrangements. Given
+the evolving environment and the multitude of object classes, it is impractical
+to include all possible classes during the training stage. In this paper, we
+aim at advancing active open-vocabulary recognition, empowering embodied agents
+to actively perceive and classify arbitrary objects. However, directly adopting
+recent open-vocabulary classification models, like Contrastive Language Image
+Pretraining (CLIP), poses its unique challenges. Specifically, we observe that
+CLIP's performance is heavily affected by the viewpoint and occlusions,
+compromising its reliability in unconstrained embodied perception scenarios.
+Further, the sequential nature of observations in agent-environment
+interactions necessitates an effective method for integrating features that
+maintains discriminative strength for open-vocabulary classification. To
+address these issues, we introduce a novel agent for active open-vocabulary
+recognition. The proposed method leverages inter-frame and inter-concept
+similarities to navigate agent movements and to fuse features, without relying
+on class-specific knowledge. Compared to baseline CLIP model with 29.6%
+accuracy on ShapeNet dataset, the proposed agent could achieve 53.3% accuracy
+for open-vocabulary recognition, without any fine-tuning to the equipped CLIP
+model. Additional experiments conducted with the Habitat simulator further
+affirm the efficacy of our method.",cs.CV,['cs.CV']
+2S-UDF: A Novel Two-stage UDF Learning Method for Robust Non-watertight Model Reconstruction from Multi-view Images,Junkai Deng · Fei Hou · Xuhui Chen · Wencheng Wang · Ying He, ,https://arxiv.org/abs/2308.09302,,2308.09302.pdf,Robust Audio Anti-Spoofing with Fusion-Reconstruction Learning on Multi-Order Spectrograms,"Robust audio anti-spoofing has been increasingly challenging due to the
+recent advancements on deepfake techniques. While spectrograms have
+demonstrated their capability for anti-spoofing, complementary information
+presented in multi-order spectral patterns have not been well explored, which
+limits their effectiveness for varying spoofing attacks. Therefore, we propose
+a novel deep learning method with a spectral fusion-reconstruction strategy,
+namely S2pecNet, to utilise multi-order spectral patterns for robust audio
+anti-spoofing representations. Specifically, spectral patterns up to
+second-order are fused in a coarse-to-fine manner and two branches are designed
+for the fine-level fusion from the spectral and temporal contexts. A
+reconstruction from the fused representation to the input spectrograms further
+reduces the potential fused information loss. Our method achieved the
+state-of-the-art performance with an EER of 0.77% on a widely used dataset:
+ASVspoof2019 LA Challenge.",cs.SD,"['cs.SD', 'cs.AI', 'cs.MM', 'eess.AS']"
+Sheared Backpropagation for Finetuning Foundation Models,Zhiyuan Yu · Li Shen · Liang Ding · Xinmei Tian · Yixin Chen · Dacheng Tao, ,https://arxiv.org/abs/2402.15017,,2402.15017.pdf,Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning,"Foundation models have emerged as a powerful tool for many AI problems.
+Despite the tremendous success of foundation models, effective adaptation to
+new tasks, particularly those with limited labels, remains an open question and
+lacks theoretical understanding. An emerging solution with recent success in
+vision and NLP involves finetuning a foundation model on a selection of
+relevant tasks, before its adaptation to a target task with limited labeled
+samples. In this paper, we study the theoretical justification of this
+multitask finetuning approach. Our theoretical analysis reveals that with a
+diverse set of related tasks, this multitask finetuning leads to reduced error
+in the target task, in comparison to directly adapting the same pretrained
+model. We quantify the relationship between finetuning tasks and target tasks
+by diversity and consistency metrics, and further propose a practical task
+selection algorithm. We substantiate our theoretical claims with extensive
+empirical evidence. Further, we present results affirming our task selection
+algorithm adeptly chooses related finetuning tasks, providing advantages to the
+model performance on target tasks. We believe our study shed new light on the
+effective adaptation of foundation models to new tasks that lack abundant
+labels. Our code is available at
+https://github.com/OliverXUZY/Foudation-Model_Multitask.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CL']"
+DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion,Tom Van Wouwe · Seunghwan Lee · Antoine Falisse · Scott Delp · Karen Liu,https://diffusionposer.github.io/,https://arxiv.org/abs/2308.16682,,,DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion,"Motion capture from a limited number of body-worn sensors, such as inertial
+measurement units (IMUs) and pressure insoles, has important applications in
+health, human performance, and entertainment. Recent work has focused on
+accurately reconstructing whole-body motion from a specific sensor
+configuration using six IMUs. While a common goal across applications is to use
+the minimal number of sensors to achieve required accuracy, the optimal
+arrangement of the sensors might differ from application to application. We
+propose a single diffusion model, DiffusionPoser, which reconstructs human
+motion in real-time from an arbitrary combination of sensors, including IMUs
+placed at specified locations, and, pressure insoles. Unlike existing methods,
+our model grants users the flexibility to determine the number and arrangement
+of sensors tailored to the specific activity of interest, without the need for
+retraining. A novel autoregressive inferencing scheme ensures real-time motion
+reconstruction that closely aligns with measured sensor signals. The generative
+nature of DiffusionPoser ensures realistic behavior, even for
+degrees-of-freedom not directly measured. Qualitative results can be found on
+our website: https://diffusionposer.github.io/.",cs.CV,['cs.CV']
+DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning,Haoran Xu · Peixi Peng · Guang Tan · Yuan Li · Xinhai Xu · Yonghong Tian,https://github.com/kyoran/DMR,,https://link.springer.com/article/10.1007/s11704-023-2444-y,,,,,nan
+Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions,Runhao Zeng · Xiaoyong Chen · Jiaming Liang · Huisi Wu · Guang-Zhong Cao · Yong Guo, ,https://arxiv.org/abs/2403.20254,,2403.20254.pdf,Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions,"Temporal action detection (TAD) aims to locate action positions and recognize
+action categories in long-term untrimmed videos. Although many methods have
+achieved promising results, their robustness has not been thoroughly studied.
+In practice, we observe that temporal information in videos can be occasionally
+corrupted, such as missing or blurred frames. Interestingly, existing methods
+often incur a significant performance drop even if only one frame is affected.
+To formally evaluate the robustness, we establish two temporal corruption
+robustness benchmarks, namely THUMOS14-C and ActivityNet-v1.3-C. In this paper,
+we extensively analyze the robustness of seven leading TAD methods and obtain
+some interesting findings: 1) Existing methods are particularly vulnerable to
+temporal corruptions, and end-to-end methods are often more susceptible than
+those with a pre-trained feature extractor; 2) Vulnerability mainly comes from
+localization error rather than classification error; 3) When corruptions occur
+in the middle of an action instance, TAD models tend to yield the largest
+performance drop. Besides building a benchmark, we further develop a simple but
+effective robust training method to defend against temporal corruptions,
+through the FrameDrop augmentation and Temporal-Robust Consistency loss.
+Remarkably, our approach not only improves robustness but also yields promising
+improvements on clean data. We believe that this study will serve as a
+benchmark for future research in robust video analysis. Source code and models
+are available at https://github.com/Alvin-Zeng/temporal-robustness-benchmark.",cs.CV,['cs.CV']
+Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera,Jiye Lee · Hanbyul Joo,https://jiyewise.github.io/projects/MocapEvery,https://arxiv.org/abs/2401.00847,,2401.00847.pdf,Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera,"We present a lightweight and affordable motion capture method based on two
+smartwatches and a head-mounted camera. In contrast to the existing approaches
+that use six or more expert-level IMU devices, our approach is much more
+cost-effective and convenient. Our method can make wearable motion capture
+accessible to everyone everywhere, enabling 3D full-body motion capture in
+diverse environments. As a key idea to overcome the extreme sparsity and
+ambiguities of sensor inputs with different modalities, we integrate 6D head
+poses obtained from the head-mounted cameras for motion estimation. To enable
+capture in expansive indoor and outdoor scenes, we propose an algorithm to
+track and update floor level changes to define head poses, coupled with a
+multi-stage Transformer-based regression module. We also introduce novel
+strategies leveraging visual cues of egocentric images to further enhance the
+motion capture quality while reducing ambiguities. We demonstrate the
+performance of our method on various challenging scenarios, including complex
+outdoor environments and everyday motions including object interactions and
+social interactions among multiple individuals.",cs.CV,"['cs.CV', 'cs.GR']"
+SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design,Seokju Yun · Youngmin Ro,https://github.com/ysj9909/SHViT,https://arxiv.org/abs/2401.16456,,2401.16456.pdf,SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design,"Recently, efficient Vision Transformers have shown great performance with low
+latency on resource-constrained devices. Conventionally, they use 4x4 patch
+embeddings and a 4-stage structure at the macro level, while utilizing
+sophisticated attention with multi-head configuration at the micro level. This
+paper aims to address computational redundancy at all design levels in a
+memory-efficient manner. We discover that using larger-stride patchify stem not
+only reduces memory access costs but also achieves competitive performance by
+leveraging token representations with reduced spatial redundancy from the early
+stages. Furthermore, our preliminary analyses suggest that attention layers in
+the early stages can be substituted with convolutions, and several attention
+heads in the latter stages are computationally redundant. To handle this, we
+introduce a single-head attention module that inherently prevents head
+redundancy and simultaneously boosts accuracy by parallelly combining global
+and local information. Building upon our solutions, we introduce SHViT, a
+Single-Head Vision Transformer that obtains the state-of-the-art speed-accuracy
+tradeoff. For example, on ImageNet-1k, our SHViT-S4 is 3.3x, 8.1x, and 2.4x
+faster than MobileViTv2 x1.0 on GPU, CPU, and iPhone12 mobile device,
+respectively, while being 1.3% more accurate. For object detection and instance
+segmentation on MS COCO using Mask-RCNN head, our model achieves performance
+comparable to FastViT-SA12 while exhibiting 3.8x and 2.0x lower backbone
+latency on GPU and mobile device, respectively.",cs.CV,['cs.CV']
+Improved Self-Training for Test-Time Adaptation,Jing Ma, ,https://arxiv.org/abs/2309.14949v1,,2309.14949v1.pdf,Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalization,"Test-Time Adaptation aims to adapt source domain model to testing data at
+inference stage with success demonstrated in adapting to unseen corruptions.
+However, these attempts may fail under more challenging real-world scenarios.
+Existing works mainly consider real-world test-time adaptation under non-i.i.d.
+data stream and continual domain shift. In this work, we first complement the
+existing real-world TTA protocol with a globally class imbalanced testing set.
+We demonstrate that combining all settings together poses new challenges to
+existing methods. We argue the failure of state-of-the-art methods is first
+caused by indiscriminately adapting normalization layers to imbalanced testing
+data. To remedy this shortcoming, we propose a balanced batchnorm layer to swap
+out the regular batchnorm at inference stage. The new batchnorm layer is
+capable of adapting without biasing towards majority classes. We are further
+inspired by the success of self-training~(ST) in learning from unlabeled data
+and adapt ST for test-time adaptation. However, ST alone is prone to over
+adaption which is responsible for the poor performance under continual domain
+shift. Hence, we propose to improve self-training under continual domain shift
+by regularizing model updates with an anchored loss. The final TTA model,
+termed as TRIBE, is built upon a tri-net architecture with balanced batchnorm
+layers. We evaluate TRIBE on four datasets representing real-world TTA
+settings. TRIBE consistently achieves the state-of-the-art performance across
+multiple evaluation protocols. The code is available at
+\url{https://github.com/Gorilla-Lab-SCUT/TRIBE}.",cs.LG,"['cs.LG', 'cs.CV']"
+APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation,Weizhao He · Yang Zhang · Wei Zhuo · Linlin Shen · Jiaqi Yang · Songhe Deng · Liang Sun, ,https://arxiv.org/abs/2405.15265,,2405.15265.pdf,Cross-Domain Few-Shot Semantic Segmentation via Doubly Matching Transformation,"Cross-Domain Few-shot Semantic Segmentation (CD-FSS) aims to train
+generalized models that can segment classes from different domains with a few
+labeled images. Previous works have proven the effectiveness of feature
+transformation in addressing CD-FSS. However, they completely rely on support
+images for feature transformation, and repeatedly utilizing a few support
+images for each class may easily lead to overfitting and overlooking
+intra-class appearance differences. In this paper, we propose a Doubly Matching
+Transformation-based Network (DMTNet) to solve the above issue. Instead of
+completely relying on support images, we propose Self-Matching Transformation
+(SMT) to construct query-specific transformation matrices based on query images
+themselves to transform domain-specific query features into domain-agnostic
+ones. Calculating query-specific transformation matrices can prevent
+overfitting, especially for the meta-testing stage where only one or several
+images are used as support images to segment hundreds or thousands of images.
+After obtaining domain-agnostic features, we exploit a Dual Hypercorrelation
+Construction (DHC) module to explore the hypercorrelations between the query
+image with the foreground and background of the support image, based on which
+foreground and background prediction maps are generated and supervised,
+respectively, to enhance the segmentation result. In addition, we propose a
+Test-time Self-Finetuning (TSF) strategy to more accurately self-tune the query
+prediction in unseen domains. Extensive experiments on four popular datasets
+show that DMTNet achieves superior performance over state-of-the-art
+approaches. Code is available at https://github.com/ChenJiayi68/DMTNet.",cs.CV,['cs.CV']
+CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation,Bo-Yuan Sun · Yuqi Yang · Le Zhang · Ming-Ming Cheng · Qibin Hou,https://github.com/BBBBchan/CorrMatch,https://arxiv.org/abs/2306.04300v3,,2306.04300v3.pdf,CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation,"This paper presents a simple but performant semi-supervised semantic
+segmentation approach, called CorrMatch. Previous approaches mostly employ
+complicated training strategies to leverage unlabeled data but overlook the
+role of correlation maps in modeling the relationships between pairs of
+locations. We observe that the correlation maps not only enable clustering
+pixels of the same category easily but also contain good shape information,
+which previous works have omitted. Motivated by these, we aim to improve the
+use efficiency of unlabeled data by designing two novel label propagation
+strategies. First, we propose to conduct pixel propagation by modeling the
+pairwise similarities of pixels to spread the high-confidence pixels and dig
+out more. Then, we perform region propagation to enhance the pseudo labels with
+accurate class-agnostic masks extracted from the correlation maps. CorrMatch
+achieves great performance on popular segmentation benchmarks. Taking the
+DeepLabV3+ with ResNet-101 backbone as our segmentation model, we receive a
+76%+ mIoU score on the Pascal VOC 2012 dataset with only 92 annotated images.
+Code is available at https://github.com/BBBBchan/CorrMatch.",cs.CV,['cs.CV']
+PixelLM: Pixel Reasoning with Large Multimodal Model,Zhongwei Ren · Zhicheng Huang · Yunchao Wei · Yao Zhao · Dongmei Fu · Jiashi Feng · Xiaojie Jin, ,https://arxiv.org/abs/2312.02228,,2312.02228.pdf,PixelLM: Pixel Reasoning with Large Multimodal Model,"While large multimodal models (LMMs) have achieved remarkable progress,
+generating pixel-level masks for image reasoning tasks involving multiple
+open-world targets remains a challenge. To bridge this gap, we introduce
+PixelLM, an effective and efficient LMM for pixel-level reasoning and
+understanding. Central to PixelLM is a novel, lightweight pixel decoder and a
+comprehensive segmentation codebook. The decoder efficiently produces masks
+from the hidden embeddings of the codebook tokens, which encode detailed
+target-relevant information. With this design, PixelLM harmonizes with the
+structure of popular LMMs and avoids the need for additional costly
+segmentation models. Furthermore, we propose a target refinement loss to
+enhance the model's ability to differentiate between multiple targets, leading
+to substantially improved mask quality. To advance research in this area, we
+construct MUSE, a high-quality multi-target reasoning segmentation benchmark.
+PixelLM excels across various pixel-level image reasoning and understanding
+tasks, outperforming well-established methods in multiple benchmarks, including
+MUSE, single- and multi-referring segmentation. Comprehensive ablations confirm
+the efficacy of each proposed component. All code, models, and datasets will be
+publicly available.",cs.CV,['cs.CV']
+EGTR: Extracting Graph from Transformer for Scene Graph Generation,Jinbae Im · JeongYeon Nam · Nokyung Park · Hyungmin Lee · Seunghyun Park,https://github.com/naver-ai/egtr,https://arxiv.org/abs/2404.02072,,2404.02072.pdf,EGTR: Extracting Graph from Transformer for Scene Graph Generation,"Scene Graph Generation (SGG) is a challenging task of detecting objects and
+predicting relationships between objects. After DETR was developed, one-stage
+SGG models based on a one-stage object detector have been actively studied.
+However, complex modeling is used to predict the relationship between objects,
+and the inherent relationship between object queries learned in the multi-head
+self-attention of the object detector has been neglected. We propose a
+lightweight one-stage SGG model that extracts the relation graph from the
+various relationships learned in the multi-head self-attention layers of the
+DETR decoder. By fully utilizing the self-attention by-products, the relation
+graph can be extracted effectively with a shallow relation extraction head.
+Considering the dependency of the relation extraction task on the object
+detection task, we propose a novel relation smoothing technique that adjusts
+the relation label adaptively according to the quality of the detected objects.
+By the relation smoothing, the model is trained according to the continuous
+curriculum that focuses on object detection task at the beginning of training
+and performs multi-task learning as the object detection performance gradually
+improves. Furthermore, we propose a connectivity prediction task that predicts
+whether a relation exists between object pairs as an auxiliary task of the
+relation extraction. We demonstrate the effectiveness and efficiency of our
+method for the Visual Genome and Open Image V6 datasets. Our code is publicly
+available at https://github.com/naver-ai/egtr.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning,Rongjie Li · Yu Wu · Xuming He, ,https://arxiv.org/abs/2404.00909v1,,2404.00909v1.pdf,Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning,"Generative vision-language models (VLMs) have shown impressive performance in
+zero-shot vision-language tasks like image captioning and visual question
+answering. However, improving their zero-shot reasoning typically requires
+second-stage instruction tuning, which relies heavily on human-labeled or large
+language model-generated annotation, incurring high labeling costs. To tackle
+this challenge, we introduce Image-Conditioned Caption Correction (ICCC), a
+novel pre-training task designed to enhance VLMs' zero-shot performance without
+the need for labeled task-aware data. The ICCC task compels VLMs to rectify
+mismatches between visual and language concepts, thereby enhancing instruction
+following and text generation conditioned on visual inputs. Leveraging language
+structure and a lightweight dependency parser, we construct data samples of
+ICCC task from image-text datasets with low labeling and computation costs.
+Experimental results on BLIP-2 and InstructBLIP demonstrate significant
+improvements in zero-shot image-text generation-based VL tasks through ICCC
+instruction tuning.",cs.CV,['cs.CV']
+GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs,Mustafa Munir · William Avery · Md Mostafijur Rahman · Radu Marculescu, ,https://arxiv.org/abs/2405.06849,,2405.06849.pdf,GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs,"Vision graph neural networks (ViG) offer a new avenue for exploration in
+computer vision. A major bottleneck in ViGs is the inefficient k-nearest
+neighbor (KNN) operation used for graph construction. To solve this issue, we
+propose a new method for designing ViGs, Dynamic Axial Graph Construction
+(DAGC), which is more efficient than KNN as it limits the number of considered
+graph connections made within an image. Additionally, we propose a novel
+CNN-GNN architecture, GreedyViG, which uses DAGC. Extensive experiments show
+that GreedyViG beats existing ViG, CNN, and ViT architectures in terms of
+accuracy, GMACs, and parameters on image classification, object detection,
+instance segmentation, and semantic segmentation tasks. Our smallest model,
+GreedyViG-S, achieves 81.1% top-1 accuracy on ImageNet-1K, 2.9% higher than
+Vision GNN and 2.2% higher than Vision HyperGraph Neural Network (ViHGNN), with
+less GMACs and a similar number of parameters. Our largest model, GreedyViG-B
+obtains 83.9% top-1 accuracy, 0.2% higher than Vision GNN, with a 66.6%
+decrease in parameters and a 69% decrease in GMACs. GreedyViG-B also obtains
+the same accuracy as ViHGNN with a 67.3% decrease in parameters and a 71.3%
+decrease in GMACs. Our work shows that hybrid CNN-GNN architectures not only
+provide a new avenue for designing efficient models, but that they can also
+exceed the performance of current state-of-the-art models.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods,Mingqi Jiang · Saeed Khorram · Li Fuxin,https://mingqij.github.io/projects/cdmmtc,,https://www.nature.com/articles/s41598-024-59384-x,,,,,nan
+LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation,Kibum Kim · Kanghoon Yoon · Jaehyeong Jeon · Yeonjun In · Jinyoung Moon · Donghyun Kim · Chanyoung Park, ,https://arxiv.org/abs/2310.10404,,2310.10404.pdf,LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation,"Weakly-Supervised Scene Graph Generation (WSSGG) research has recently
+emerged as an alternative to the fully-supervised approach that heavily relies
+on costly annotations. In this regard, studies on WSSGG have utilized image
+captions to obtain unlocalized triplets while primarily focusing on grounding
+the unlocalized triplets over image regions. However, they have overlooked the
+two issues involved in the triplet formation process from the captions: 1)
+Semantic over-simplification issue arises when extracting triplets from
+captions, where fine-grained predicates in captions are undesirably converted
+into coarse-grained predicates, resulting in a long-tailed predicate
+distribution, and 2) Low-density scene graph issue arises when aligning the
+triplets in the caption with entity/predicate classes of interest, where many
+triplets are discarded and not used in training, leading to insufficient
+supervision. To tackle the two issues, we propose a new approach, i.e., Large
+Language Model for weakly-supervised SGG (LLM4SGG), where we mitigate the two
+issues by leveraging the LLM's in-depth understanding of language and reasoning
+ability during the extraction of triplets from captions and alignment of
+entity/predicate classes with target data. To further engage the LLM in these
+processes, we adopt the idea of Chain-of-Thought and the in-context few-shot
+learning strategy. To validate the effectiveness of LLM4SGG, we conduct
+extensive experiments on Visual Genome and GQA datasets, showing significant
+improvements in both Recall@K and mean Recall@K compared to the
+state-of-the-art WSSGG methods. A further appeal is that LLM4SGG is
+data-efficient, enabling effective model training with a small amount of
+training images.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+DUSt3R: Geometric 3D Vision Made Easy,Shuzhe Wang · Vincent Leroy · Yohann Cabon · Boris Chidlovskii · Jerome Revaud, ,https://arxiv.org/abs/2312.14132v1,,2312.14132v1.pdf,DUSt3R: Geometric 3D Vision Made Easy,"Multi-view stereo reconstruction (MVS) in the wild requires to first estimate
+the camera parameters e.g. intrinsic and extrinsic parameters. These are
+usually tedious and cumbersome to obtain, yet they are mandatory to triangulate
+corresponding pixels in 3D space, which is the core of all best performing MVS
+algorithms. In this work, we take an opposite stance and introduce DUSt3R, a
+radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction
+of arbitrary image collections, i.e. operating without prior information about
+camera calibration nor viewpoint poses. We cast the pairwise reconstruction
+problem as a regression of pointmaps, relaxing the hard constraints of usual
+projective camera models. We show that this formulation smoothly unifies the
+monocular and binocular reconstruction cases. In the case where more than two
+images are provided, we further propose a simple yet effective global alignment
+strategy that expresses all pairwise pointmaps in a common reference frame. We
+base our network architecture on standard Transformer encoders and decoders,
+allowing us to leverage powerful pretrained models. Our formulation directly
+provides a 3D model of the scene as well as depth information, but
+interestingly, we can seamlessly recover from it, pixel matches, relative and
+absolute camera. Exhaustive experiments on all these tasks showcase that the
+proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on
+monocular/multi-view depth estimation as well as relative pose estimation. In
+summary, DUSt3R makes many geometric 3D vision tasks easy.",cs.CV,['cs.CV']
+Latent Modulated Function for Computational Optimal Continuous Image Representation,Zongyao He · Zhi Jin,https://github.com/HeZongyao/LMF,https://arxiv.org/abs/2404.16451,,2404.16451.pdf,Latent Modulated Function for Computational Optimal Continuous Image Representation,"The recent work Local Implicit Image Function (LIIF) and subsequent Implicit
+Neural Representation (INR) based works have achieved remarkable success in
+Arbitrary-Scale Super-Resolution (ASSR) by using MLP to decode Low-Resolution
+(LR) features. However, these continuous image representations typically
+implement decoding in High-Resolution (HR) High-Dimensional (HD) space, leading
+to a quadratic increase in computational cost and seriously hindering the
+practical applications of ASSR. To tackle this problem, we propose a novel
+Latent Modulated Function (LMF), which decouples the HR-HD decoding process
+into shared latent decoding in LR-HD space and independent rendering in HR
+Low-Dimensional (LD) space, thereby realizing the first computational optimal
+paradigm of continuous image representation. Specifically, LMF utilizes an HD
+MLP in latent space to generate latent modulations of each LR feature vector.
+This enables a modulated LD MLP in render space to quickly adapt to any input
+feature vector and perform rendering at arbitrary resolution. Furthermore, we
+leverage the positive correlation between modulation intensity and input image
+complexity to design a Controllable Multi-Scale Rendering (CMSR) algorithm,
+offering the flexibility to adjust the decoding efficiency based on the
+rendering precision. Extensive experiments demonstrate that converting existing
+INR-based ASSR methods to LMF can reduce the computational cost by up to 99.9%,
+accelerate inference by up to 57 times, and save up to 76% of parameters, while
+maintaining competitive performance. The code is available at
+https://github.com/HeZongyao/LMF.",cs.CV,"['cs.CV', 'cs.AI']"
+Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding,Zhihao Yuan · Jinke Ren · Chun-Mei Feng · Hengshuang Zhao · Shuguang Cui · Zhen Li,https://curryyuan.github.io/ZSVG3D/,https://arxiv.org/abs/2311.15383,,,Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding,"3D Visual Grounding (3DVG) aims at localizing 3D object based on textual
+descriptions. Conventional supervised methods for 3DVG often necessitate
+extensive annotations and a predefined vocabulary, which can be restrictive. To
+address this issue, we propose a novel visual programming approach for
+zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language
+models (LLMs). Our approach begins with a unique dialog-based method, engaging
+with LLMs to establish a foundational understanding of zero-shot 3DVG. Building
+on this, we design a visual program that consists of three types of modules,
+i.e., view-independent, view-dependent, and functional modules. These modules,
+specifically tailored for 3D scenarios, work collaboratively to perform complex
+reasoning and inference. Furthermore, we develop an innovative language-object
+correlation module to extend the scope of existing 3D object detectors into
+open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot
+approach can outperform some supervised baselines, marking a significant stride
+towards effective 3DVG.",cs.CV,['cs.CV']
+NICE: Neurogenesis Inspired Contextual Encoding for Replay-free Class Incremental Learning,Mustafa B Gurbuz · Jean Moorman · Constantine Dovrolis,https://github.com/BurakGurbuz97/NICE,https://arxiv.org/abs/2310.03898,,2310.03898.pdf,Class-Incremental Learning Using Generative Experience Replay Based on Time-aware Regularization,"Learning new tasks accumulatively without forgetting remains a critical
+challenge in continual learning. Generative experience replay addresses this
+challenge by synthesizing pseudo-data points for past learned tasks and later
+replaying them for concurrent training along with the new tasks' data.
+Generative replay is the best strategy for continual learning under a strict
+class-incremental setting when certain constraints need to be met: (i) constant
+model size, (ii) no pre-training dataset, and (iii) no memory buffer for
+storing past tasks' data. Inspired by the biological nervous system mechanisms,
+we introduce a time-aware regularization method to dynamically fine-tune the
+three training objective terms used for generative replay: supervised learning,
+latent regularization, and data reconstruction. Experimental results on major
+benchmarks indicate that our method pushes the limit of brain-inspired
+continual learners under such strict settings, improves memory retention, and
+increases the average performance over continually arriving tasks.",cs.LG,['cs.LG']
+A Simple Recipe for Language-guided Domain Generalized Segmentation,Mohammad Fahes · TUAN-HUNG VU · Andrei Bursuc · Patrick Pérez · Raoul de Charette,https://astra-vision.github.io/FAMix/,https://arxiv.org/abs/2311.17922,,2311.17922.pdf,A Simple Recipe for Language-guided Domain Generalized Segmentation,"Generalization to new domains not seen during training is one of the
+long-standing challenges in deploying neural networks in real-world
+applications. Existing generalization techniques either necessitate external
+images for augmentation, and/or aim at learning invariant representations by
+imposing various alignment constraints. Large-scale pretraining has recently
+shown promising generalization capabilities, along with the potential of
+binding different modalities. For instance, the advent of vision-language
+models like CLIP has opened the doorway for vision models to exploit the
+textual modality. In this paper, we introduce a simple framework for
+generalizing semantic segmentation networks by employing language as the source
+of randomization. Our recipe comprises three key ingredients: (i) the
+preservation of the intrinsic CLIP robustness through minimal fine-tuning, (ii)
+language-driven local style augmentation, and (iii) randomization by locally
+mixing the source and augmented styles during training. Extensive experiments
+report state-of-the-art results on various generalization benchmarks. Code is
+accessible at https://github.com/astra-vision/FAMix .",cs.CV,['cs.CV']
+Self-Calibrating Vicinal Risk Minimisation for Model Calibration,Jiawei Liu · Changkun Ye · Ruikai Cui · Nick Barnes, ,https://arxiv.org/abs/2307.13539,,2307.13539.pdf,Model Calibration in Dense Classification with Adaptive Label Perturbation,"For safety-related applications, it is crucial to produce trustworthy deep
+neural networks whose prediction is associated with confidence that can
+represent the likelihood of correctness for subsequent decision-making.
+Existing dense binary classification models are prone to being over-confident.
+To improve model calibration, we propose Adaptive Stochastic Label Perturbation
+(ASLP) which learns a unique label perturbation level for each training image.
+ASLP employs our proposed Self-Calibrating Binary Cross Entropy (SC-BCE) loss,
+which unifies label perturbation processes including stochastic approaches
+(like DisturbLabel), and label smoothing, to correct calibration while
+maintaining classification rates. ASLP follows Maximum Entropy Inference of
+classic statistical mechanics to maximise prediction entropy with respect to
+missing information. It performs this while: (1) preserving classification
+accuracy on known data as a conservative solution, or (2) specifically improves
+model calibration degree by minimising the gap between the prediction accuracy
+and expected confidence of the target training label. Extensive results
+demonstrate that ASLP can significantly improve calibration degrees of dense
+binary classification models on both in-distribution and out-of-distribution
+data. The code is available on https://github.com/Carlisle-Liu/ASLP.",cs.CV,"['cs.CV', 'cs.LG']"
+Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation,Junyan Wang · Zhenhong Sun · Stewart Tan · Xuanbai Chen · Weihua Chen · li · Cheng Zhang · Yang Song, ,https://arxiv.org/abs/2403.05239,,2403.05239.pdf,Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation,"Vanilla text-to-image diffusion models struggle with generating accurate
+human images, commonly resulting in imperfect anatomies such as unnatural
+postures or disproportionate limbs.Existing methods address this issue mostly
+by fine-tuning the model with extra images or adding additional controls --
+human-centric priors such as pose or depth maps -- during the image generation
+phase. This paper explores the integration of these human-centric priors
+directly into the model fine-tuning stage, essentially eliminating the need for
+extra conditions at the inference stage. We realize this idea by proposing a
+human-centric alignment loss to strengthen human-related information from the
+textual prompts within the cross-attention maps. To ensure semantic detail
+richness and human structural accuracy during fine-tuning, we introduce
+scale-aware and step-wise constraints within the diffusion process, according
+to an in-depth analysis of the cross-attention layer. Extensive experiments
+show that our method largely improves over state-of-the-art text-to-image
+models to synthesize high-quality human images based on user-written prompts.
+Project page: \url{https://hcplayercvpr2024.github.io}.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability,Jaehui Hwang · Junghyuk Lee · Jong-Seok Lee, ,https://arxiv.org/abs/2312.10634,,2312.10634.pdf,Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability,"With the advancement of generative models, the assessment of generated images
+becomes more and more important. Previous methods measure distances between
+features of reference and generated images from trained vision models. In this
+paper, we conduct an extensive investigation into the relationship between the
+representation space and input space around generated images. We first propose
+two measures related to the presence of unnatural elements within images:
+complexity, which indicates how non-linear the representation space is, and
+vulnerability, which is related to how easily the extracted feature changes by
+adversarial input changes. Based on these, we introduce a new metric to
+evaluating image-generative models called anomaly score (AS). Moreover, we
+propose AS-i (anomaly score for individual images) that can effectively
+evaluate generated images individually. Experimental results demonstrate the
+validity of the proposed approach.",cs.CV,"['cs.CV', 'cs.LG']"
+MuseChat: A Conversational Music Recommendation System for Videos,Zhikang Dong · Bin Chen · Xiulong Liu · Pawel Polak · Peng Zhang, ,https://arxiv.org/abs/2310.06282,,2310.06282.pdf,MuseChat: A Conversational Music Recommendation System for Videos,"Music recommendation for videos attracts growing interest in multi-modal
+research. However, existing systems focus primarily on content compatibility,
+often ignoring the users' preferences. Their inability to interact with users
+for further refinements or to provide explanations leads to a less satisfying
+experience. We address these issues with MuseChat, a first-of-its-kind
+dialogue-based recommendation system that personalizes music suggestions for
+videos. Our system consists of two key functionalities with associated modules:
+recommendation and reasoning. The recommendation module takes a video along
+with optional information including previous suggested music and user's
+preference as inputs and retrieves an appropriate music matching the context.
+The reasoning module, equipped with the power of Large Language Model
+(Vicuna-7B) and extended to multi-modal inputs, is able to provide reasonable
+explanation for the recommended music. To evaluate the effectiveness of
+MuseChat, we build a large-scale dataset, conversational music recommendation
+for videos, that simulates a two-turn interaction between a user and a
+recommender based on accurate music track information. Experiment results show
+that MuseChat achieves significant improvements over existing video-based music
+retrieval methods as well as offers strong interpretability and
+interactability.",cs.LG,"['cs.LG', 'cs.CV', 'cs.IR']"
+Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration,Lianxin Xie · csbingbing zheng · Wen Xue · Le Jiang · Cheng Liu · Si Wu · Hau San Wong, ,https://arxiv.org/abs/2402.06106,,2402.06106.pdf,CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models,"Recent generative-prior-based methods have shown promising blind face
+restoration performance. They usually project the degraded images to the latent
+space and then decode high-quality faces either by single-stage latent
+optimization or directly from the encoding. Generating fine-grained facial
+details faithful to inputs remains a challenging problem. Most existing methods
+produce either overly smooth outputs or alter the identity as they attempt to
+balance between generation and reconstruction. This may be attributed to the
+typical trade-off between quality and resolution in the latent space. If the
+latent space is highly compressed, the decoded output is more robust to
+degradations but shows worse fidelity. On the other hand, a more flexible
+latent space can capture intricate facial details better, but is extremely
+difficult to optimize for highly degraded faces using existing techniques. To
+address these issues, we introduce a diffusion-based-prior inside a VQGAN
+architecture that focuses on learning the distribution over uncorrupted latent
+embeddings. With such knowledge, we iteratively recover the clean embedding
+conditioning on the degraded counterpart. Furthermore, to ensure the reverse
+diffusion trajectory does not deviate from the underlying identity, we train a
+separate Identity Recovery Network and use its output to constrain the reverse
+diffusion process. Specifically, using a learnable latent mask, we add
+gradients from a face-recognition network to a subset of latent features that
+correlates with the finer identity-related details in the pixel space, leaving
+the other features untouched. Disentanglement between perception and fidelity
+in the latent space allows us to achieve the best of both worlds. We perform
+extensive evaluations on multiple real and synthetic datasets to validate the
+superiority of our approach.",cs.CV,['cs.CV']
+Faces that Speak: Jointly Synthesising Talking Face and Speech from Text,Youngjoon Jang · Jihoon Kim · Junseok Ahn · Doyeop Kwak · Hongsun Yang · Yooncheol Ju · ILHWAN KIM · Byeong-Yeol Kim · Joon Chung,https://mm.kaist.ac.kr/projects/faces-that-speak/,https://arxiv.org/abs/2405.10272,,2405.10272.pdf,Faces that Speak: Jointly Synthesising Talking Face and Speech from Text,"The goal of this work is to simultaneously generate natural talking faces and
+speech outputs from text. We achieve this by integrating Talking Face
+Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We
+address the main challenges of each task: (1) generating a range of head poses
+representative of real-world scenarios, and (2) ensuring voice consistency
+despite variations in facial motion for the same identity. To tackle these
+issues, we introduce a motion sampler based on conditional flow matching, which
+is capable of high-quality motion code generation in an efficient way.
+Moreover, we introduce a novel conditioning method for the TTS system, which
+utilises motion-removed features from the TFG model to yield uniform speech
+outputs. Our extensive experiments demonstrate that our method effectively
+creates natural-looking talking faces and speech that accurately match the
+input text. To our knowledge, this is the first effort to build a multimodal
+synthesis system that can generalise to unseen identities.",cs.CV,"['cs.CV', 'cs.AI', 'cs.SD', 'eess.AS', 'eess.IV']"
+Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance,Yuto Enyo · Ko Nishino, ,https://arxiv.org/abs/2312.04529,,2312.04529.pdf,Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance,"Reflectance bounds the frequency spectrum of illumination in the object
+appearance. In this paper, we introduce the first stochastic inverse rendering
+method, which recovers the attenuated frequency spectrum of an illumination
+jointly with the reflectance of an object of known geometry from a single
+image. Our key idea is to solve this blind inverse problem in the reflectance
+map, an appearance representation invariant to the underlying geometry, by
+learning to reverse the image formation with a novel diffusion model which we
+refer to as the Diffusion Reflectance Map Network (DRMNet). Given an observed
+reflectance map converted and completed from the single input image, DRMNet
+generates a reflectance map corresponding to a perfect mirror sphere while
+jointly estimating the reflectance. The forward process can be understood as
+gradually filtering a natural illumination with lower and lower frequency
+reflectance and additive Gaussian noise. DRMNet learns to invert this process
+with two subnetworks, IllNet and RefNet, which work in concert towards this
+joint estimation. The network is trained on an extensive synthetic dataset and
+is demonstrated to generalize to real images, showing state-of-the-art accuracy
+on established datasets.",cs.CV,['cs.CV']
+PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild,Kun Yuan · Hongbo Liu · Mading Li · Muyi Sun · Ming Sun · Jiachao Gong · Jinhua Hao · Chao Zhou · Yansong Tang, ,https://arxiv.org/abs/2405.17765,,2405.17765.pdf,PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild,"Video quality assessment (VQA) is a challenging problem due to the numerous
+factors that can affect the perceptual quality of a video, \eg, content
+attractiveness, distortion type, motion pattern, and level. However, annotating
+the Mean opinion score (MOS) for videos is expensive and time-consuming, which
+limits the scale of VQA datasets, and poses a significant obstacle for deep
+learning-based methods. In this paper, we propose a VQA method named PTM-VQA,
+which leverages PreTrained Models to transfer knowledge from models pretrained
+on various pre-tasks, enabling benefits for VQA from different aspects.
+  Specifically, we extract features of videos from different pretrained models
+with frozen weights and integrate them to generate representation. Since these
+models possess various fields of knowledge and are often trained with labels
+irrelevant to quality, we propose an Intra-Consistency and Inter-Divisibility
+(ICID) loss to impose constraints on features extracted by multiple pretrained
+models. The intra-consistency constraint ensures that features extracted by
+different pretrained models are in the same unified quality-aware latent space,
+while the inter-divisibility introduces pseudo clusters based on the annotation
+of samples and tries to separate features of samples from different clusters.
+Furthermore, with a constantly growing number of pretrained models, it is
+crucial to determine which models to use and how to use them. To address this
+problem, we propose an efficient scheme to select suitable candidates. Models
+with better clustering performance on VQA datasets are chosen to be our
+candidates. Extensive experiments demonstrate the effectiveness of the proposed
+method.",cs.CV,['cs.CV']
+Plug-and-Play Diffusion Distillation,Yi-Ting Hsiao · Siavash Khodadadeh · Kevin Duarte · Wei-An Lin · Hui Qu · Mingi Kwon · Ratheesh Kalarot,https://5410tiffany.github.io/plug-and-play-diffusion-distillation.github.io/,https://arxiv.org/abs/2403.12015,,2403.12015.pdf,Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,"Diffusion models are the main driver of progress in image and video
+synthesis, but suffer from slow inference speed. Distillation methods, like the
+recently introduced adversarial diffusion distillation (ADD) aim to shift the
+model from many-shot to single-step inference, albeit at the cost of expensive
+and difficult optimization due to its reliance on a fixed pretrained DINOv2
+discriminator. We introduce Latent Adversarial Diffusion Distillation (LADD), a
+novel distillation approach overcoming the limitations of ADD. In contrast to
+pixel-based ADD, LADD utilizes generative features from pretrained latent
+diffusion models. This approach simplifies training and enhances performance,
+enabling high-resolution multi-aspect ratio image synthesis. We apply LADD to
+Stable Diffusion 3 (8B) to obtain SD3-Turbo, a fast model that matches the
+performance of state-of-the-art text-to-image generators using only four
+unguided sampling steps. Moreover, we systematically investigate its scaling
+behavior and demonstrate LADD's effectiveness in various applications such as
+image editing and inpainting.",cs.CV,['cs.CV']
+Masked and Shuffled Blind Spot Denoising for Real-World Images,Hamadi Chihaoui · Paolo Favaro, ,https://arxiv.org/abs/2404.09389,,2404.09389.pdf,Masked and Shuffled Blind Spot Denoising for Real-World Images,"We introduce a novel approach to single image denoising based on the Blind
+Spot Denoising principle, which we call MAsked and SHuffled Blind Spot
+Denoising (MASH). We focus on the case of correlated noise, which often plagues
+real images. MASH is the result of a careful analysis to determine the
+relationships between the level of blindness (masking) of the input and the
+(unknown) noise correlation. Moreover, we introduce a shuffling technique to
+weaken the local correlation of noise, which in turn yields an additional
+denoising performance improvement. We evaluate MASH via extensive experiments
+on real-world noisy image datasets. We demonstrate on par or better results
+compared to existing self-supervised denoising methods.",cs.CV,"['cs.CV', 'cs.LG']"
+Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation,Li Hu, ,https://arxiv.org/abs/2311.17117,,2311.17117.pdf,Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation,"Character Animation aims to generating character videos from still images
+through driving signals. Currently, diffusion models have become the mainstream
+in visual generation research, owing to their robust generative capabilities.
+However, challenges persist in the realm of image-to-video, especially in
+character animation, where temporally maintaining consistency with detailed
+information from character remains a formidable problem. In this paper, we
+leverage the power of diffusion models and propose a novel framework tailored
+for character animation. To preserve consistency of intricate appearance
+features from reference image, we design ReferenceNet to merge detail features
+via spatial attention. To ensure controllability and continuity, we introduce
+an efficient pose guider to direct character's movements and employ an
+effective temporal modeling approach to ensure smooth inter-frame transitions
+between video frames. By expanding the training data, our approach can animate
+arbitrary characters, yielding superior results in character animation compared
+to other image-to-video methods. Furthermore, we evaluate our method on
+benchmarks for fashion video and human dance synthesis, achieving
+state-of-the-art results.",cs.CV,['cs.CV']
+Bootstrapping SparseFormers from Vision Foundation Models,Ziteng Gao · Zhan Tong · Kevin Qinghong Lin · Joya Chen · Mike Zheng Shou,https://github.com/showlab/sparseformer,https://arxiv.org/abs/2312.01987,,2312.01987.pdf,Bootstrapping SparseFormers from Vision Foundation Models,"The recently proposed SparseFormer architecture provides an alternative
+approach to visual understanding by utilizing a significantly lower number of
+visual tokens via adjusting RoIs, greatly reducing computational costs while
+still achieving promising performance. However, training SparseFormers from
+scratch is still expensive, and scaling up the number of parameters can be
+challenging. In this paper, we propose to bootstrap SparseFormers from
+ViT-based vision foundation models in a simple and efficient way. Since the
+majority of SparseFormer blocks are the standard transformer ones, we can
+inherit weights from large-scale pre-trained vision transformers and freeze
+them as much as possible. Therefore, we only need to train the
+SparseFormer-specific lightweight focusing transformer to adjust token RoIs and
+fine-tune a few early pre-trained blocks to align the final token
+representation. In such a way, we can bootstrap SparseFormer architectures from
+various large-scale pre-trained models (e.g., IN-21K pre-trained AugRegs or
+CLIPs) using a rather smaller amount of training samples (e.g., IN-1K) and
+without labels or captions within just a few hours. As a result, the
+bootstrapped unimodal SparseFormer (from AugReg-ViT-L/16-384) can reach 84.9%
+accuracy on IN-1K with only 49 tokens, and the multimodal SparseFormer from
+CLIPs also demonstrates notable zero-shot performance with highly reduced
+computational cost without seeing any caption during the bootstrapping
+procedure. In addition, CLIP-bootstrapped SparseFormers, which align the output
+space with language without seeing a word, can serve as efficient vision
+encoders in multimodal large language models. Code and models are available at
+https://github.com/showlab/sparseformer",cs.CV,['cs.CV']
+Self-Supervised Dual Contouring,Ramana Sundararaman · Roman Klokov · Maks Ovsjanikov, ,https://arxiv.org/abs/2405.18131,,2405.18131.pdf,Self-Supervised Dual Contouring,"Learning-based isosurface extraction methods have recently emerged as a
+robust and efficient alternative to axiomatic techniques. However, the vast
+majority of such approaches rely on supervised training with axiomatically
+computed ground truths, thus potentially inheriting biases and data artifacts
+of the corresponding axiomatic methods. Steering away from such dependencies,
+we propose a self-supervised training scheme for the Neural Dual Contouring
+meshing framework, resulting in our method: Self-Supervised Dual Contouring
+(SDC). Instead of optimizing predicted mesh vertices with supervised training,
+we use two novel self-supervised loss functions that encourage the consistency
+between distances to the generated mesh up to the first order. Meshes
+reconstructed by SDC surpass existing data-driven methods in capturing
+intricate details while being more robust to possible irregularities in the
+input. Furthermore, we use the same self-supervised training objective linking
+inferred mesh and input SDF, to regularize the training process of Deep
+Implicit Networks (DINs). We demonstrate that the resulting DINs produce
+higher-quality implicit functions, ultimately leading to more accurate and
+detail-preserving surfaces compared to prior baselines for different input
+modalities. Finally, we demonstrate that our self-supervised losses improve
+meshing performance in the single-view reconstruction task by enabling joint
+training of predicted SDF and resulting output mesh. We open-source our code at
+https://github.com/Sentient07/SDC",cs.CV,['cs.CV']
+Wired Perspectives: Multi-View Wire Art Embraces Generative AI,Zhiyu Qu · LAN YANG · Honggang Zhang · Tao Xiang · Kaiyue Pang · Yi-Zhe Song,https://dreamwireart.github.io/,https://arxiv.org/abs/2311.15421,,,Wired Perspectives: Multi-View Wire Art Embraces Generative AI,"Creating multi-view wire art (MVWA), a static 3D sculpture with diverse
+interpretations from different viewpoints, is a complex task even for skilled
+artists. In response, we present DreamWire, an AI system enabling everyone to
+craft MVWA easily. Users express their vision through text prompts or
+scribbles, freeing them from intricate 3D wire organisation. Our approach
+synergises 3D B\'ezier curves, Prim's algorithm, and knowledge distillation
+from diffusion models or their variants (e.g., ControlNet). This blend enables
+the system to represent 3D wire art, ensuring spatial continuity and overcoming
+data scarcity. Extensive evaluation and analysis are conducted to shed insight
+on the inner workings of the proposed system, including the trade-off between
+connectivity and visual aesthetics.",cs.CV,"['cs.CV', 'cs.AI']"
+SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation,Yuxuan Zhang · Yiren Song · Jiaming Liu · Rui Wang · Jinpeng Yu · Hao Tang · Huaxia Li · Xu Tang · Yao Hu · Han Pan · Zhongliang Jing,https://ssr-encoder.github.io/,https://arxiv.org/abs/2312.16272,,2312.16272.pdf,SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation,"Recent advancements in subject-driven image generation have led to zero-shot
+generation, yet precise selection and focus on crucial subject representations
+remain challenging. Addressing this, we introduce the SSR-Encoder, a novel
+architecture designed for selectively capturing any subject from single or
+multiple reference images. It responds to various query modalities including
+text and masks, without necessitating test-time fine-tuning. The SSR-Encoder
+combines a Token-to-Patch Aligner that aligns query inputs with image patches
+and a Detail-Preserving Subject Encoder for extracting and preserving fine
+features of the subjects, thereby generating subject embeddings. These
+embeddings, used in conjunction with original text embeddings, condition the
+generation process. Characterized by its model generalizability and efficiency,
+the SSR-Encoder adapts to a range of custom models and control modules.
+Enhanced by the Embedding Consistency Regularization Loss for improved
+training, our extensive experiments demonstrate its effectiveness in versatile
+and high-quality image generation, indicating its broad applicability. Project
+page: https://ssr-encoder.github.io",cs.CV,['cs.CV']
+Dual Pose-invariant Embeddings: Learning Category and Object-specific Discriminative Representations for Recognition and Retrieval,Rohan Sarkar · Avinash Kak, ,https://arxiv.org/abs/2403.00272,,2403.00272.pdf,Dual Pose-invariant Embeddings: Learning Category and Object-specific Discriminative Representations for Recognition and Retrieval,"In the context of pose-invariant object recognition and retrieval, we
+demonstrate that it is possible to achieve significant improvements in
+performance if both the category-based and the object-identity-based embeddings
+are learned simultaneously during training. In hindsight, that sounds intuitive
+because learning about the categories is more fundamental than learning about
+the individual objects that correspond to those categories. However, to the
+best of what we know, no prior work in pose-invariant learning has demonstrated
+this effect. This paper presents an attention-based dual-encoder architecture
+with specially designed loss functions that optimize the inter- and intra-class
+distances simultaneously in two different embedding spaces, one for the
+category embeddings and the other for the object-level embeddings. The loss
+functions we have proposed are pose-invariant ranking losses that are designed
+to minimize the intra-class distances and maximize the inter-class distances in
+the dual representation spaces. We demonstrate the power of our approach with
+three challenging multi-view datasets, ModelNet-40, ObjectPI, and FG3D. With
+our dual approach, for single-view object recognition, we outperform the
+previous best by 20.0% on ModelNet40, 2.0% on ObjectPI, and 46.5% on FG3D. On
+the other hand, for single-view object retrieval, we outperform the previous
+best by 33.7% on ModelNet40, 18.8% on ObjectPI, and 56.9% on FG3D.",cs.CV,"['cs.CV', 'cs.IR', 'cs.LG']"
+Symphonize 3D Semantic Scene Completion with Contextual Instance Queries,Haoyi Jiang · Tianheng Cheng · Naiyu Gao · Haoyang Zhang · Tianwei Lin · Wenyu Liu · Xinggang Wang, ,https://arxiv.org/abs/2306.15670v2,,2306.15670v2.pdf,Symphonize 3D Semantic Scene Completion with Contextual Instance Queries,"`3D Semantic Scene Completion (SSC) has emerged as a nascent and pivotal
+undertaking in autonomous driving, aiming to predict voxel occupancy within
+volumetric scenes. However, prevailing methodologies primarily focus on
+voxel-wise feature aggregation, while neglecting instance semantics and scene
+context. In this paper, we present a novel paradigm termed Symphonies
+(Scene-from-Insts), that delves into the integration of instance queries to
+orchestrate 2D-to-3D reconstruction and 3D scene modeling. Leveraging our
+proposed Serial Instance-Propagated Attentions, Symphonies dynamically encodes
+instance-centric semantics, facilitating intricate interactions between
+image-based and volumetric domains. Simultaneously, Symphonies enables holistic
+scene comprehension by capturing context through the efficient fusion of
+instance queries, alleviating geometric ambiguity such as occlusion and
+perspective errors through contextual scene reasoning. Experimental results
+demonstrate that Symphonies achieves state-of-the-art performance on
+challenging benchmarks SemanticKITTI and SSCBench-KITTI-360, yielding
+remarkable mIoU scores of 15.04 and 18.58, respectively. These results showcase
+the paradigm's promising advancements. The code is available at
+https://github.com/hustvl/Symphonies.",cs.CV,"['cs.CV', 'cs.RO']"
+KeyPoint Relative Position Encoding for Face Recognition,Minchul Kim · Feng Liu · Yiyang Su · Anil Jain · Xiaoming Liu, ,https://arxiv.org/abs/2403.14852,,2403.14852.pdf,KeyPoint Relative Position Encoding for Face Recognition,"In this paper, we address the challenge of making ViT models more robust to
+unseen affine transformations. Such robustness becomes useful in various
+recognition tasks such as face recognition when image alignment failures occur.
+We propose a novel method called KP-RPE, which leverages key points
+(e.g.~facial landmarks) to make ViT more resilient to scale, translation, and
+pose variations. We begin with the observation that Relative Position Encoding
+(RPE) is a good way to bring affine transform generalization to ViTs. RPE,
+however, can only inject the model with prior knowledge that nearby pixels are
+more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this
+principle, where the significance of pixels is not solely dictated by their
+proximity but also by their relative positions to specific keypoints within the
+image. By anchoring the significance of pixels around keypoints, the model can
+more effectively retain spatial relationships, even when those relationships
+are disrupted by affine transformations. We show the merit of KP-RPE in face
+and gait recognition. The experimental results demonstrate the effectiveness in
+improving face recognition performance from low-quality images, particularly
+where alignment is prone to failure. Code and pre-trained models are available.",cs.CV,['cs.CV']
+Feedback-Guided Autonomous Driving,Jimuyang Zhang · Zanming Huang · Arijit Ray · Eshed Ohn-Bar, ,https://arxiv.org/abs/2306.10014,,2306.10014.pdf,Coaching a Teachable Student,"We propose a novel knowledge distillation framework for effectively teaching
+a sensorimotor student agent to drive from the supervision of a privileged
+teacher agent. Current distillation for sensorimotor agents methods tend to
+result in suboptimal learned driving behavior by the student, which we
+hypothesize is due to inherent differences between the input, modeling
+capacity, and optimization processes of the two agents. We develop a novel
+distillation scheme that can address these limitations and close the gap
+between the sensorimotor agent and its privileged teacher. Our key insight is
+to design a student which learns to align their input features with the
+teacher's privileged Bird's Eye View (BEV) space. The student then can benefit
+from direct supervision by the teacher over the internal representation
+learning. To scaffold the difficult sensorimotor learning task, the student
+model is optimized via a student-paced coaching mechanism with various
+auxiliary supervision. We further propose a high-capacity imitation learned
+privileged agent that surpasses prior privileged agents in CARLA and ensures
+the student learns safe driving behavior. Our proposed sensorimotor agent
+results in a robust image-based behavior cloning agent in CARLA, improving over
+current models by over 20.6% in driving score without requiring LiDAR,
+historical observations, ensemble of models, on-policy data aggregation or
+reinforcement learning.",cs.CV,"['cs.CV', 'cs.AI', 'cs.RO']"
+Look-Up Table Compression for Efficient Image Restoration,Yinglong Li · Jiacheng Li · Zhiwei Xiong, ,https://arxiv.org/abs/2307.08544,,2307.08544.pdf,Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution,"Look-up table(LUT)-based methods have shown the great efficacy in single
+image super-resolution (SR) task. However, previous methods ignore the
+essential reason of restricted receptive field (RF) size in LUT, which is
+caused by the interaction of space and channel features in vanilla convolution.
+They can only increase the RF at the cost of linearly increasing LUT size. To
+enlarge RF with contained LUT sizes, we propose a novel Reconstructed
+Convolution(RC) module, which decouples channel-wise and spatial calculation.
+It can be formulated as $n^2$ 1D LUTs to maintain $n\times n$ receptive field,
+which is obviously smaller than $n\times n$D LUT formulated before. The LUT
+generated by our RC module reaches less than 1/10000 storage compared with
+SR-LUT baseline. The proposed Reconstructed Convolution module based LUT
+method, termed as RCLUT, can enlarge the RF size by 9 times than the
+state-of-the-art LUT-based SR method and achieve superior performance on five
+popular benchmark dataset. Moreover, the efficient and robust RC module can be
+used as a plugin to improve other LUT-based SR methods. The code is available
+at https://github.com/liuguandu/RC-LUT.",eess.IV,"['eess.IV', 'cs.CV']"
+WaveMo: Learning Wavefront Modulations to See Through Scattering,Mingyang Xie · Haiyun Guo · Brandon Y. Feng · Lingbo Jin · Ashok Veeraraghavan · Christopher Metzler,https://wavemo-2024.github.io/,https://arxiv.org/abs/2404.07985v1,,2404.07985v1.pdf,WaveMo: Learning Wavefront Modulations to See Through Scattering,"Imaging through scattering media is a fundamental and pervasive challenge in
+fields ranging from medical diagnostics to astronomy. A promising strategy to
+overcome this challenge is wavefront modulation, which induces measurement
+diversity during image acquisition. Despite its importance, designing optimal
+wavefront modulations to image through scattering remains under-explored. This
+paper introduces a novel learning-based framework to address the gap. Our
+approach jointly optimizes wavefront modulations and a computationally
+lightweight feedforward ""proxy"" reconstruction network. This network is trained
+to recover scenes obscured by scattering, using measurements that are modified
+by these modulations. The learned modulations produced by our framework
+generalize effectively to unseen scattering scenarios and exhibit remarkable
+versatility. During deployment, the learned modulations can be decoupled from
+the proxy network to augment other more computationally expensive restoration
+algorithms. Through extensive experiments, we demonstrate our approach
+significantly advances the state of the art in imaging through scattering
+media. Our project webpage is at https://wavemo-2024.github.io/.",cs.CV,"['cs.CV', 'eess.IV']"
+Constrained Layout Generation with Factor Graphs,Mohammed Haroon Dupty · Yanfei Dong · Sicong Leng · Guoji Fu · Yong Liang Goh · Wei Lu · Wee Sun Lee, ,https://arxiv.org/abs/2404.00385,,2404.00385.pdf,Constrained Layout Generation with Factor Graphs,"This paper addresses the challenge of object-centric layout generation under
+spatial constraints, seen in multiple domains including floorplan design
+process. The design process typically involves specifying a set of spatial
+constraints that include object attributes like size and inter-object relations
+such as relative positioning. Existing works, which typically represent objects
+as single nodes, lack the granularity to accurately model complex interactions
+between objects. For instance, often only certain parts of an object, like a
+room's right wall, interact with adjacent objects. To address this gap, we
+introduce a factor graph based approach with four latent variable nodes for
+each room, and a factor node for each constraint. The factor nodes represent
+dependencies among the variables to which they are connected, effectively
+capturing constraints that are potentially of a higher order. We then develop
+message-passing on the bipartite graph, forming a factor graph neural network
+that is trained to produce a floorplan that aligns with the desired
+requirements. Our approach is simple and generates layouts faithful to the user
+requirements, demonstrated by a large improvement in IOU scores over existing
+methods. Additionally, our approach, being inferential and accurate, is
+well-suited to the practical human-in-the-loop design process where
+specifications evolve iteratively, offering a practical and powerful tool for
+AI-guided design.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Prompt Learning via Meta-Regularization,Jinyoung Park · Juyeon Ko · Hyunwoo J. Kim, ,https://arxiv.org/abs/2404.00851,,2404.00851.pdf,Prompt Learning via Meta-Regularization,"Pre-trained vision-language models have shown impressive success on various
+computer vision tasks with their zero-shot generalizability. Recently, prompt
+learning approaches have been explored to efficiently and effectively adapt the
+vision-language models to a variety of downstream tasks. However, most existing
+prompt learning methods suffer from task overfitting since the general
+knowledge of the pre-trained vision language models is forgotten while the
+prompts are finetuned on a small data set from a specific target task. To
+address this issue, we propose a Prompt Meta-Regularization (ProMetaR) to
+improve the generalizability of prompt learning for vision-language models.
+Specifically, ProMetaR meta-learns both the regularizer and the soft prompts to
+harness the task-specific knowledge from the downstream tasks and task-agnostic
+general knowledge from the vision-language models. Further, ProMetaR augments
+the task to generate multiple virtual tasks to alleviate the meta-overfitting.
+In addition, we provide the analysis to comprehend how ProMetaR improves the
+generalizability of prompt tuning in the perspective of the gradient alignment.
+Our extensive experiments demonstrate that our ProMetaR improves the
+generalizability of conventional prompt learning methods under
+base-to-base/base-to-new and domain generalization settings. The code of
+ProMetaR is available at https://github.com/mlvlab/ProMetaR.",cs.CV,['cs.CV']
+Bi-level Learning of Task-Specific Decoders for Joint Registration and One-Shot Medical Image Segmentation,Xin Fan · Xiaolin Wang · Jiaxin Gao · Jia Wang · Zhongxuan Luo · Risheng Liu, ,,https://dl.acm.org/doi/10.1145/3580305.3599452,,,,,nan
+NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis,Zinuo You · Andreas Geiger · Anpei Chen,https://sinoyou.github.io/nelf-pro/,https://arxiv.org/abs/2312.13328,,2312.13328.pdf,NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis,"We present NeLF-Pro, a novel representation to model and reconstruct light
+fields in diverse natural scenes that vary in extent and spatial granularity.
+In contrast to previous fast reconstruction methods that represent the 3D scene
+globally, we model the light field of a scene as a set of local light field
+feature probes, parameterized with position and multi-channel 2D feature maps.
+Our central idea is to bake the scene's light field into spatially varying
+learnable representations and to query point features by weighted blending of
+probes close to the camera - allowing for mipmap representation and rendering.
+We introduce a novel vector-matrix-matrix (VMM) factorization technique that
+effectively represents the light field feature probes as products of core
+factors (i.e., VM) shared among local feature probes, and a basis factor (i.e.,
+M) - efficiently encoding internal relationships and patterns within the scene.
+Experimentally, we demonstrate that NeLF-Pro significantly boosts the
+performance of feature grid-based representations, and achieves fast
+reconstruction with better rendering quality while maintaining compact
+modeling. Project webpage https://sinoyou.github.io/nelf-pro/.",cs.CV,['cs.CV']
+ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers,Narges Norouzi · Svetlana Orlova · Daan de Geus · Gijs Dubbelman,https://www.tue-mps.org/ALGM/,https://arxiv.org/abs/2405.14467,,2405.14467.pdf,Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation,"Utilizing transformer architectures for semantic segmentation of
+high-resolution images is hindered by the attention's quadratic computational
+complexity in the number of tokens. A solution to this challenge involves
+decreasing the number of tokens through token merging, which has exhibited
+remarkable enhancements in inference speed, training efficiency, and memory
+utilization for image classification tasks. In this paper, we explore various
+token merging strategies within the framework of the Segformer architecture and
+perform experiments on multiple semantic segmentation and human pose estimation
+datasets. Notably, without model re-training, we, for example, achieve an
+inference acceleration of 61% on the Cityscapes dataset while maintaining the
+mIoU performance. Consequently, this paper facilitates the deployment of
+transformer-based architectures on resource-constrained devices and in
+real-time applications.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance,Junkai Fan · Jiangwei Weng · Kun Wang · Yijun Yang · Jianjun Qian · Jun Li · Jian Yang,https://fanjunkai1.github.io/projectpage/DVD/index.html,https://arxiv.org/abs/2405.09996,,2405.09996.pdf,Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance,"Real driving-video dehazing poses a significant challenge due to the inherent
+difficulty in acquiring precisely aligned hazy/clear video pairs for effective
+model training, especially in dynamic driving scenarios with unpredictable
+weather conditions. In this paper, we propose a pioneering approach that
+addresses this challenge through a nonaligned regularization strategy. Our core
+concept involves identifying clear frames that closely match hazy frames,
+serving as references to supervise a video dehazing network. Our approach
+comprises two key components: reference matching and video dehazing. Firstly,
+we introduce a non-aligned reference frame matching module, leveraging an
+adaptive sliding window to match high-quality reference frames from clear
+videos. Video dehazing incorporates flow-guided cosine attention sampler and
+deformable cosine attention fusion modules to enhance spatial multiframe
+alignment and fuse their improved information. To validate our approach, we
+collect a GoProHazy dataset captured effortlessly with GoPro cameras in diverse
+rural and urban road environments. Extensive experiments demonstrate the
+superiority of the proposed method over current state-of-the-art methods in the
+challenging task of real driving-video dehazing. Project page.",cs.CV,['cs.CV']
+Koala: Key frame-conditioned long video-LLM,Reuben Tan · Ximeng Sun · Ping Hu · Jui-Hsien Wang · Hanieh Deilamsalehy · Bryan A. Plummer · Bryan Russell · Kate Saenko, ,https://arxiv.org/abs/2404.04346,,2404.04346.pdf,Koala: Key frame-conditioned long video-LLM,"Long video question answering is a challenging task that involves recognizing
+short-term activities and reasoning about their fine-grained relationships.
+State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
+solution due to their demonstrated emergent capabilities on new tasks. However,
+despite being trained on millions of short seconds-long videos, vLLMs are
+unable to understand minutes-long videos and accurately answer questions about
+them. To address this limitation, we propose a lightweight and self-supervised
+approach, Key frame-conditioned long video-LLM (Koala), that introduces
+learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
+longer videos. Our approach introduces two new tokenizers that condition on
+visual tokens computed from sparse video key frames for understanding short and
+long video moments. We train our proposed approach on HowTo100M and demonstrate
+its effectiveness on zero-shot long video understanding benchmarks, where it
+outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
+all tasks. Surprisingly, we also empirically show that our approach not only
+helps a pretrained vLLM to understand long videos but also improves its
+accuracy on short-term action recognition.",cs.CV,['cs.CV']
+Hyperspherical Classification with Dynamic Label-to-Prototype Assignment,Mohammad Saadabadi Saadabadi · Ali Dabouei · Sahar Rahimi Malakshan · Nasser Nasrabadi, ,https://arxiv.org/abs/2403.16937,,2403.16937.pdf,Hyperspherical Classification with Dynamic Label-to-Prototype Assignment,"Aiming to enhance the utilization of metric space by the parametric softmax
+classifier, recent studies suggest replacing it with a non-parametric
+alternative. Although a non-parametric classifier may provide better metric
+space utilization, it introduces the challenge of capturing inter-class
+relationships. A shared characteristic among prior non-parametric classifiers
+is the static assignment of labels to prototypes during the training, ie, each
+prototype consistently represents a class throughout the training course.
+Orthogonal to previous works, we present a simple yet effective method to
+optimize the category assigned to each prototype (label-to-prototype
+assignment) during the training. To this aim, we formalize the problem as a
+two-step optimization objective over network parameters and label-to-prototype
+assignment mapping. We solve this optimization using a sequential combination
+of gradient descent and Bipartide matching. We demonstrate the benefits of the
+proposed approach by conducting experiments on balanced and long-tail
+classification problems using different backbone network architectures. In
+particular, our method outperforms its competitors by 1.22\% accuracy on
+CIFAR-100, and 2.15\% on ImageNet-200 using a metric space dimension half of
+the size of its competitors. Code:
+https://github.com/msed-Ebrahimi/DL2PA_CVPR24",cs.CV,['cs.CV']
+From Activation to Initialization: Scaling Insights for Optimizing Neural Fields,Hemanth Saratchandran · Sameera Ramasinghe · Simon Lucey, ,https://arxiv.org/abs/2403.19205,,2403.19205.pdf,From Activation to Initialization: Scaling Insights for Optimizing Neural Fields,"In the realm of computer vision, Neural Fields have gained prominence as a
+contemporary tool harnessing neural networks for signal representation. Despite
+the remarkable progress in adapting these networks to solve a variety of
+problems, the field still lacks a comprehensive theoretical framework. This
+article aims to address this gap by delving into the intricate interplay
+between initialization and activation, providing a foundational basis for the
+robust optimization of Neural Fields. Our theoretical insights reveal a
+deep-seated connection among network initialization, architectural choices, and
+the optimization process, emphasizing the need for a holistic approach when
+designing cutting-edge Neural Fields.",cs.CV,"['cs.CV', 'cs.LG']"
+Tune-An-Ellipse: CLIP Has Potential to Find What You Want,Jinheng Xie · Songhe Deng · Bing Li · Haozhe Liu · Yawen Huang · Yefeng Zheng · Jürgen Schmidhuber · Bernard Ghanem · Linlin Shen · Mike Zheng Shou, ,,https://cloud.tencent.com/developer/article/2396040,,,,,nan
+Neural Lineage,Runpeng Yu · Xinchao Wang, ,https://arxiv.org/abs/2312.02470v1,,2312.02470v1.pdf,Generator Born from Classifier,"In this paper, we make a bold attempt toward an ambitious task: given a
+pre-trained classifier, we aim to reconstruct an image generator, without
+relying on any data samples. From a black-box perspective, this challenge seems
+intractable, since it inevitably involves identifying the inverse function for
+a classifier, which is, by nature, an information extraction process. As such,
+we resort to leveraging the knowledge encapsulated within the parameters of the
+neural network. Grounded on the theory of Maximum-Margin Bias of gradient
+descent, we propose a novel learning paradigm, in which the generator is
+trained to ensure that the convergence conditions of the network parameters are
+satisfied over the generated distribution of the samples. Empirical validation
+from various image generation tasks substantiates the efficacy of our strategy.",cs.LG,"['cs.LG', 'cs.CV']"
+Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labels,Tianming Liang · Chaolei Tan · Beihao Xia · Wei-Shi Zheng · Jian-Fang Hu, ,https://arxiv.org/abs/2403.14430,,2403.14430.pdf,Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labels,"This paper focuses on open-ended video question answering, which aims to find
+the correct answers from a large answer set in response to a video-related
+question. This is essentially a multi-label classification task, since a
+question may have multiple answers. However, due to annotation costs, the
+labels in existing benchmarks are always extremely insufficient, typically one
+answer per question. As a result, existing works tend to directly treat all the
+unlabeled answers as negative labels, leading to limited ability for
+generalization. In this work, we introduce a simple yet effective ranking
+distillation framework (RADI) to mitigate this problem without additional
+manual annotation. RADI employs a teacher model trained with incomplete labels
+to generate rankings for potential answers, which contain rich knowledge about
+label priority as well as label-associated visual cues, thereby enriching the
+insufficient labeling information. To avoid overconfidence in the imperfect
+teacher model, we further present two robust and parameter-free ranking
+distillation approaches: a pairwise approach which introduces adaptive soft
+margins to dynamically refine the optimization constraints on various pairwise
+rankings, and a listwise approach which adopts sampling-based partial listwise
+learning to resist the bias in teacher ranking. Extensive experiments on five
+popular benchmarks consistently show that both our pairwise and listwise RADIs
+outperform state-of-the-art methods. Further analysis demonstrates the
+effectiveness of our methods on the insufficient labeling problem.",cs.CV,['cs.CV']
+Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation,Jingyun Wang · Guoliang Kang, ,https://arxiv.org/abs/2403.04547,,2403.04547.pdf,CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?,"We study the effectiveness of data-balancing for mitigating biases in
+contrastive language-image pretraining (CLIP), identifying areas of strength
+and limitation. First, we reaffirm prior conclusions that CLIP models can
+inadvertently absorb societal stereotypes. To counter this, we present a novel
+algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both
+representation and association biases (i.e. in first- and second-order
+statistics) in multimodal data. We use M4 to conduct an in-depth analysis
+taking into account various factors, such as the model, representation, and
+data size. Our study also explores the dynamic nature of how CLIP learns and
+unlearns biases. In particular, we find that fine-tuning is effective in
+countering representation biases, though its impact diminishes for association
+biases. Also, data balancing has a mixed impact on quality: it tends to improve
+classification but can hurt retrieval. Interestingly, data and architectural
+improvements seem to mitigate the negative impact of data balancing on
+performance; e.g. applying M4 to SigLIP-B/16 with data quality filters improves
+COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and
+ImageNet 0-shot classification from 77% to 77.5%! Finally, we conclude with
+recommendations for improving the efficacy of data balancing in multimodal
+systems.",cs.LG,"['cs.LG', 'cs.AI']"
+FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions,Zhen Liu · Hao Zhu · Qi Zhang · Jingde Fu · Weibing Deng · Zhan Ma · Yanwen Guo · Xun Cao, ,https://arxiv.org/abs/2312.02434,,2312.02434.pdf,FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions,"Implicit Neural Representation (INR), which utilizes a neural network to map
+coordinate inputs to corresponding attributes, is causing a revolution in the
+field of signal processing. However, current INR techniques suffer from a
+restricted capability to tune their supported frequency set, resulting in
+imperfect performance when representing complex signals with multiple
+frequencies. We have identified that this frequency-related problem can be
+greatly alleviated by introducing variable-periodic activation functions, for
+which we propose FINER. By initializing the bias of the neural network within
+different ranges, sub-functions with various frequencies in the
+variable-periodic function are selected for activation. Consequently, the
+supported frequency set of FINER can be flexibly tuned, leading to improved
+performance in signal representation. We demonstrate the capabilities of FINER
+in the contexts of 2D image fitting, 3D signed distance field representation,
+and 5D neural radiance fields optimization, and we show that it outperforms
+existing INRs.",cs.CV,['cs.CV']
+InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning,Yan-Shuo Liang · Wu-Jun Li,https://github.com/liangyanshuo/InfLoRA,https://arxiv.org/abs/2404.00228,,2404.00228.pdf,InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning,"Continual learning requires the model to learn multiple tasks sequentially.
+In continual learning, the model should possess the ability to maintain its
+performance on old tasks (stability) and the ability to adapt to new tasks
+continuously (plasticity). Recently, parameter-efficient fine-tuning (PEFT),
+which involves freezing a pre-trained model and injecting a small number of
+learnable parameters to adapt to downstream tasks, has gained increasing
+popularity in continual learning. Although existing continual learning methods
+based on PEFT have demonstrated superior performance compared to those not
+based on PEFT, most of them do not consider how to eliminate the interference
+of the new task on the old tasks, which inhibits the model from making a good
+trade-off between stability and plasticity. In this work, we propose a new PEFT
+method, called interference-free low-rank adaptation (InfLoRA), for continual
+learning. InfLoRA injects a small number of parameters to reparameterize the
+pre-trained weights and shows that fine-tuning these injected parameters is
+equivalent to fine-tuning the pre-trained weights within a subspace.
+Furthermore, InfLoRA designs this subspace to eliminate the interference of the
+new task on the old tasks, making a good trade-off between stability and
+plasticity. Experimental results show that InfLoRA outperforms existing
+state-of-the-art continual learning methods on multiple datasets.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentation,Yuanchen Wu · Xichen Ye · KequanYang · Jide Li · Xiaoqiang Li, ,https://arxiv.org/abs/2403.11184,,2403.11184.pdf,DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentation,"Recently, One-stage Weakly Supervised Semantic Segmentation (WSSS) with
+image-level labels has gained increasing interest due to simplification over
+its cumbersome multi-stage counterpart. Limited by the inherent ambiguity of
+Class Activation Map (CAM), we observe that one-stage pipelines often encounter
+confirmation bias caused by incorrect CAM pseudo-labels, impairing their final
+segmentation performance. Although recent works discard many unreliable
+pseudo-labels to implicitly alleviate this issue, they fail to exploit
+sufficient supervision for their models. To this end, we propose a dual student
+framework with trustworthy progressive learning (DuPL). Specifically, we
+propose a dual student network with a discrepancy loss to yield diverse CAMs
+for each sub-net. The two sub-nets generate supervision for each other,
+mitigating the confirmation bias caused by learning their own incorrect
+pseudo-labels. In this process, we progressively introduce more trustworthy
+pseudo-labels to be involved in the supervision through dynamic threshold
+adjustment with an adaptive noise filtering strategy. Moreover, we believe that
+every pixel, even discarded from supervision due to its unreliability, is
+important for WSSS. Thus, we develop consistency regularization on these
+discarded regions, providing supervision of every pixel. Experiment results
+demonstrate the superiority of the proposed DuPL over the recent
+state-of-the-art alternatives on PASCAL VOC 2012 and MS COCO datasets. Code is
+available at https://github.com/Wu0409/DuPL.",cs.CV,['cs.CV']
+Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding,Chaolei Tan · Jianhuang Lai · Wei-Shi Zheng · Jian-Fang Hu, ,https://arxiv.org/abs/2403.11463v2,,2403.11463v2.pdf,Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding,"Video Paragraph Grounding (VPG) is an emerging task in video-language
+understanding, which aims at localizing multiple sentences with semantic
+relations and temporal order from an untrimmed video. However, existing VPG
+approaches are heavily reliant on a considerable number of temporal labels that
+are laborious and time-consuming to acquire. In this work, we introduce and
+explore Weakly-Supervised Video Paragraph Grounding (WSVPG) to eliminate the
+need of temporal annotations. Different from previous weakly-supervised
+grounding frameworks based on multiple instance learning or reconstruction
+learning for two-stage candidate ranking, we propose a novel siamese learning
+framework that jointly learns the cross-modal feature alignment and temporal
+coordinate regression without timestamp labels to achieve concise one-stage
+localization for WSVPG. Specifically, we devise a Siamese Grounding TRansformer
+(SiamGTR) consisting of two weight-sharing branches for learning complementary
+supervision. An Augmentation Branch is utilized for directly regressing the
+temporal boundaries of a complete paragraph within a pseudo video, and an
+Inference Branch is designed to capture the order-guided feature correspondence
+for localizing multiple sentences in a normal video. We demonstrate by
+extensive experiments that our paradigm has superior practicability and
+flexibility to achieve efficient weakly-supervised or semi-supervised learning,
+outperforming state-of-the-art methods trained with the same or stronger
+supervision.",cs.CV,['cs.CV']
+Prompt Augmentation for Self-supervised Text-guided Image Manipulation,Rumeysa Bodur · Binod Bhattarai · Tae-Kyun Kim, ,https://arxiv.org/html/2403.10255v1,,2403.10255v1.pdf,Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder,"Super-resolution (SR) and image generation are important tasks in computer
+vision and are widely adopted in real-world applications. Most existing
+methods, however, generate images only at fixed-scale magnification and suffer
+from over-smoothing and artifacts. Additionally, they do not offer enough
+diversity of output images nor image consistency at different scales. Most
+relevant work applied Implicit Neural Representation (INR) to the denoising
+diffusion model to obtain continuous-resolution yet diverse and high-quality SR
+results. Since this model operates in the image space, the larger the
+resolution of image is produced, the more memory and inference time is
+required, and it also does not maintain scale-specific consistency. We propose
+a novel pipeline that can super-resolve an input image or generate from a
+random noise a novel image at arbitrary scales. The method consists of a
+pretrained auto-encoder, a latent diffusion model, and an implicit neural
+decoder, and their learning strategies. The proposed method adopts diffusion
+processes in a latent space, thus efficient, yet aligned with output image
+space decoded by MLPs at arbitrary scales. More specifically, our
+arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling
+from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in
+series. The latent diffusion process is learnt by the denoising and the
+alignment losses jointly. Errors in output images are backpropagated via the
+fixed decoder, improving the quality of output images. In the extensive
+experiments using multiple public benchmarks on the two tasks i.e. image
+super-resolution and novel image generation at arbitrary scales, the proposed
+method outperforms relevant methods in metrics of image quality, diversity and
+scale consistency. It is significantly better than the relevant prior-art in
+the inference speed and memory usage.",cs.CV,['cs.CV']
+"Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation",ZHIXIANG WEI · Lin Chen · Xiaoxiao Ma · Huaian Chen · Tianle Liu · Pengyang Ling · Jinjin Zheng · Ben Wang · Yi Jin,https://zxwei.site/rein/,https://arxiv.org/abs/2312.04265,,2312.04265.pdf,"Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation","In this paper, we first assess and harness various Vision Foundation Models
+(VFMs) in the context of Domain Generalized Semantic Segmentation (DGSS).
+Driven by the motivation that Leveraging Stronger pre-trained models and Fewer
+trainable parameters for Superior generalizability, we introduce a robust
+fine-tuning approach, namely Rein, to parameter-efficiently harness VFMs for
+DGSS. Built upon a set of trainable tokens, each linked to distinct instances,
+Rein precisely refines and forwards the feature maps from each layer to the
+next layer within the backbone. This process produces diverse refinements for
+different categories within a single image. With fewer trainable parameters,
+Rein efficiently fine-tunes VFMs for DGSS tasks, surprisingly surpassing full
+parameter fine-tuning. Extensive experiments across various settings
+demonstrate that Rein significantly outperforms state-of-the-art methods.
+Remarkably, with just an extra 1% of trainable parameters within the frozen
+backbone, Rein achieves a mIoU of 78.4% on the Cityscapes, without accessing
+any real urban-scene datasets.Code is available at
+https://github.com/w1oves/Rein.git.",cs.CV,['cs.CV']
+PoseGPT: Chatting about 3D Human Pose,Yao Feng · Jing Lin · Sai Kumar Dwivedi · Yu Sun · Priyanka Patel · Michael J. Black,https://yfeng95.github.io/ChatPose/,https://arxiv.org/abs/2311.18836,,2311.18836.pdf,ChatPose: Chatting about 3D Human Pose,"We introduce ChatPose, a framework employing Large Language Models (LLMs) to
+understand and reason about 3D human poses from images or textual descriptions.
+Our work is motivated by the human ability to intuitively understand postures
+from a single image or a brief description, a process that intertwines image
+interpretation, world knowledge, and an understanding of body language.
+Traditional human pose estimation and generation methods often operate in
+isolation, lacking semantic understanding and reasoning abilities. ChatPose
+addresses these limitations by embedding SMPL poses as distinct signal tokens
+within a multimodal LLM, enabling the direct generation of 3D body poses from
+both textual and visual inputs. Leveraging the powerful capabilities of
+multimodal LLMs, ChatPose unifies classical 3D human pose and generation tasks
+while offering user interactions. Additionally, ChatPose empowers LLMs to apply
+their extensive world knowledge in reasoning about human poses, leading to two
+advanced tasks: speculative pose generation and reasoning about pose
+estimation. These tasks involve reasoning about humans to generate 3D poses
+from subtle text queries, possibly accompanied by images. We establish
+benchmarks for these tasks, moving beyond traditional 3D pose generation and
+estimation methods. Our results show that ChatPose outperforms existing
+multimodal LLMs and task-specific methods on these newly proposed tasks.
+Furthermore, ChatPose's ability to understand and generate 3D human poses based
+on complex reasoning opens new directions in human pose analysis.",cs.CV,['cs.CV']
+SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds,Minghao Chen · Junyu Xie · Iro Laina · Andrea Vedaldi, ,,https://huggingface.co/papers/2312.09246,,,,,nan
+Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach,Beichen Zhang · Xiaoxing Wang · Xiaohan Qin · Junchi Yan, ,https://arxiv.org/abs/2403.11380,,2403.11380.pdf,Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach,"Supernet is a core component in many recent Neural Architecture Search (NAS)
+methods. It not only helps embody the search space but also provides a
+(relative) estimation of the final performance of candidate architectures.
+Thus, it is critical that the top architectures ranked by a supernet should be
+consistent with those ranked by true performance, which is known as the
+order-preserving ability. In this work, we analyze the order-preserving ability
+on the whole search space (global) and a sub-space of top architectures
+(local), and empirically show that the local order-preserving for current
+two-stage NAS methods still need to be improved. To rectify this, we propose a
+novel concept of Supernet Shifting, a refined search strategy combining
+architecture searching with supernet fine-tuning. Specifically, apart from
+evaluating, the training loss is also accumulated in searching and the supernet
+is updated every iteration. Since superior architectures are sampled more
+frequently in evolutionary searching, the supernet is encouraged to focus on
+top architectures, thus improving local order-preserving. Besides, a
+pre-trained supernet is often un-reusable for one-shot methods. We show that
+Supernet Shifting can fulfill transferring supernet to a new dataset.
+Specifically, the last classifier layer will be unset and trained through
+evolutionary searching. Comprehensive experiments show that our method has
+better order-preserving ability and can find a dominating architecture.
+Moreover, the pre-trained supernet can be easily transferred into a new dataset
+with no loss of performance.",cs.CV,['cs.CV']
+Towards General Robustness Verification of MaxPool-based Convolutional Neural Networks via Tightening Linear Approximation,Yuan Xiao · Shiqing Ma · Juan Zhai · Chunrong Fang · Jinyuan Jia · Zhenyu Chen,https://github.com/xiaoyuanpigo/maxlin,,https://software.nju.edu.cn/English/News/Selected/20240228/i260151.html,,,,,nan
+Harnessing Meta-Learning for Improving Full-Frame Video Stabilization,Muhammad Kashif Ali · Eun Woo Im · Dongjin Kim · Tae Hyun Kim, ,https://arxiv.org/abs/2403.03662v1,,2403.03662v1.pdf,Harnessing Meta-Learning for Improving Full-Frame Video Stabilization,"Video stabilization is a longstanding computer vision problem, particularly
+pixel-level synthesis solutions for video stabilization which synthesize full
+frames add to the complexity of this task. These techniques aim to stabilize
+videos by synthesizing full frames while enhancing the stability of the
+considered video. This intensifies the complexity of the task due to the
+distinct mix of unique motion profiles and visual content present in each video
+sequence, making robust generalization with fixed parameters difficult. In our
+study, we introduce a novel approach to enhance the performance of pixel-level
+synthesis solutions for video stabilization by adapting these models to
+individual input video sequences. The proposed adaptation exploits low-level
+visual cues accessible during test-time to improve both the stability and
+quality of resulting videos. We highlight the efficacy of our methodology of
+""test-time adaptation"" through simple fine-tuning of one of these models,
+followed by significant stability gain via the integration of meta-learning
+techniques. Notably, significant improvement is achieved with only a single
+adaptation step. The versatility of the proposed algorithm is demonstrated by
+consistently improving the performance of various pixel-level synthesis models
+for video stabilization in real-world scenarios.",cs.CV,['cs.CV']
+SEED-Bench: Benchmarking Multimodal Large Language Models,Bohao Li · Yuying Ge · Yixiao Ge · Guangzhi Wang · Rui Wang · Ruimao Zhang · Ying Shan, ,https://arxiv.org/abs/2307.16125,,2307.16125.pdf,SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension,"Based on powerful Large Language Models (LLMs), recent generative Multimodal
+Large Language Models (MLLMs) have gained prominence as a pivotal research
+area, exhibiting remarkable capability for both comprehension and generation.
+In this work, we address the evaluation of generative comprehension in MLLMs as
+a preliminary step towards a comprehensive assessment of generative models, by
+introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple
+choice questions with accurate human annotations (x 6 larger than existing
+benchmarks), which spans 12 evaluation dimensions including the comprehension
+of both the image and video modality. We develop an advanced pipeline for
+generating multiple-choice questions that target specific evaluation
+dimensions, integrating both automatic filtering and manual verification
+processes. Multiple-choice questions with groundtruth options derived from
+human annotation enables an objective and efficient assessment of model
+performance, eliminating the need for human or GPT intervention during
+evaluation. We further evaluate the performance of 18 models across all 12
+dimensions, covering both the spatial and temporal understanding. By revealing
+the limitations of existing MLLMs through evaluation results, we aim for
+SEED-Bench to provide insights for motivating future research. We will launch
+and consistently maintain a leaderboard to provide a platform for the community
+to assess and investigate model capability.",cs.CL,"['cs.CL', 'cs.CV']"
+Fusing Personal and Environmental Cues for Identification and Segmentation of First-Person Camera Wearers in Third-Person Views,Ziwei Zhao · Yuchen Wang · Chuhua Wang, ,,,,,,,nan
+Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models,Zhang Li · Biao Yang · Qiang Liu · Zhiyin Ma · Shuo Zhang · Jingxu Yang · Yabo Sun · Yuliang Liu · Xiang Bai, ,https://arxiv.org/abs/2311.06607,,2311.06607.pdf,Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models,"Large Multimodal Models (LMMs) have shown promise in vision-language tasks
+but struggle with high-resolution input and detailed scene understanding.
+Addressing these challenges, we introduce Monkey to enhance LMM capabilities.
+Firstly, Monkey processes input images by dividing them into uniform patches,
+each matching the size (e.g., 448x448) used in the original training of the
+well-trained vision encoder. Equipped with individual adapter for each patch,
+Monkey can handle higher resolutions up to 1344x896 pixels, enabling the
+detailed capture of complex visual information. Secondly, it employs a
+multi-level description generation method, enriching the context for
+scene-object associations. This two-part strategy ensures more effective
+learning from generated data: the higher resolution allows for a more detailed
+capture of visuals, which in turn enhances the effectiveness of comprehensive
+descriptions. Extensive ablative results validate the effectiveness of our
+designs. Additionally, experiments on 18 datasets further demonstrate that
+Monkey surpasses existing LMMs in many tasks like Image Captioning and various
+Visual Question Answering formats. Specially, in qualitative tests focused on
+dense text question answering, Monkey has exhibited encouraging results
+compared with GPT4V. Code is available at
+https://github.com/Yuliang-Liu/Monkey.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+CPP-Net: Embracing Multi-Scale Feature Fusion into Deep Unfolding CP-PPA Network for Compressive Sensing,Zhen Guo · Hongping Gan, ,,https://www.mdpi.com/1099-4300/25/12/1579,,,,,nan
+Revisiting Counterfactual Problems in Referring Expression Comprehension,Zhihan Yu · Ruifan Li, ,,https://link.springer.com/chapter/10.1007/978-3-031-41682-8_25,,,,,nan
+AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring,Xintian Mao · Xiwen Gao · Yan Wang,https://github.com/DeepMed-Lab-ECNU/Single-Image-Deblur,https://arxiv.org/abs/2402.06117,,2402.06117.pdf,Spatially-Attentive Patch-Hierarchical Network with Adaptive Sampling for Motion Deblurring,"This paper tackles the problem of motion deblurring of dynamic scenes.
+Although end-to-end fully convolutional designs have recently advanced the
+state-of-the-art in non-uniform motion deblurring, their performance-complexity
+trade-off is still sub-optimal. Most existing approaches achieve a large
+receptive field by increasing the number of generic convolution layers and
+kernel size. In this work, we propose a pixel adaptive and feature attentive
+design for handling large blur variations across different spatial locations
+and process each test image adaptively. We design a content-aware global-local
+filtering module that significantly improves performance by considering not
+only global dependencies but also by dynamically exploiting neighboring pixel
+information. We further introduce a pixel-adaptive non-uniform sampling
+strategy that implicitly discovers the difficult-to-restore regions present in
+the image and, in turn, performs fine-grained refinement in a progressive
+manner. Extensive qualitative and quantitative comparisons with prior art on
+deblurring benchmarks demonstrate that our approach performs favorably against
+the state-of-the-art deblurring algorithms.",cs.CV,['cs.CV']
+E-GPS: Explainable Geometry Problem Solving via Top-Down Solver and Bottom-Up Generator,Wenjun Wu · Lingling Zhang · Jun Liu · Xi Tang · Yaxian Wang · Shaowei Wang · QianYing Wang, ,https://arxiv.org/abs/2401.16287,,2401.16287.pdf,GAPS: Geometry-Aware Problem Solver,"Geometry problem solving presents a formidable challenge within the NLP
+community. Existing approaches often rely on models designed for solving math
+word problems, neglecting the unique characteristics of geometry math problems.
+Additionally, the current research predominantly focuses on geometry
+calculation problems, while overlooking other essential aspects like proving.
+In this study, we address these limitations by proposing the Geometry-Aware
+Problem Solver (GAPS) model. GAPS is specifically designed to generate solution
+programs for geometry math problems of various types with the help of its
+unique problem-type classifier. To achieve this, GAPS treats the solution
+program as a composition of operators and operands, segregating their
+generation processes. Furthermore, we introduce the geometry elements
+enhancement method, which enhances the ability of GAPS to recognize geometry
+elements accurately. By leveraging these improvements, GAPS showcases
+remarkable performance in resolving geometry math problems. Our experiments
+conducted on the UniGeo dataset demonstrate the superiority of GAPS over the
+state-of-the-art model, Geoformer. Specifically, GAPS achieves an accuracy
+improvement of more than 5.3% for calculation tasks and an impressive 41.1% for
+proving tasks. Notably, GAPS achieves an impressive accuracy of 97.5% on
+proving problems, representing a significant advancement in solving geometry
+proving tasks.",cs.AI,"['cs.AI', 'cs.CL']"
+IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection,Junbo Yin · Wenguan Wang · Runnan Chen · Wei Li · Ruigang Yang · Pascal Frossard · Jianbing Shen,https://github.com/yinjunbo/IS-Fusion,https://arxiv.org/abs/2403.15241,,2403.15241.pdf,IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection,"Bird's eye view (BEV) representation has emerged as a dominant solution for
+describing 3D space in autonomous driving scenarios. However, objects in the
+BEV representation typically exhibit small sizes, and the associated point
+cloud context is inherently sparse, which leads to great challenges for
+reliable 3D perception. In this paper, we propose IS-Fusion, an innovative
+multimodal fusion framework that jointly captures the Instance- and Scene-level
+contextual information. IS-Fusion essentially differs from existing approaches
+that only focus on the BEV scene-level fusion by explicitly incorporating
+instance-level multimodal information, thus facilitating the instance-centric
+tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF)
+module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid
+and Grid-to-Region transformers to capture the multimodal scene context at
+different granularities. IGF mines instance candidates, explores their
+relationships, and aggregates the local multimodal context for each instance.
+These instances then serve as guidance to enhance the scene feature and yield
+an instance-aware BEV representation. On the challenging nuScenes benchmark,
+IS-Fusion outperforms all the published multimodal works to date. Code is
+available at: https://github.com/yinjunbo/IS-Fusion.",cs.CV,['cs.CV']
+Open-Vocabulary Semantic Segmentation with Image Embedding Balancing,Xiangheng Shan · Dongyue Wu · Guilin Zhu · Yuanjie Shao · Nong Sang · Changxin Gao, ,https://arxiv.org/abs/2312.04089,,2312.04089.pdf,Open-Vocabulary Segmentation with Semantic-Assisted Calibration,"This paper studies open-vocabulary segmentation (OVS) through calibrating
+in-vocabulary and domain-biased embedding space with generalized contextual
+prior of CLIP. As the core of open-vocabulary understanding, alignment of
+visual content with the semantics of unbounded text has become the bottleneck
+of this field. To address this challenge, recent works propose to utilize CLIP
+as an additional classifier and aggregate model predictions with CLIP
+classification results. Despite their remarkable progress, performance of OVS
+methods in relevant scenarios is still unsatisfactory compared with supervised
+counterparts. We attribute this to the in-vocabulary embedding and
+domain-biased CLIP prediction. To this end, we present a Semantic-assisted
+CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior
+of CLIP into proposal embedding to avoid collapsing on known categories.
+Besides, a contextual shift strategy is applied to mitigate the lack of global
+context and unnatural background noise. With above designs, SCAN achieves
+state-of-the-art performance on all popular open-vocabulary segmentation
+benchmarks. Furthermore, we also focus on the problem of existing evaluation
+system that ignores semantic duplication across categories, and propose a new
+metric called Semantic-Guided IoU (SG-IoU).",cs.CV,['cs.CV']
+Doubly Abductive Counterfactual Inference for Text-based Image Editing,Xue Song · Jiequan Cui · Hanwang Zhang · Jingjing Chen · Richang Hong · Yu-Gang Jiang,https://github.com/xuesong39/DAC,https://arxiv.org/abs/2403.02981,,2403.02981.pdf,Doubly Abductive Counterfactual Inference for Text-based Image Editing,"We study text-based image editing (TBIE) of a single image by counterfactual
+inference because it is an elegant formulation to precisely address the
+requirement: the edited image should retain the fidelity of the original one.
+Through the lens of the formulation, we find that the crux of TBIE is that
+existing techniques hardly achieve a good trade-off between editability and
+fidelity, mainly due to the overfitting of the single-image fine-tuning. To
+this end, we propose a Doubly Abductive Counterfactual inference framework
+(DAC). We first parameterize an exogenous variable as a UNet LoRA, whose
+abduction can encode all the image details. Second, we abduct another exogenous
+variable parameterized by a text encoder LoRA, which recovers the lost
+editability caused by the overfitted first abduction. Thanks to the second
+abduction, which exclusively encodes the visual transition from post-edit to
+pre-edit, its inversion -- subtracting the LoRA -- effectively reverts pre-edit
+back to post-edit, thereby accomplishing the edit. Through extensive
+experiments, our DAC achieves a good trade-off between editability and
+fidelity. Thus, we can support a wide spectrum of user editing intents,
+including addition, removal, manipulation, replacement, style transfer, and
+facial change, which are extensively validated in both qualitative and
+quantitative evaluations. Codes are in https://github.com/xuesong39/DAC.",cs.CV,['cs.CV']
+SfmCAD: Unsupervised CAD Reconstruction by Learning Sketch-based Feature Modeling Operations,Pu Li · Jianwei Guo · HUIBIN LI · Bedrich Benes · Dong-Ming Yan, ,https://ar5iv.labs.arxiv.org/html/2303.10613,,2303.10613.pdf,SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operations,"Reverse engineering CAD models from raw geometry is a classic but strenuous
+research problem. Previous learning-based methods rely heavily on labels due to
+the supervised design patterns or reconstruct CAD shapes that are not easily
+editable. In this work, we introduce SECAD-Net, an end-to-end neural network
+aimed at reconstructing compact and easy-to-edit CAD models in a
+self-supervised manner. Drawing inspiration from the modeling language that is
+most commonly used in modern CAD software, we propose to learn 2D sketches and
+3D extrusion parameters from raw shapes, from which a set of extrusion
+cylinders can be generated by extruding each sketch from a 2D plane into a 3D
+body. By incorporating the Boolean operation (i.e., union), these cylinders can
+be combined to closely approximate the target geometry. We advocate the use of
+implicit fields for sketch representation, which allows for creating CAD
+variations by interpolating latent codes in the sketch latent space. Extensive
+experiments on both ABC and Fusion 360 datasets demonstrate the effectiveness
+of our method, and show superiority over state-of-the-art alternatives
+including the closely related method for supervised CAD reconstruction. We
+further apply our approach to CAD editing and single-view CAD reconstruction.
+The code is released at https://github.com/BunnySoCrazy/SECAD-Net.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Grounding and Enhancing Grid-based Models for Neural Fields,Zelin Zhao · FENGLEI FAN · Wenlong Liao · Junchi Yan,https://sites.google.com/view/cvpr24-2034-submission/home,https://arxiv.org/abs/2403.20002,,2403.20002.pdf,Grounding and Enhancing Grid-based Models for Neural Fields,"Many contemporary studies utilize grid-based models for neural field
+representation, but a systematic analysis of grid-based models is still
+missing, hindering the improvement of those models. Therefore, this paper
+introduces a theoretical framework for grid-based models. This framework points
+out that these models' approximation and generalization behaviors are
+determined by grid tangent kernels (GTK), which are intrinsic properties of
+grid-based models. The proposed framework facilitates a consistent and
+systematic analysis of diverse grid-based models. Furthermore, the introduced
+framework motivates the development of a novel grid-based model named the
+Multiplicative Fourier Adaptive Grid (MulFAGrid). The numerical analysis
+demonstrates that MulFAGrid exhibits a lower generalization bound than its
+predecessors, indicating its robust generalization performance. Empirical
+studies reveal that MulFAGrid achieves state-of-the-art performance in various
+tasks, including 2D image fitting, 3D signed distance field (SDF)
+reconstruction, and novel view synthesis, demonstrating superior representation
+ability. The project website is available at
+https://sites.google.com/view/cvpr24-2034-submission/home.",cs.CV,['cs.CV']
+Language Model Guided Interpretable Video Action Reasoning,Ning Wang · Guangming Zhu · Hongsheng Li · Liang Zhang · Syed Afaq Ali Shah · Mohammed Bennamoun, ,https://arxiv.org/abs/2404.01591,,2404.01591.pdf,Language Model Guided Interpretable Video Action Reasoning,"While neural networks have excelled in video action recognition tasks, their
+black-box nature often obscures the understanding of their decision-making
+processes. Recent approaches used inherently interpretable models to analyze
+video actions in a manner akin to human reasoning. These models, however,
+usually fall short in performance compared to their black-box counterparts. In
+this work, we present a new framework named Language-guided Interpretable
+Action Recognition framework (LaIAR). LaIAR leverages knowledge from language
+models to enhance both the recognition capabilities and the interpretability of
+video models. In essence, we redefine the problem of understanding video model
+decisions as a task of aligning video and language models. Using the logical
+reasoning captured by the language model, we steer the training of the video
+model. This integrated approach not only improves the video model's
+adaptability to different domains but also boosts its overall performance.
+Extensive experiments on two complex video action datasets, Charades & CAD-120,
+validates the improved performance and interpretability of our LaIAR framework.
+The code of LaIAR is available at https://github.com/NingWang2049/LaIAR.",cs.CV,['cs.CV']
+4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling,Sherwin Bahmani · Ivan Skorokhodov · Victor Rong · Gordon Wetzstein · Leonidas Guibas · Peter Wonka · Sergey Tulyakov · Jeong Joon Park · Andrea Tagliasacchi · David B. Lindell,https://sherwinbahmani.github.io/4dfy,https://arxiv.org/abs/2311.17984,,2311.17984.pdf,4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling,"Recent breakthroughs in text-to-4D generation rely on pre-trained
+text-to-image and text-to-video models to generate dynamic 3D scenes. However,
+current text-to-4D methods face a three-way tradeoff between the quality of
+scene appearance, 3D structure, and motion. For example, text-to-image models
+and their 3D-aware variants are trained on internet-scale image datasets and
+can be used to produce scenes with realistic appearance and 3D structure -- but
+no motion. Text-to-video models are trained on relatively smaller video
+datasets and can produce scenes with motion, but poorer appearance and 3D
+structure. While these models have complementary strengths, they also have
+opposing weaknesses, making it difficult to combine them in a way that
+alleviates this three-way tradeoff. Here, we introduce hybrid score
+distillation sampling, an alternating optimization procedure that blends
+supervision signals from multiple pre-trained diffusion models and incorporates
+benefits of each for high-fidelity text-to-4D generation. Using hybrid SDS, we
+demonstrate synthesis of 4D scenes with compelling appearance, 3D structure,
+and motion.",cs.CV,['cs.CV']
+Single-Model and Any-Modality for Video Object Tracking,Zongwei Wu · Jilai Zheng · Xiangxuan Ren · Florin-Alexandru Vasluianu · Chao Ma · Danda Paudel · Luc Van Gool · Radu Timofte,https://github.com/Zongwei97/UnTrack,https://arxiv.org/abs/2311.15851,,2311.15851.pdf,Single-Model and Any-Modality for Video Object Tracking,"In the realm of video object tracking, auxiliary modalities such as depth,
+thermal, or event data have emerged as valuable assets to complement the RGB
+trackers. In practice, most existing RGB trackers learn a single set of
+parameters to use them across datasets and applications. However, a similar
+single-model unification for multi-modality tracking presents several
+challenges. These challenges stem from the inherent heterogeneity of inputs --
+each with modality-specific representations, the scarcity of multi-modal
+datasets, and the absence of all the modalities at all times. In this work, we
+introduce Un-Track, a Unified Tracker of a single set of parameters for any
+modality. To handle any modality, our method learns their common latent space
+through low-rank factorization and reconstruction techniques. More importantly,
+we use only the RGB-X pairs to learn the common latent space. This unique
+shared representation seamlessly binds all modalities together, enabling
+effective unification and accommodating any missing modality, all within a
+single transformer-based architecture. Our Un-Track achieves +8.1 absolute
+F-score gain, on the DepthTrack dataset, by introducing only +2.14 (over 21.50)
+GFLOPs with +6.6M (over 93M) parameters, through a simple yet efficient
+prompting strategy. Extensive comparisons on five benchmark datasets with
+different modalities show that Un-Track surpasses both SOTA unified trackers
+and modality-specific counterparts, validating our effectiveness and
+practicality. The source code is publicly available at
+https://github.com/Zongwei97/UnTrack.",cs.CV,['cs.CV']
+Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion,Xunpeng Yi · Han Xu · HAO ZHANG · Linfeng Tang · Jiayi Ma, ,https://arxiv.org/abs/2403.16387,,2403.16387.pdf,Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion,"Image fusion aims to combine information from different source images to
+create a comprehensively representative image. Existing fusion methods are
+typically helpless in dealing with degradations in low-quality source images
+and non-interactive to multiple subjective and objective needs. To solve them,
+we introduce a novel approach that leverages semantic text guidance image
+fusion model for degradation-aware and interactive image fusion task, termed as
+Text-IF. It innovatively extends the classical image fusion to the text guided
+image fusion along with the ability to harmoniously address the degradation and
+interaction issues during fusion. Through the text semantic encoder and
+semantic interaction fusion decoder, Text-IF is accessible to the all-in-one
+infrared and visible image degradation-aware processing and the interactive
+flexible fusion outcomes. In this way, Text-IF achieves not only multi-modal
+image fusion, but also multi-modal information fusion. Extensive experiments
+prove that our proposed text guided image fusion strategy has obvious
+advantages over SOTA methods in the image fusion performance and degradation
+treatment. The code is available at https://github.com/XunpengYi/Text-IF.",cs.CV,['cs.CV']
+TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation,Sai Kumar Dwivedi · Yu Sun · Priyanka Patel · Yao Feng · Michael J. Black,https://tokenhmr.is.tue.mpg.de/,https://arxiv.org/abs/2404.16752,,2404.16752.pdf,TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation,"We address the problem of regressing 3D human pose and shape from a single
+image, with a focus on 3D accuracy. The current best methods leverage large
+datasets of 3D pseudo-ground-truth (p-GT) and 2D keypoints, leading to robust
+performance. With such methods, we observe a paradoxical decline in 3D pose
+accuracy with increasing 2D accuracy. This is caused by biases in the p-GT and
+the use of an approximate camera projection model. We quantify the error
+induced by current camera models and show that fitting 2D keypoints and p-GT
+accurately causes incorrect 3D poses. Our analysis defines the invalid
+distances within which minimizing 2D and p-GT losses is detrimental. We use
+this to formulate a new loss Threshold-Adaptive Loss Scaling (TALS) that
+penalizes gross 2D and p-GT losses but not smaller ones. With such a loss,
+there are many 3D poses that could equally explain the 2D evidence. To reduce
+this ambiguity we need a prior over valid human poses but such priors can
+introduce unwanted bias. To address this, we exploit a tokenized representation
+of human pose and reformulate the problem as token prediction. This restricts
+the estimated poses to the space of valid poses, effectively providing a
+uniform prior. Extensive experiments on the EMDB and 3DPW datasets show that
+our reformulated keypoint loss and tokenization allows us to train on
+in-the-wild data while improving 3D accuracy over the state-of-the-art. Our
+models and code are available for research at https://tokenhmr.is.tue.mpg.de.",cs.CV,['cs.CV']
+Unifying Top-down and Bottom-up Scanpath Prediction using Transformers,Zhibo Yang · Sounak Mondal · Seoyoung Ahn · Ruoyu Xue · Gregory Zelinsky · Minh Hoai · Dimitris Samaras,https://github.com/cvlab-stonybrook/HAT,https://arxiv.org/html/2303.09383v3,,2303.09383v3.pdf,Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers,"Most models of visual attention aim at predicting either top-down or
+bottom-up control, as studied using different visual search and free-viewing
+tasks. In this paper we propose the Human Attention Transformer (HAT), a single
+model that predicts both forms of attention control. HAT uses a novel
+transformer-based architecture and a simplified foveated retina that
+collectively create a spatio-temporal awareness akin to the dynamic visual
+working memory of humans. HAT not only establishes a new state-of-the-art in
+predicting the scanpath of fixations made during target-present and
+target-absent visual search and ``taskless'' free viewing, but also makes human
+gaze behavior interpretable. Unlike previous methods that rely on a coarse grid
+of fixation cells and experience information loss due to fixation
+discretization, HAT features a sequential dense prediction architecture and
+outputs a dense heatmap for each fixation, thus avoiding discretizing
+fixations. HAT sets a new standard in computational attention, which emphasizes
+effectiveness, generality, and interpretability. HAT's demonstrated scope and
+applicability will likely inspire the development of new attention models that
+can better predict human behavior in various attention-demanding scenarios.
+Code is available at https://github.com/cvlab-stonybrook/HAT.",cs.CV,"['cs.CV', 'cs.AI']"
+Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning,Chen Zhao · Shuming Liu · Karttikeya Mangalam · Guocheng Qian · Fatimah Zohra · Abdulmohsen Alghannam · Jitendra Malik · Bernard Ghanem, ,https://arxiv.org/abs/2401.04105,,2401.04105.pdf,Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning,"Large pretrained models are increasingly crucial in modern computer vision
+tasks. These models are typically used in downstream tasks by end-to-end
+finetuning, which is highly memory-intensive for tasks with high-resolution
+data, e.g., video understanding, small object detection, and point cloud
+analysis. In this paper, we propose Dynamic Reversible Dual-Residual Networks,
+or Dr$^2$Net, a novel family of network architectures that acts as a surrogate
+network to finetune a pretrained model with substantially reduced memory
+consumption. Dr$^2$Net contains two types of residual connections, one
+maintaining the residual structure in the pretrained models, and the other
+making the network reversible. Due to its reversibility, intermediate
+activations, which can be reconstructed from output, are cleared from memory
+during training. We use two coefficients on either type of residual connections
+respectively, and introduce a dynamic training strategy that seamlessly
+transitions the pretrained model to a reversible network with much higher
+numerical precision. We evaluate Dr$^2$Net on various pretrained models and
+various tasks, and show that it can reach comparable performance to
+conventional finetuning but with significantly less memory usage.",cs.CV,"['cs.CV', 'cs.AI']"
+SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction,Pin Tang · Zhongdao Wang · Guoqing Wang · Jilai Zheng · Xiangxuan Ren · Bailan Feng · Chao Ma, ,https://arxiv.org/abs/2404.09502,,2404.09502.pdf,SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction,"Vision-based perception for autonomous driving requires an explicit modeling
+of a 3D space, where 2D latent representations are mapped and subsequent 3D
+operators are applied. However, operating on dense latent spaces introduces a
+cubic time and space complexity, which limits scalability in terms of
+perception range or spatial resolution. Existing approaches compress the dense
+representation using projections like Bird's Eye View (BEV) or Tri-Perspective
+View (TPV). Although efficient, these projections result in information loss,
+especially for tasks like semantic occupancy prediction. To address this, we
+propose SparseOcc, an efficient occupancy network inspired by sparse point
+cloud processing. It utilizes a lossless sparse latent representation with
+three key innovations. Firstly, a 3D sparse diffuser performs latent completion
+using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature
+pyramid and sparse interpolation enhance scales with information from others.
+Finally, the transformer head is redesigned as a sparse variant. SparseOcc
+achieves a remarkable 74.9% reduction on FLOPs over the dense baseline.
+Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in
+part can be attributed to the sparse representation's ability to avoid
+hallucinations on empty voxels.",cs.CV,['cs.CV']
+4D Gaussian Splatting for Real-Time Dynamic Scene Rendering,Guanjun Wu · Taoran Yi · Jiemin Fang · Lingxi Xie · Xiaopeng Zhang · Wei Wei · Wenyu Liu · Qi Tian · Xinggang Wang,guanjunwu.github.io/4dgs,https://arxiv.org/abs/2310.08528,,2310.08528.pdf,4D Gaussian Splatting for Real-Time Dynamic Scene Rendering,"Representing and rendering dynamic scenes has been an important but
+challenging task. Especially, to accurately model complex motions, high
+efficiency is usually hard to guarantee. To achieve real-time dynamic scene
+rendering while also enjoying high training and storage efficiency, we propose
+4D Gaussian Splatting (4D-GS) as a holistic representation for dynamic scenes
+rather than applying 3D-GS for each individual frame. In 4D-GS, a novel
+explicit representation containing both 3D Gaussians and 4D neural voxels is
+proposed. A decomposed neural voxel encoding algorithm inspired by HexPlane is
+proposed to efficiently build Gaussian features from 4D neural voxels and then
+a lightweight MLP is applied to predict Gaussian deformations at novel
+timestamps. Our 4D-GS method achieves real-time rendering under high
+resolutions, 82 FPS at an 800$\times$800 resolution on an RTX 3090 GPU while
+maintaining comparable or better quality than previous state-of-the-art
+methods. More demos and code are available at
+https://guanjunwu.github.io/4dgs/.",cs.CV,"['cs.CV', 'cs.GR']"
+Open-Set Domain Adaptation for Semantic Segmentation,Seun-An Choe · Ah-Hyung Shin · Keon Hee Park · Jinwoo Choi · Gyeong-Moon Park, ,https://arxiv.org/abs/2405.19899,,2405.19899.pdf,Open-Set Domain Adaptation for Semantic Segmentation,"Unsupervised domain adaptation (UDA) for semantic segmentation aims to
+transfer the pixel-wise knowledge from the labeled source domain to the
+unlabeled target domain. However, current UDA methods typically assume a shared
+label space between source and target, limiting their applicability in
+real-world scenarios where novel categories may emerge in the target domain. In
+this paper, we introduce Open-Set Domain Adaptation for Semantic Segmentation
+(OSDA-SS) for the first time, where the target domain includes unknown classes.
+We identify two major problems in the OSDA-SS scenario as follows: 1) the
+existing UDA methods struggle to predict the exact boundary of the unknown
+classes, and 2) they fail to accurately predict the shape of the unknown
+classes. To address these issues, we propose Boundary and Unknown Shape-Aware
+open-set domain adaptation, coined BUS. Our BUS can accurately discern the
+boundaries between known and unknown classes in a contrastive manner using a
+novel dilation-erosion-based contrastive loss. In addition, we propose
+OpenReMix, a new domain mixing augmentation method that guides our model to
+effectively learn domain and size-invariant features for improving the shape
+detection of the known and unknown classes. Through extensive experiments, we
+demonstrate that our proposed BUS effectively detects unknown classes in the
+challenging OSDA-SS scenario compared to the previous methods by a large
+margin. The code is available at https://github.com/KHU-AGI/BUS.",cs.CV,"['cs.CV', 'cs.AI']"
+Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening,Yule Duan · Xiao Wu · Haoyu Deng · Liang-Jian Deng,https://github.com/Duanyll/CANConv,https://arxiv.org/abs/2404.07543,,2404.07543.pdf,Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening,"Currently, machine learning-based methods for remote sensing pansharpening
+have progressed rapidly. However, existing pansharpening methods often do not
+fully exploit differentiating regional information in non-local spaces, thereby
+limiting the effectiveness of the methods and resulting in redundant learning
+parameters. In this paper, we introduce a so-called content-adaptive non-local
+convolution (CANConv), a novel method tailored for remote sensing image
+pansharpening. Specifically, CANConv employs adaptive convolution, ensuring
+spatial adaptability, and incorporates non-local self-similarity through the
+similarity relationship partition (SRP) and the partition-wise adaptive
+convolution (PWAC) sub-modules. Furthermore, we also propose a corresponding
+network architecture, called CANNet, which mainly utilizes the multi-scale
+self-similarity. Extensive experiments demonstrate the superior performance of
+CANConv, compared with recent promising fusion methods. Besides, we
+substantiate the method's effectiveness through visualization, ablation
+experiments, and comparison with existing methods on multiple test sets. The
+source code is publicly available at https://github.com/duanyll/CANConv.",cs.CV,"['cs.CV', 'eess.IV']"
+GSVA: Generalized Segmentation via Multimodal Large Language Models,Zhuofan Xia · Dongchen Han · Yizeng Han · Xuran Pan · Shiji Song · Gao Huang,https://github.com/LeapLabTHU/GSVA,https://arxiv.org/abs/2312.10103,,2312.10103.pdf,GSVA: Generalized Segmentation via Multimodal Large Language Models,"Generalized Referring Expression Segmentation (GRES) extends the scope of
+classic RES to refer to multiple objects in one expression or identify the
+empty targets absent in the image. GRES poses challenges in modeling the
+complex spatial relationships of the instances in the image and identifying
+non-existing referents. Multimodal Large Language Models (MLLMs) have recently
+shown tremendous progress in these complicated vision-language tasks.
+Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient
+in understanding contexts with visual inputs. Among them, LISA, as a
+representative, adopts a special [SEG] token to prompt a segmentation mask
+decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing
+solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot
+correctly handle the cases where users might reference multiple subjects in a
+singular prompt or provide descriptions incongruent with any image target. In
+this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to
+address this gap. Specifically, GSVA reuses the [SEG] token to prompt the
+segmentation model towards supporting multiple mask references simultaneously
+and innovatively learns to generate a [REJ] token to reject the null targets
+explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue,
+marking a notable enhancement and setting a new record on the GRES benchmark
+gRefCOCO dataset. GSVA also proves effective across various classic referring
+segmentation and comprehension tasks.",cs.CV,['cs.CV']
+S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data,Xuyang Li · Danfeng Hong · Jocelyn Chanussot, ,https://arxiv.org/abs/2311.07113,,2311.07113.pdf,SpectralGPT: Spectral Remote Sensing Foundation Model,"The foundation model has recently garnered significant attention due to its
+potential to revolutionize the field of visual representation learning in a
+self-supervised manner. While most foundation models are tailored to
+effectively process RGB images for various visual tasks, there is a noticeable
+gap in research focused on spectral data, which offers valuable information for
+scene understanding, especially in remote sensing (RS) applications. To fill
+this gap, we created for the first time a universal RS foundation model, named
+SpectralGPT, which is purpose-built to handle spectral RS images using a novel
+3D generative pretrained transformer (GPT). Compared to existing foundation
+models, SpectralGPT 1) accommodates input images with varying sizes,
+resolutions, time series, and regions in a progressive training fashion,
+enabling full utilization of extensive RS big data; 2) leverages 3D token
+generation for spatial-spectral coupling; 3) captures spectrally sequential
+patterns via multi-target reconstruction; 4) trains on one million spectral RS
+images, yielding models with over 600 million parameters. Our evaluation
+highlights significant performance improvements with pretrained SpectralGPT
+models, signifying substantial potential in advancing spectral RS big data
+applications within the field of geoscience across four downstream tasks:
+single/multi-label scene classification, semantic segmentation, and change
+detection.",cs.CV,['cs.CV']
+PointOBB: Learning Oriented Object Detection via Single Point Supervision,Junwei Luo · Xue Yang · Yi Yu · Qingyun Li · Junchi Yan · Yansheng Li, ,https://arxiv.org/abs/2311.14757,,2311.14757.pdf,PointOBB: Learning Oriented Object Detection via Single Point Supervision,"Single point-supervised object detection is gaining attention due to its
+cost-effectiveness. However, existing approaches focus on generating horizontal
+bounding boxes (HBBs) while ignoring oriented bounding boxes (OBBs) commonly
+used for objects in aerial images. This paper proposes PointOBB, the first
+single Point-based OBB generation method, for oriented object detection.
+PointOBB operates through the collaborative utilization of three distinctive
+views: an original view, a resized view, and a rotated/flipped (rot/flp) view.
+Upon the original view, we leverage the resized and rot/flp views to build a
+scale augmentation module and an angle acquisition module, respectively. In the
+former module, a Scale-Sensitive Consistency (SSC) loss is designed to enhance
+the deep network's ability to perceive the object scale. For accurate object
+angle predictions, the latter module incorporates self-supervised learning to
+predict angles, which is associated with a scale-guided Dense-to-Sparse (DS)
+matching strategy for aggregating dense angles corresponding to sparse objects.
+The resized and rot/flp views are switched using a progressive multi-view
+switching strategy during training to achieve coupled optimization of scale and
+angle. Experimental results on the DIOR-R and DOTA-v1.0 datasets demonstrate
+that PointOBB achieves promising performance, and significantly outperforms
+potential point-supervised baselines.",cs.CV,"['cs.CV', 'cs.AI']"
+Long-Tail Class Incremental Learning via Independent Sub-prototype Construction,Xi Wang · Xu Yang · Jie Yin · Kun Wei · Cheng Deng, ,https://ar5iv.labs.arxiv.org/html/2210.00266,,2210.00266.pdf,Long-Tailed Class Incremental Learning,"In class incremental learning (CIL) a model must learn new classes in a
+sequential manner without forgetting old ones. However, conventional CIL
+methods consider a balanced distribution for each new task, which ignores the
+prevalence of long-tailed distributions in the real world. In this work we
+propose two long-tailed CIL scenarios, which we term ordered and shuffled
+LT-CIL. Ordered LT-CIL considers the scenario where we learn from head classes
+collected with more samples than tail classes which have few. Shuffled LT-CIL,
+on the other hand, assumes a completely random long-tailed distribution for
+each task. We systematically evaluate existing methods in both LT-CIL scenarios
+and demonstrate very different behaviors compared to conventional CIL
+scenarios. Additionally, we propose a two-stage learning baseline with a
+learnable weight scaling layer for reducing the bias caused by long-tailed
+distribution in LT-CIL and which in turn also improves the performance of
+conventional CIL due to the limited exemplars. Our results demonstrate the
+superior performance (up to 6.44 points in average incremental accuracy) of our
+approach on CIFAR-100 and ImageNet-Subset. The code is available at
+https://github.com/xialeiliu/Long-Tailed-CIL",cs.CV,['cs.CV']
+FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer,Dongyeong Hwang · Hyunju Kim · Sunwoo Kim · Kijung Shin, ,https://arxiv.org/abs/2403.12821,,2403.12821.pdf,FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer,"The success of a specific neural network architecture is closely tied to the
+dataset and task it tackles; there is no one-size-fits-all solution. Thus,
+considerable efforts have been made to quickly and accurately estimate the
+performances of neural architectures, without full training or evaluation, for
+given tasks and datasets. Neural architecture encoding has played a crucial
+role in the estimation, and graphbased methods, which treat an architecture as
+a graph, have shown prominent performance. For enhanced representation learning
+of neural architectures, we introduce FlowerFormer, a powerful graph
+transformer that incorporates the information flows within a neural
+architecture. FlowerFormer consists of two key components: (a) bidirectional
+asynchronous message passing, inspired by the flows; (b) global attention built
+on flow-based masking. Our extensive experiments demonstrate the superiority of
+FlowerFormer over existing neural encoding methods, and its effectiveness
+extends beyond computer vision models to include graph neural networks and auto
+speech recognition models. Our code is available at
+http://github.com/y0ngjaenius/CVPR2024_FLOWERFormer.",cs.LG,"['cs.LG', 'cs.AI']"
+Convolutional Prompting meets Language Models for Continual Learning,Anurag Roy · Riddhiman Moulick · Vinay Verma · Saptarshi Ghosh · Abir Das,https://cvir.github.io/projects/convprompt.html,https://arxiv.org/abs/2403.20317,,2403.20317.pdf,Convolutional Prompting meets Language Models for Continual Learning,"Continual Learning (CL) enables machine learning models to learn from
+continuously shifting new training data in absence of data from old tasks.
+Recently, pretrained vision transformers combined with prompt tuning have shown
+promise for overcoming catastrophic forgetting in CL. These approaches rely on
+a pool of learnable prompts which can be inefficient in sharing knowledge
+across tasks leading to inferior performance. In addition, the lack of
+fine-grained layer specific prompts does not allow these to fully express the
+strength of the prompts for CL. We address these limitations by proposing
+ConvPrompt, a novel convolutional prompt creation mechanism that maintains
+layer-wise shared embeddings, enabling both layer-specific learning and better
+concept transfer across tasks. The intelligent use of convolution enables us to
+maintain a low parameter overhead without compromising performance. We further
+leverage Large Language Models to generate fine-grained text descriptions of
+each category which are used to get task similarity and dynamically decide the
+number of prompts to be learned. Extensive experiments demonstrate the
+superiority of ConvPrompt and improves SOTA by ~3% with significantly less
+parameter overhead. We also perform strong ablation over various modules to
+disentangle the importance of different components.",cs.CV,['cs.CV']
+As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors,Seungwoo Yoo · Kunho Kim · Vladimir G. Kim · Minhyuk Sung, ,https://arxiv.org/abs/2311.16739,,2311.16739.pdf,As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors,"We present As-Plausible-as-Possible (APAP) mesh deformation technique that
+leverages 2D diffusion priors to preserve the plausibility of a mesh under
+user-controlled deformation. Our framework uses per-face Jacobians to represent
+mesh deformations, where mesh vertex coordinates are computed via a
+differentiable Poisson Solve. The deformed mesh is rendered, and the resulting
+2D image is used in the Score Distillation Sampling (SDS) process, which
+enables extracting meaningful plausibility priors from a pretrained 2D
+diffusion model. To better preserve the identity of the edited mesh, we
+fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a
+user-prescribed handle displacement are then backpropagated to the per-face
+Jacobians, and we use iterative gradient descent to compute the final
+deformation that balances between the user edit and the output plausibility. We
+evaluate our method with 2D and 3D meshes and demonstrate qualitative and
+quantitative improvements when using plausibility priors over
+geometry-preservation or distortion-minimization priors used by previous
+techniques. Our project page is at: https://as-plausible-aspossible.github.io/",cs.CV,"['cs.CV', 'cs.GR']"
+MR-VNet: Media Restoration using Volterra Networks,Siddharth Roheda · Amit Unde · Loay Rashid, ,,https://ieeexplore.ieee.org/document/10251925,,,,,nan
+Low-Latency Neural Stereo Streaming,Qiqi Hou · Farzad Farhadzadeh · Amir Said · Guillaume Sautiere · Hoang Le, ,https://arxiv.org/html/2403.17879v1,,2403.17879v1.pdf,Low-Latency Neural Stereo Streaming,"The rise of new video modalities like virtual reality or autonomous driving
+has increased the demand for efficient multi-view video compression methods,
+both in terms of rate-distortion (R-D) performance and in terms of delay and
+runtime. While most recent stereo video compression approaches have shown
+promising performance, they compress left and right views sequentially, leading
+to poor parallelization and runtime performance. This work presents Low-Latency
+neural codec for Stereo video Streaming (LLSS), a novel parallel stereo video
+coding method designed for fast and efficient low-latency stereo video
+streaming. Instead of using a sequential cross-view motion compensation like
+existing methods, LLSS introduces a bidirectional feature shifting module to
+directly exploit mutual information among views and encode them effectively
+with a joint cross-view prior model for entropy coding. Thanks to this design,
+LLSS processes left and right views in parallel, minimizing latency; all while
+substantially improving R-D performance compared to both existing neural and
+conventional codecs.",cs.CV,"['cs.CV', 'eess.IV']"
+SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction,Zhiyang Yao · Shuyang Liu · Xiaoyun Yuan · Lu Fang, ,,https://ieeexplore.ieee.org/document/10463068/,,,,,nan
+MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation,Sumanth Udupa · Prajwal Gurunath · Aniruddh Sikdar · Suresh Sundaram,https://arxiv.org/abs/2311.18331,https://arxiv.org/abs/2311.18331v1,,2311.18331v1.pdf,MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation,"Deep neural networks have shown exemplary performance on semantic scene
+understanding tasks on source domains, but due to the absence of style
+diversity during training, enhancing performance on unseen target domains using
+only single source domain data remains a challenging task. Generation of
+simulated data is a feasible alternative to retrieving large style-diverse
+real-world datasets as it is a cumbersome and budget-intensive process.
+However, the large domain-specific inconsistencies between simulated and
+real-world data pose a significant generalization challenge in semantic
+segmentation. In this work, to alleviate this problem, we propose a novel
+MultiResolution Feature Perturbation (MRFP) technique to randomize
+domain-specific fine-grained features and perturb style of coarse features. Our
+experimental results on various urban-scene segmentation datasets clearly
+indicate that, along with the perturbation of style-information, perturbation
+of fine-feature components is paramount to learn domain invariant robust
+feature maps for semantic segmentation models. MRFP is a simple and
+computationally efficient, transferable module with no additional learnable
+parameters or objective functions, that helps state-of-the-art deep neural
+networks to learn robust domain invariant features for simulation-to-real
+semantic segmentation.",cs.CV,"['cs.CV', 'cs.AI']"
+Theoretically Achieving Continuous Representation of Oriented Bounding Boxes,Zikai Xiao · Guo-Ye Yang · Xue Yang · Tai-Jiang Mu · Junchi Yan · Shi-Min Hu, ,https://arxiv.org/abs/2402.18975v1,,2402.18975v1.pdf,Theoretically Achieving Continuous Representation of Oriented Bounding Boxes,"Considerable efforts have been devoted to Oriented Object Detection (OOD).
+However, one lasting issue regarding the discontinuity in Oriented Bounding Box
+(OBB) representation remains unresolved, which is an inherent bottleneck for
+extant OOD methods. This paper endeavors to completely solve this issue in a
+theoretically guaranteed manner and puts an end to the ad-hoc efforts in this
+direction. Prior studies typically can only address one of the two cases of
+discontinuity: rotation and aspect ratio, and often inadvertently introduce
+decoding discontinuity, e.g. Decoding Incompleteness (DI) and Decoding
+Ambiguity (DA) as discussed in literature. Specifically, we propose a novel
+representation method called Continuous OBB (COBB), which can be readily
+integrated into existing detectors e.g. Faster-RCNN as a plugin. It can
+theoretically ensure continuity in bounding box regression which to our best
+knowledge, has not been achieved in literature for rectangle-based object
+representation. For fairness and transparency of experiments, we have developed
+a modularized benchmark based on the open-source deep learning framework
+Jittor's detection toolbox JDet for OOD evaluation. On the popular DOTA
+dataset, by integrating Faster-RCNN as the same baseline model, our new method
+outperforms the peer method Gliding Vertex by 1.13% mAP50 (relative improvement
+1.54%), and 2.46% mAP75 (relative improvement 5.91%), without any tricks.",cs.CV,"['cs.CV', 'cs.AI']"
+MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors,He Zhang · Shenghao Ren · Haolei Yuan · Jianhui Zhao · Fan Li · Shuangpeng Sun · Zhenghao Liang · Tao Yu · Qiu Shen · Xun Cao,https://metaverse-ai-lab-thu.github.io/MMVP-Dataset/,https://arxiv.org/abs/2403.17610,,2403.17610.pdf,MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors,"Foot contact is an important cue for human motion capture, understanding, and
+generation. Existing datasets tend to annotate dense foot contact using visual
+matching with thresholding or incorporating pressure signals. However, these
+approaches either suffer from low accuracy or are only designed for small-range
+and slow motion. There is still a lack of a vision-pressure multimodal dataset
+with large-range and fast human motion, as well as accurate and dense
+foot-contact annotation. To fill this gap, we propose a Multimodal MoCap
+Dataset with Vision and Pressure sensors, named MMVP. MMVP provides accurate
+and dense plantar pressure signals synchronized with RGBD observations, which
+is especially useful for both plausible shape estimation, robust pose fitting
+without foot drifting, and accurate global translation tracking. To validate
+the dataset, we propose an RGBD-P SMPL fitting method and also a
+monocular-video-based baseline framework, VP-MoCap, for human motion capture.
+Experiments demonstrate that our RGBD-P SMPL Fitting results significantly
+outperform pure visual motion capture. Moreover, VP-MoCap outperforms SOTA
+methods in foot-contact and global translation estimation accuracy. We believe
+the configuration of the dataset and the baseline frameworks will stimulate the
+research in this direction and also provide a good reference for MoCap
+applications in various domains. Project page:
+https://metaverse-ai-lab-thu.github.io/MMVP-Dataset/.",cs.CV,['cs.CV']
+Learning Correlation Structures for Vision Transformers,Manjin Kim · Paul Hongsuck Seo · Cordelia Schmid · Minsu Cho, ,https://arxiv.org/abs/2404.03924,,2404.03924.pdf,Learning Correlation Structures for Vision Transformers,"We introduce a new attention mechanism, dubbed structural self-attention
+(StructSA), that leverages rich correlation patterns naturally emerging in
+key-query interactions of attention. StructSA generates attention maps by
+recognizing space-time structures of key-query correlations via convolution and
+uses them to dynamically aggregate local contexts of value features. This
+effectively leverages rich structural patterns in images and videos such as
+scene layouts, object motion, and inter-object relations. Using StructSA as a
+main building block, we develop the structural vision transformer (StructViT)
+and evaluate its effectiveness on both image and video classification tasks,
+achieving state-of-the-art results on ImageNet-1K, Kinetics-400,
+Something-Something V1 & V2, Diving-48, and FineGym.",cs.CV,['cs.CV']
+Image Restoration by Denoising Diffusion Models With Iteratively Preconditioned Guidance,Tomer Garber · Tom Tirer,https://github.com/tirer-lab/DDPG,https://arxiv.org/abs/2312.16519,,2312.16519.pdf,Image Restoration by Denoising Diffusion Models with Iteratively Preconditioned Guidance,"Training deep neural networks has become a common approach for addressing
+image restoration problems. An alternative for training a ""task-specific""
+network for each observation model is to use pretrained deep denoisers for
+imposing only the signal's prior within iterative algorithms, without
+additional training. Recently, a sampling-based variant of this approach has
+become popular with the rise of diffusion/score-based generative models. Using
+denoisers for general purpose restoration requires guiding the iterations to
+ensure agreement of the signal with the observations. In low-noise settings,
+guidance that is based on back-projection (BP) has been shown to be a promising
+strategy (used recently also under the names ""pseudoinverse"" or
+""range/null-space"" guidance). However, the presence of noise in the
+observations hinders the gains from this approach. In this paper, we propose a
+novel guidance technique, based on preconditioning that allows traversing from
+BP-based guidance to least squares based guidance along the restoration scheme.
+The proposed approach is robust to noise while still having much simpler
+implementation than alternative methods (e.g., it does not require SVD or a
+large number of iterations). We use it within both an optimization scheme and a
+sampling-based scheme, and demonstrate its advantages over existing methods for
+image deblurring and super-resolution.",eess.IV,"['eess.IV', 'cs.CV']"
+Resource-Efficient Transformer Pruning for Finetuning of Large Models,Fatih Ilhan · Gong Su · Selim Tekin · Tiansheng Huang · Sihao Hu · Ling Liu, ,https://arxiv.org/abs/2403.14608,,2403.14608.pdf,Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey,"Large models represent a groundbreaking advancement in multiple application
+fields, enabling remarkable achievements across various tasks. However, their
+unprecedented scale comes with significant computational costs. These models,
+often consisting of billions of parameters, require vast amounts of
+computational resources for execution. Especially, the expansive scale and
+computational demands pose considerable challenges when customizing them for
+particular downstream tasks, particularly over the hardware platforms
+constrained by computational capabilities. Parameter Efficient Fine-Tuning
+(PEFT) provides a practical solution by efficiently adapt the large models over
+the various downstream tasks. In particular, PEFT refers to the process of
+adjusting the parameters of a pre-trained large models to adapt it to a
+specific task while minimizing the number of additional parameters introduced
+or computational resources required. This approach is particularly important
+when dealing with large language models with high parameter counts, as
+fine-tuning these models from scratch can be computationally expensive and
+resource-intensive, posing considerable challenges in the supporting system
+platform design. In this survey, we present comprehensive studies of various
+PEFT algorithms, examining their performance and computational overhead.
+Moreover, we provide an overview of applications developed using different PEFT
+algorithms and discuss common techniques employed to mitigate computation costs
+for PEFT. In addition to the algorithmic perspective, we overview various
+real-world system designs to investigate the implementation costs associated
+with different PEFT algorithms. This survey serves as an indispensable resource
+for researchers aiming to understand both the PEFT algorithm and its system
+implementation, offering detailed insights into recent advancements and
+practical applications.",cs.LG,['cs.LG']
+HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation,Zhiying Leng · Tolga Birdal · Xiaohui Liang · Federico Tombari, ,https://arxiv.org/abs/2403.00372,,2403.00372.pdf,HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation,"3D shape generation from text is a fundamental task in 3D representation
+learning. The text-shape pairs exhibit a hierarchical structure, where a
+general text like ``chair"" covers all 3D shapes of the chair, while more
+detailed prompts refer to more specific shapes. Furthermore, both text and 3D
+shapes are inherently hierarchical structures. However, existing Text2Shape
+methods, such as SDFusion, do not exploit that. In this work, we propose
+HyperSDFusion, a dual-branch diffusion model that generates 3D shapes from a
+given text. Since hyperbolic space is suitable for handling hierarchical data,
+we propose to learn the hierarchical representations of text and 3D shapes in
+hyperbolic space. First, we introduce a hyperbolic text-image encoder to learn
+the sequential and multi-modal hierarchical features of text in hyperbolic
+space. In addition, we design a hyperbolic text-graph convolution module to
+learn the hierarchical features of text in hyperbolic space. In order to fully
+utilize these text features, we introduce a dual-branch structure to embed text
+features in 3D feature space. At last, to endow the generated 3D shapes with a
+hierarchical structure, we devise a hyperbolic hierarchical loss. Our method is
+the first to explore the hyperbolic hierarchical representation for
+text-to-shape generation. Experimental results on the existing text-to-shape
+paired dataset, Text2Shape, achieved state-of-the-art results. We release our
+implementation under HyperSDFusion.github.io.",cs.CV,['cs.CV']
+Condition-Aware Neural Network for Controlled Image Generation,Han Cai · Muyang Li · Qinsheng Zhang · Ming-Yu Liu · Song Han, ,https://arxiv.org/abs/2404.01143,,2404.01143.pdf,Condition-Aware Neural Network for Controlled Image Generation,"We present Condition-Aware Neural Network (CAN), a new method for adding
+control to image generative models. In parallel to prior conditional control
+methods, CAN controls the image generation process by dynamically manipulating
+the weight of the neural network. This is achieved by introducing a
+condition-aware weight generation module that generates conditional weight for
+convolution/linear layers based on the input condition. We test CAN on
+class-conditional image generation on ImageNet and text-to-image generation on
+COCO. CAN consistently delivers significant improvements for diffusion
+transformer models, including DiT and UViT. In particular, CAN combined with
+EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2
+while requiring 52x fewer MACs per sampling step.",cs.CV,"['cs.CV', 'cs.AI']"
+TULIP: Transformer for Upsampling of LiDAR Point Cloud,Bin Yang · Patrick Pfreundschuh · Roland Siegwart · Marco Hutter · Peyman Moghadam · Vaishakh Patil,https://github.com/ethz-asl/TULIP,https://arxiv.org/abs/2312.06733,,2312.06733.pdf,TULIP: Transformer for Upsampling of LiDAR Point Clouds,"LiDAR Upsampling is a challenging task for the perception systems of robots
+and autonomous vehicles, due to the sparse and irregular structure of
+large-scale scene contexts. Recent works propose to solve this problem by
+converting LiDAR data from 3D Euclidean space into an image super-resolution
+problem in 2D image space. Although their methods can generate high-resolution
+range images with fine-grained details, the resulting 3D point clouds often
+blur out details and predict invalid points. In this paper, we propose TULIP, a
+new method to reconstruct high-resolution LiDAR point clouds from
+low-resolution LiDAR input. We also follow a range image-based approach but
+specifically modify the patch and window geometries of a Swin-Transformer-based
+network to better fit the characteristics of range images. We conducted several
+experiments on three public real-world and simulated datasets. TULIP
+outperforms state-of-the-art methods in all relevant metrics and generates
+robust and more realistic point clouds than prior works.",cs.CV,['cs.CV']
+PARA-Drive: Parallelized Architecture for Real-time Autonomous Driving,Xinshuo Weng · Boris Ivanovic · Yan Wang · Yue Wang · Marco Pavone, ,https://arxiv.org/abs/2311.02077,,2311.02077.pdf,EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision,"We present EmerNeRF, a simple yet powerful approach for learning
+spatial-temporal representations of dynamic driving scenes. Grounded in neural
+fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
+and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
+First, it stratifies scenes into static and dynamic fields. This decomposition
+emerges purely from self-supervision, enabling our model to learn from general,
+in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
+from the dynamic field and uses this flow field to further aggregate
+multi-frame features, amplifying the rendering precision of dynamic objects.
+Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
+represent highly-dynamic scenes self-sufficiently, without relying on ground
+truth object annotations or pre-trained models for dynamic object segmentation
+or optical flow estimation. Our method achieves state-of-the-art performance in
+sensor simulation, significantly outperforming previous methods when
+reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
+addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
+foundation model features into 4D space-time and address a general positional
+bias in modern Transformers, significantly boosting 3D perception performance
+(e.g., 37.50% relative improvement in occupancy prediction accuracy on
+average). Finally, we construct a diverse and challenging 120-sequence dataset
+to benchmark neural fields under extreme and highly-dynamic settings.",cs.CV,['cs.CV']
+Driving Everywhere with Large Language Model Policy Adaptation,Boyi Li · Yue Wang · Jiageng Mao · Boris Ivanovic · Sushant Veer · Karen Leung · Marco Pavone, ,https://arxiv.org/abs/2402.05932,,2402.05932.pdf,Driving Everywhere with Large Language Model Policy Adaptation,"Adapting driving behavior to new environments, customs, and laws is a
+long-standing problem in autonomous driving, precluding the widespread
+deployment of autonomous vehicles (AVs). In this paper, we present LLaDA, a
+simple yet powerful tool that enables human drivers and autonomous vehicles
+alike to drive everywhere by adapting their tasks and motion plans to traffic
+rules in new locations. LLaDA achieves this by leveraging the impressive
+zero-shot generalizability of large language models (LLMs) in interpreting the
+traffic rules in the local driver handbook. Through an extensive user study, we
+show that LLaDA's instructions are useful in disambiguating in-the-wild
+unexpected situations. We also demonstrate LLaDA's ability to adapt AV motion
+planning policies in real-world datasets; LLaDA outperforms baseline planning
+approaches on all our metrics. Please check our website for more details:
+https://boyiliee.github.io/llada.",cs.RO,"['cs.RO', 'cs.AI', 'cs.CL']"
+Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer,Yuwen Tan · Qinhao Zhou · Xiang Xiang · Ke Wang · Yuchuan Wu · Yongbin Li, ,https://arxiv.org/abs/2403.19979,,2403.19979.pdf,Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer,"Class-incremental learning (CIL) aims to enable models to continuously learn
+new classes while overcoming catastrophic forgetting. The introduction of
+pre-trained models has brought new tuning paradigms to CIL. In this paper, we
+revisit different parameter-efficient tuning (PET) methods within the context
+of continual learning. We observe that adapter tuning demonstrates superiority
+over prompt-based methods, even without parameter expansion in each learning
+session. Motivated by this, we propose incrementally tuning the shared adapter
+without imposing parameter update constraints, enhancing the learning capacity
+of the backbone. Additionally, we employ feature sampling from stored
+prototypes to retrain a unified classifier, further improving its performance.
+We estimate the semantic shift of old prototypes without access to past samples
+and update stored prototypes session by session. Our proposed method eliminates
+model expansion and avoids retaining any image samples. It surpasses previous
+pre-trained model-based CIL methods and demonstrates remarkable continual
+learning capabilities. Experimental results on five CIL benchmarks validate the
+effectiveness of our approach, achieving state-of-the-art (SOTA) performance.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks,Shangqian Gao · Yanfu Zhang · Feihu Huang · Heng Huang, ,https://arxiv.org/abs/2402.17862v1,,2402.17862v1.pdf,REPrune: Channel Pruning via Kernel Representative Selection,"Channel pruning is widely accepted to accelerate modern convolutional neural
+networks (CNNs). The resulting pruned model benefits from its immediate
+deployment on general-purpose software and hardware resources. However, its
+large pruning granularity, specifically at the unit of a convolution filter,
+often leads to undesirable accuracy drops due to the inflexibility of deciding
+how and where to introduce sparsity to the CNNs. In this paper, we propose
+REPrune, a novel channel pruning technique that emulates kernel pruning, fully
+exploiting the finer but structured granularity. REPrune identifies similar
+kernels within each channel using agglomerative clustering. Then, it selects
+filters that maximize the incorporation of kernel representatives while
+optimizing the maximum cluster coverage problem. By integrating with a
+simultaneous training-pruning paradigm, REPrune promotes efficient, progressive
+pruning throughout training CNNs, avoiding the conventional
+train-prune-finetune sequence. Experimental results highlight that REPrune
+performs better in computer vision tasks than existing methods, effectively
+achieving a balance between acceleration ratio and performance retention.",cs.CV,"['cs.CV', 'cs.AI']"
+DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing,Jia-Wei Liu · Yan-Pei Cao · Jay Zhangjie Wu · Weijia Mao · Yuchao Gu · Rui Zhao · Jussi Keppo · Ying Shan · Mike Zheng Shou, ,https://arxiv.org/abs/2310.10624,,2310.10624.pdf,DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing,"Despite recent progress in diffusion-based video editing, existing methods
+are limited to short-length videos due to the contradiction between long-range
+consistency and frame-wise editing. Prior attempts to address this challenge by
+introducing video-2D representations encounter significant difficulties with
+large-scale motion- and view-change videos, especially in human-centric
+scenarios. To overcome this, we propose to introduce the dynamic Neural
+Radiance Fields (NeRF) as the innovative video representation, where the
+editing can be performed in the 3D spaces and propagated to the entire video
+via the deformation field. To provide consistent and controllable editing, we
+propose the image-based video-NeRF editing pipeline with a set of innovative
+designs, including multi-view multi-pose Score Distillation Sampling (SDS) from
+both the 2D personalized diffusion prior and 3D diffusion prior, reconstruction
+losses, text-guided local parts super-resolution, and style transfer. Extensive
+experiments demonstrate that our method, dubbed as DynVideo-E, significantly
+outperforms SOTA approaches on two challenging datasets by a large margin of
+50% ~ 95% for human preference. Code will be released at
+https://showlab.github.io/DynVideo-E/.",cs.CV,['cs.CV']
+X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model,Lingmin Ran · Xiaodong Cun · Jia-Wei Liu · Rui Zhao · Song Zijie · Xintao Wang · Jussi Keppo · Mike Zheng Shou, ,https://arxiv.org/abs/2312.02238,,2312.02238.pdf,X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model,"We introduce X-Adapter, a universal upgrader to enable the pretrained
+plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
+upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
+We achieve this goal by training an additional network to control the frozen
+upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
+frozen copy of the old model to preserve the connectors of different plugins.
+Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
+from models of different versions for feature remapping. The remapped features
+will be used as guidance for the upgraded model. To enhance the guidance
+ability of X-Adapter, we employ a null-text training strategy for the upgraded
+model. After training, we also introduce a two-stage denoising strategy to
+align the initial latents of X-Adapter and the upgraded model. Thanks to our
+strategies, X-Adapter demonstrates universal compatibility with various plugins
+and also enables plugins of different versions to work together, thereby
+expanding the functionalities of diffusion community. To verify the
+effectiveness of the proposed method, we conduct extensive experiments and the
+results show that X-Adapter may facilitate wider application in the upgraded
+foundational diffusion model.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM']"
+MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model,Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou, ,https://arxiv.org/abs/2311.16498,,2311.16498.pdf,MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model,"This paper studies the human image animation task, which aims to generate a
+video of a certain reference identity following a particular motion sequence.
+Existing animation works typically employ the frame-warping technique to
+animate the reference image towards the target motion. Despite achieving
+reasonable results, these approaches face challenges in maintaining temporal
+consistency throughout the animation due to the lack of temporal modeling and
+poor preservation of reference identity. In this work, we introduce
+MagicAnimate, a diffusion-based framework that aims at enhancing temporal
+consistency, preserving reference image faithfully, and improving animation
+fidelity. To achieve this, we first develop a video diffusion model to encode
+temporal information. Second, to maintain the appearance coherence across
+frames, we introduce a novel appearance encoder to retain the intricate details
+of the reference image. Leveraging these two innovations, we further employ a
+simple video fusion technique to encourage smooth transitions for long video
+animation. Empirical results demonstrate the superiority of our method over
+baseline approaches on two benchmarks. Notably, our approach outperforms the
+strongest baseline by over 38% in terms of video fidelity on the challenging
+TikTok dancing dataset. Code and model will be made available.",cs.CV,"['cs.CV', 'cs.GR']"
+VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence,Yuchao Gu · Yipin Zhou · Bichen Wu · Licheng Yu · Jia-Wei Liu · Rui Zhao · Jay Zhangjie Wu · David Junhao Zhang · Mike Zheng Shou · Kevin Tang, ,https://arxiv.org/abs/2312.02087,,2312.02087.pdf,VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence,"Current diffusion-based video editing primarily focuses on
+structure-preserved editing by utilizing various dense correspondences to
+ensure temporal consistency and motion alignment. However, these approaches are
+often ineffective when the target edit involves a shape change. To embark on
+video editing with shape change, we explore customized video subject swapping
+in this work, where we aim to replace the main subject in a source video with a
+target subject having a distinct identity and potentially different shape. In
+contrast to previous methods that rely on dense correspondences, we introduce
+the VideoSwap framework that exploits semantic point correspondences, inspired
+by our observation that only a small number of semantic points are necessary to
+align the subject's motion trajectory and modify its shape. We also introduce
+various user-point interactions (\eg, removing points and dragging points) to
+address various semantic point correspondence. Extensive experiments
+demonstrate state-of-the-art video subject swapping results across a variety of
+real-world videos.",cs.CV,['cs.CV']
+LIVE: Online Large Video-Language Model for Streaming Video,Joya Chen · Zhaoyang Lv · Shiwei Wu · Kevin Qinghong Lin · Chenan Song · Difei Gao · Jia-Wei Liu · Ziteng Gao · Dongxing Mao · Mike Zheng Shou, ,https://arxiv.org/abs/2405.16009,,2405.16009.pdf,Streaming Long Video Understanding with Large Language Models,"This paper presents VideoStreaming, an advanced vision-language large model
+(VLLM) for video understanding, that capably understands arbitrary-length video
+with a constant number of video tokens streamingly encoded and adaptively
+selected. The challenge of video understanding in the vision language area
+mainly lies in the significant computational burden caused by the great number
+of tokens extracted from long videos. Previous works rely on sparse sampling or
+frame compression to reduce tokens. However, such approaches either disregard
+temporal information in a long time span or sacrifice spatial details,
+resulting in flawed compression. To address these limitations, our
+VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and
+Adaptive Memory Selection. The Memory-Propagated Streaming Encoding
+architecture segments long videos into short clips and sequentially encodes
+each clip with a propagated memory. In each iteration, we utilize the encoded
+results of the preceding clip as historical memory, which is integrated with
+the current clip to distill a condensed representation that encapsulates the
+video content up to the current timestamp. After the encoding process, the
+Adaptive Memory Selection strategy selects a constant number of
+question-related memories from all the historical memories and feeds them into
+the LLM to generate informative responses. The question-related selection
+reduces redundancy within the memories, enabling efficient and precise video
+understanding. Meanwhile, the disentangled video extraction and reasoning
+design allows the LLM to answer different questions about a video by directly
+selecting corresponding memories, without the need to encode the whole video
+for each question. Our model achieves superior performance and higher
+efficiency on long video benchmarks, showcasing precise temporal comprehension
+for detailed question answering.",cs.CV,['cs.CV']
+Restoration by Generation with Constrained Priors,Zheng Ding · Xuaner Zhang · Zhuowen Tu · Zhihao Xia,https://gen2res.github.io,https://arxiv.org/abs/2312.17161,,2312.17161.pdf,Restoration by Generation with Constrained Priors,"The inherent generative power of denoising diffusion models makes them
+well-suited for image restoration tasks where the objective is to find the
+optimal high-quality image within the generative space that closely resembles
+the input image. We propose a method to adapt a pretrained diffusion model for
+image restoration by simply adding noise to the input image to be restored and
+then denoise. Our method is based on the observation that the space of a
+generative model needs to be constrained. We impose this constraint by
+finetuning the generative model with a set of anchor images that capture the
+characteristics of the input image. With the constrained space, we can then
+leverage the sampling strategy used for generation to do image restoration. We
+evaluate against previous methods and show superior performances on multiple
+real-world restoration datasets in preserving identity and image quality. We
+also demonstrate an important and practical application on personalized
+restoration, where we use a personal album as the anchor images to constrain
+the generative space. This approach allows us to produce results that
+accurately preserve high-frequency details, which previous works are unable to
+do. Project webpage: https://gen2res.github.io.",cs.CV,['cs.CV']
+3D Multi-frame Fusion for Video Stabilization,Zhan Peng · Xinyi Ye · Weiyue Zhao · TIANQI LIU · Huiqiang Sun · Baopu Li · Zhiguo Cao, ,https://arxiv.org/abs/2404.12887,,2404.12887.pdf,3D Multi-frame Fusion for Video Stabilization,"In this paper, we present RStab, a novel framework for video stabilization
+that integrates 3D multi-frame fusion through volume rendering. Departing from
+conventional methods, we introduce a 3D multi-frame perspective to generate
+stabilized images, addressing the challenge of full-frame generation while
+preserving structure. The core of our approach lies in Stabilized Rendering
+(SR), a volume rendering module, which extends beyond the image fusion by
+incorporating feature fusion. The core of our RStab framework lies in
+Stabilized Rendering (SR), a volume rendering module, fusing multi-frame
+information in 3D space. Specifically, SR involves warping features and colors
+from multiple frames by projection, fusing them into descriptors to render the
+stabilized image. However, the precision of warped information depends on the
+projection accuracy, a factor significantly influenced by dynamic regions. In
+response, we introduce the Adaptive Ray Range (ARR) module to integrate depth
+priors, adaptively defining the sampling range for the projection process.
+Additionally, we propose Color Correction (CC) assisting geometric constraints
+with optical flow for accurate color aggregation. Thanks to the three modules,
+our RStab demonstrates superior performance compared with previous stabilizers
+in the field of view (FOV), image quality, and video stability across various
+datasets.",cs.CV,"['cs.CV', 'eess.IV']"
+3D Facial Expressions through Analysis-by-Neural-Synthesis,George Retsinas · Panagiotis Filntisis · Radek Danecek · Victoria Abrevaya · Anastasios Roussos · Timo Bolkart · Petros Maragos,https://georgeretsi.github.io/smirk/,https://arxiv.org/abs/2404.04104,,2404.04104.pdf,3D Facial Expressions through Analysis-by-Neural-Synthesis,"While existing methods for 3D face reconstruction from in-the-wild images
+excel at recovering the overall face shape, they commonly miss subtle, extreme,
+asymmetric, or rarely observed expressions. We improve upon these methods with
+SMIRK (Spatial Modeling for Image-based Reconstruction of Kinesics), which
+faithfully reconstructs expressive 3D faces from images. We identify two key
+limitations in existing methods: shortcomings in their self-supervised training
+formulation, and a lack of expression diversity in the training images. For
+training, most methods employ differentiable rendering to compare a predicted
+face mesh with the input image, along with a plethora of additional loss
+functions. This differentiable rendering loss not only has to provide
+supervision to optimize for 3D face geometry, camera, albedo, and lighting,
+which is an ill-posed optimization problem, but the domain gap between
+rendering and input image further hinders the learning process. Instead, SMIRK
+replaces the differentiable rendering with a neural rendering module that,
+given the rendered predicted mesh geometry, and sparsely sampled pixels of the
+input image, generates a face image. As the neural rendering gets color
+information from sampled image pixels, supervising with neural rendering-based
+reconstruction loss can focus solely on the geometry. Further, it enables us to
+generate images of the input identity with varying expressions while training.
+These are then utilized as input to the reconstruction model and used as
+supervision with ground truth geometry. This effectively augments the training
+data and enhances the generalization for diverse expressions. Our qualitative,
+quantitative and particularly our perceptual evaluations demonstrate that SMIRK
+achieves the new state-of-the art performance on accurate expression
+reconstruction. Project webpage: https://georgeretsi.github.io/smirk/.",cs.CV,['cs.CV']
+SVGDreamer: Text Guided SVG Generation with Diffusion Model,XiMing Xing · Chuang Wang · Haitao Zhou · Jing Zhang · Dong Xu · Qian Yu,https://github.com/ximinng/SVGDreamer,https://arxiv.org/abs/2312.16476,,2312.16476.pdf,SVGDreamer: Text Guided SVG Generation with Diffusion Model,"Recently, text-guided scalable vector graphics (SVGs) synthesis has shown
+promise in domains such as iconography and sketch. However, existing
+text-to-SVG generation methods lack editability and struggle with visual
+quality and result diversity. To address these limitations, we propose a novel
+text-guided vector graphics synthesis method called SVGDreamer. SVGDreamer
+incorporates a semantic-driven image vectorization (SIVE) process that enables
+the decomposition of synthesis into foreground objects and background, thereby
+enhancing editability. Specifically, the SIVE process introduces
+attention-based primitive control and an attention-mask loss function for
+effective control and manipulation of individual elements. Additionally, we
+propose a Vectorized Particle-based Score Distillation (VPSD) approach to
+address issues of shape over-smoothing, color over-saturation, limited
+diversity, and slow convergence of the existing text-to-SVG generation methods
+by modeling SVGs as distributions of control points and colors. Furthermore,
+VPSD leverages a reward model to re-weight vector particles, which improves
+aesthetic appeal and accelerates convergence. Extensive experiments are
+conducted to validate the effectiveness of SVGDreamer, demonstrating its
+superiority over baseline methods in terms of editability, visual quality, and
+diversity. Project page:
+\href{https://ximinng.github.io/SVGDreamer-project/}{https://ximinng.github.io/SVGDreamer-project/}",cs.CV,"['cs.CV', 'cs.AI']"
+ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations,Rwiddhi Chakraborty · Adrian de Sena Sletten · Michael C. Kampffmeyer, ,https://arxiv.org/abs/2403.13870,,2403.13870.pdf,ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations,"Group robustness strategies aim to mitigate learned biases in deep learning
+models that arise from spurious correlations present in their training
+datasets. However, most existing methods rely on the access to the label
+distribution of the groups, which is time-consuming and expensive to obtain. As
+a result, unsupervised group robustness strategies are sought. Based on the
+insight that a trained model's classification strategies can be inferred
+accurately based on explainability heatmaps, we introduce ExMap, an
+unsupervised two stage mechanism designed to enhance group robustness in
+traditional classifiers. ExMap utilizes a clustering module to infer
+pseudo-labels based on a model's explainability heatmaps, which are then used
+during training in lieu of actual labels. Our empirical studies validate the
+efficacy of ExMap - We demonstrate that it bridges the performance gap with its
+supervised counterparts and outperforms existing partially supervised and
+unsupervised methods. Additionally, ExMap can be seamlessly integrated with
+existing group robustness learning strategies. Finally, we demonstrate its
+potential in tackling the emerging issue of multiple shortcut
+mitigation\footnote{Code available at \url{https://github.com/rwchakra/exmap}}.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning Triangular Distribution in Visual World,Ping Chen · Xingpeng Zhang · Chengtao Zhou · dichao Fan · Peng Tu · Le Zhang · Yanlin Qian, ,https://arxiv.org/abs/2311.18605,,2311.18605.pdf,Learning Triangular Distribution in Visual World,"Convolution neural network is successful in pervasive vision tasks, including
+label distribution learning, which usually takes the form of learning an
+injection from the non-linear visual features to the well-defined labels.
+However, how the discrepancy between features is mapped to the label
+discrepancy is ambient, and its correctness is not guaranteed.To address these
+problems, we study the mathematical connection between feature and its label,
+presenting a general and simple framework for label distribution learning. We
+propose a so-called Triangular Distribution Transform (TDT) to build an
+injective function between feature and label, guaranteeing that any symmetric
+feature discrepancy linearly reflects the difference between labels. The
+proposed TDT can be used as a plug-in in mainstream backbone networks to
+address different label distribution learning tasks. Experiments on Facial Age
+Recognition, Illumination Chromaticity Estimation, and Aesthetics assessment
+show that TDT achieves on-par or better results than the prior arts.",cs.CV,['cs.CV']
+ToonerGAN: Reinforcing GANs for Obfuscating Automated Facial Indexing,Kartik Thakral · Shashikant Prasad · Stuti Aswani · Mayank Vatsa · Richa Singh, ,,https://github.com/Kartik-3004/facexformer,,,,,nan
+Boosting Adversarial Training via Fisher-Rao Norm-based Regularization,Xiangyu Yin · Wenjie Ruan, ,https://arxiv.org/abs/2403.17520,,2403.17520.pdf,Boosting Adversarial Training via Fisher-Rao Norm-based Regularization,"Adversarial training is extensively utilized to improve the adversarial
+robustness of deep neural networks. Yet, mitigating the degradation of standard
+generalization performance in adversarial-trained models remains an open
+problem. This paper attempts to resolve this issue through the lens of model
+complexity. First, We leverage the Fisher-Rao norm, a geometrically invariant
+metric for model complexity, to establish the non-trivial bounds of the
+Cross-Entropy Loss-based Rademacher complexity for a ReLU-activated Multi-Layer
+Perceptron. Then we generalize a complexity-related variable, which is
+sensitive to the changes in model width and the trade-off factors in
+adversarial training. Moreover, intensive empirical evidence validates that
+this variable highly correlates with the generalization gap of Cross-Entropy
+loss between adversarial-trained and standard-trained models, especially during
+the initial and final phases of the training process. Building upon this
+observation, we propose a novel regularization framework, called Logit-Oriented
+Adversarial Training (LOAT), which can mitigate the trade-off between
+robustness and accuracy while imposing only a negligible increase in
+computational overhead. Our extensive experiments demonstrate that the proposed
+regularization strategy can boost the performance of the prevalent adversarial
+training algorithms, including PGD-AT, TRADES, TRADES (LSE), MART, and DM-AT,
+across various network architectures. Our code will be available at
+https://github.com/TrustAI/LOAT.",cs.LG,"['cs.LG', 'cs.CV']"
+CORES: Convolutional Response-based Score for Out-of-distribution Detection,Keke Tang · Chao Hou · Weilong Peng · Runnan Chen · Peican Zhu · Wenping Wang · Zhihong Tian, ,https://arxiv.org/abs/2405.01662,,2405.01662.pdf,Out-of-distribution detection based on subspace projection of high-dimensional features output by the last convolutional layer,"Out-of-distribution (OOD) detection, crucial for reliable pattern
+classification, discerns whether a sample originates outside the training
+distribution. This paper concentrates on the high-dimensional features output
+by the final convolutional layer, which contain rich image features. Our key
+idea is to project these high-dimensional features into two specific feature
+subspaces, leveraging the dimensionality reduction capacity of the network's
+linear layers, trained with Predefined Evenly-Distribution Class Centroids
+(PEDCC)-Loss. This involves calculating the cosines of three projection angles
+and the norm values of features, thereby identifying distinctive information
+for in-distribution (ID) and OOD data, which assists in OOD detection. Building
+upon this, we have modified the batch normalization (BN) and ReLU layer
+preceding the fully connected layer, diminishing their impact on the output
+feature distributions and thereby widening the distribution gap between ID and
+OOD data features. Our method requires only the training of the classification
+network model, eschewing any need for input pre-processing or specific OOD data
+pre-tuning. Extensive experiments on several benchmark datasets demonstrates
+that our approach delivers state-of-the-art performance. Our code is available
+at https://github.com/Hewell0/ProjOOD.",cs.CV,['cs.CV']
+Higher-order Relational Reasoning for Pedestrian Trajectory Prediction,Sungjune Kim · Hyung-gun Chi · Hyerin Lim · Karthik Ramani · Jinkyu Kim · Sangpil Kim, ,https://arxiv.org/abs/2403.08032,,2403.08032.pdf,LG-Traj: LLM Guided Pedestrian Trajectory Prediction,"Accurate pedestrian trajectory prediction is crucial for various
+applications, and it requires a deep understanding of pedestrian motion
+patterns in dynamic environments. However, existing pedestrian trajectory
+prediction methods still need more exploration to fully leverage these motion
+patterns. This paper investigates the possibilities of using Large Language
+Models (LLMs) to improve pedestrian trajectory prediction tasks by inducing
+motion cues. We introduce LG-Traj, a novel approach incorporating LLMs to
+generate motion cues present in pedestrian past/observed trajectories. Our
+approach also incorporates motion cues present in pedestrian future
+trajectories by clustering future trajectories of training data using a mixture
+of Gaussians. These motion cues, along with pedestrian coordinates, facilitate
+a better understanding of the underlying representation. Furthermore, we
+utilize singular value decomposition to augment the observed trajectories,
+incorporating them into the model learning process to further enhance
+representation learning. Our method employs a transformer-based architecture
+comprising a motion encoder to model motion patterns and a social decoder to
+capture social interactions among pedestrians. We demonstrate the effectiveness
+of our approach on popular pedestrian trajectory prediction benchmarks, namely
+ETH-UCY and SDD, and present various ablation experiments to validate our
+approach.",cs.CV,"['cs.CV', 'cs.AI']"
+LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising,Yuxing Duan, ,https://arxiv.org/abs/2405.19718,,2405.19718.pdf,LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising,"Event camera has significant advantages in capturing dynamic scene
+information while being prone to noise interference, particularly in
+challenging conditions like low threshold and low illumination. However, most
+existing research focuses on gentle situations, hindering event camera
+applications in realistic complex scenarios. To tackle this limitation and
+advance the field, we construct a new paired real-world event denoising dataset
+(LED), including 3K sequences with 18K seconds of high-resolution (1200*680)
+event streams and showing three notable distinctions compared to others:
+diverse noise levels and scenes, larger-scale with high-resolution, and
+high-quality GT. Specifically, it contains stepped parameters and varying
+illumination with diverse scenarios. Moreover, based on the property of noise
+events inconsistency and signal events consistency, we propose a novel
+effective denoising framework(DED) using homogeneous dual events to generate
+the GT with better separating noise from the raw. Furthermore, we design a
+bio-inspired baseline leveraging Leaky-Integrate-and-Fire (LIF) neurons with
+dynamic thresholds to realize accurate denoising. The experimental results
+demonstrate that the remarkable performance of the proposed approach on
+different datasets.The dataset and code are at https://github.com/Yee-Sing/led.",cs.CV,['cs.CV']
+Point2CAD: Reverse Engineering CAD Models from 3D Point Clouds,Yujia Liu · Anton Obukhov · Jan D. Wegner · Konrad Schindler, ,https://arxiv.org/abs/2312.04962,,2312.04962.pdf,Point2CAD: Reverse Engineering CAD Models from 3D Point Clouds,"Computer-Aided Design (CAD) model reconstruction from point clouds is an
+important problem at the intersection of computer vision, graphics, and machine
+learning; it saves the designer significant time when iterating on in-the-wild
+objects. Recent advancements in this direction achieve relatively reliable
+semantic segmentation but still struggle to produce an adequate topology of the
+CAD model. In this work, we analyze the current state of the art for that
+ill-posed task and identify shortcomings of existing methods. We propose a
+hybrid analytic-neural reconstruction scheme that bridges the gap between
+segmented point clouds and structured CAD models and can be readily combined
+with different segmentation backbones. Moreover, to power the surface fitting
+stage, we propose a novel implicit neural representation of freeform surfaces,
+driving up the performance of our overall CAD reconstruction scheme. We
+extensively evaluate our method on the popular ABC benchmark of CAD models and
+set a new state-of-the-art for that dataset. Project page:
+https://www.obukhov.ai/point2cad}{https://www.obukhov.ai/point2cad.",cs.CV,['cs.CV']
+Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation,Hoang Chuong Nguyen · Tianyu Wang · Jose M. Alvarez · Miaomiao Liu, ,https://arxiv.org/abs/2404.14908,,2404.14908.pdf,Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation,"This paper focuses on self-supervised monocular depth estimation in dynamic
+scenes trained on monocular videos. Existing methods jointly estimate
+pixel-wise depth and motion, relying mainly on an image reconstruction loss.
+Dynamic regions1 remain a critical challenge for these methods due to the
+inherent ambiguity in depth and motion estimation, resulting in inaccurate
+depth estimation. This paper proposes a self-supervised training framework
+exploiting pseudo depth labels for dynamic regions from training data. The key
+contribution of our framework is to decouple depth estimation for static and
+dynamic regions of images in the training data. We start with an unsupervised
+depth estimation approach, which provides reliable depth estimates for static
+regions and motion cues for dynamic regions and allows us to extract moving
+object information at the instance level. In the next stage, we use an object
+network to estimate the depth of those moving objects assuming rigid motions.
+Then, we propose a new scale alignment module to address the scale ambiguity
+between estimated depths for static and dynamic regions. We can then use the
+depth labels generated to train an end-to-end depth estimation network and
+improve its performance. Extensive experiments on the Cityscapes and KITTI
+datasets show that our self-training strategy consistently outperforms existing
+self/unsupervised depth estimation methods.",cs.CV,['cs.CV']
+Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baseline,Anas Al-lahham · Muhammad Zaigham Zaheer · Nurbek Tastan · Karthik Nandakumar,https://anasemad11.github.io/CLAP/,https://arxiv.org/abs/2404.00847,,2404.00847.pdf,Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baseline,"Unsupervised (US) video anomaly detection (VAD) in surveillance applications
+is gaining more popularity recently due to its practical real-world
+applications. As surveillance videos are privacy sensitive and the availability
+of large-scale video data may enable better US-VAD systems, collaborative
+learning can be highly rewarding in this setting. However, due to the extremely
+challenging nature of the US-VAD task, where learning is carried out without
+any annotations, privacy-preserving collaborative learning of US-VAD systems
+has not been studied yet. In this paper, we propose a new baseline for anomaly
+detection capable of localizing anomalous events in complex surveillance videos
+in a fully unsupervised fashion without any labels on a privacy-preserving
+participant-based distributed training configuration. Additionally, we propose
+three new evaluation protocols to benchmark anomaly detection approaches on
+various scenarios of collaborations and data availability. Based on these
+protocols, we modify existing VAD datasets to extensively evaluate our approach
+as well as existing US SOTA methods on two large-scale datasets including
+UCF-Crime and XD-Violence. All proposed evaluation protocols, dataset splits,
+and codes are available here: https://github.com/AnasEmad11/CLAP",cs.CV,['cs.CV']
+Unlocking Pretrained Image Backbones for Semantic Image Synthesis,Tariq Berrada · Jakob Verbeek · camille couprie · Karteek Alahari, ,https://arxiv.org/abs/2312.13314,,2312.13314.pdf,Unlocking Pre-trained Image Backbones for Semantic Image Synthesis,"Semantic image synthesis, i.e., generating images from user-provided semantic
+label maps, is an important conditional image generation task as it allows to
+control both the content as well as the spatial layout of generated images.
+Although diffusion models have pushed the state of the art in generative image
+modeling, the iterative nature of their inference process makes them
+computationally demanding. Other approaches such as GANs are more efficient as
+they only need a single feed-forward pass for generation, but the image quality
+tends to suffer on large and diverse datasets. In this work, we propose a new
+class of GAN discriminators for semantic image synthesis that generates highly
+realistic images by exploiting feature backbone networks pre-trained for tasks
+such as image classification. We also introduce a new generator architecture
+with better context modeling and using cross-attention to inject noise into
+latent variables, leading to more diverse generated images. Our model, which we
+dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
+consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
+surpassing recent diffusion models while requiring two orders of magnitude less
+compute for inference.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Point2RBox: Combine Knowledge from Synthetic Visual Patterns for End-to-end Oriented Object Detection with Single Point Supervision,Yi Yu · Xue Yang · Qingyun Li · Feipeng Da · Jifeng Dai · Yu Qiao · Junchi Yan, ,https://arxiv.org/abs/2311.14758,,2311.14758.pdf,Point2RBox: Combine Knowledge from Synthetic Visual Patterns for End-to-end Oriented Object Detection with Single Point Supervision,"With the rapidly increasing demand for oriented object detection (OOD),
+recent research involving weakly-supervised detectors for learning rotated box
+(RBox) from the horizontal box (HBox) has attracted more and more attention. In
+this paper, we explore a more challenging yet label-efficient setting, namely
+single point-supervised OOD, and present our approach called Point2RBox.
+Specifically, we propose to leverage two principles: 1) Synthetic pattern
+knowledge combination: By sampling around each labeled point on the image, we
+spread the object feature to synthetic visual patterns with known boxes to
+provide the knowledge for box regression. 2) Transform self-supervision: With a
+transformed input image (e.g. scaled/rotated), the output RBoxes are trained to
+follow the same transformation so that the network can perceive the relative
+size/rotation between objects. The detector is further enhanced by a few
+devised techniques to cope with peripheral issues, e.g. the anchor/layer
+assignment as the size of the object is not available in our point supervision
+setting. To our best knowledge, Point2RBox is the first end-to-end solution for
+point-supervised OOD. In particular, our method uses a lightweight paradigm,
+yet it achieves a competitive performance among point-supervised alternatives,
+41.05%/27.62%/80.01% on DOTA/DIOR/HRSC datasets.",cs.CV,"['cs.CV', 'cs.AI']"
+A Versatile Framework for Continual Test-Time Domain Adaptation: Balancing Discriminability and Generalizability,Xu Yang · Xuan chen · Moqi Li · Kun Wei · Cheng Deng, ,https://arxiv.org/abs/2405.14602,,2405.14602.pdf,Controllable Continual Test-Time Adaptation,"Continual Test-Time Adaptation (CTTA) is an emerging and challenging task
+where a model trained in a source domain must adapt to continuously changing
+conditions during testing, without access to the original source data. CTTA is
+prone to error accumulation due to uncontrollable domain shifts, leading to
+blurred decision boundaries between categories. Existing CTTA methods primarily
+focus on suppressing domain shifts, which proves inadequate during the
+unsupervised test phase. In contrast, we introduce a novel approach that guides
+rather than suppresses these shifts. Specifically, we propose
+$\textbf{C}$ontrollable $\textbf{Co}$ntinual $\textbf{T}$est-$\textbf{T}$ime
+$\textbf{A}$daptation (C-CoTTA), which explicitly prevents any single category
+from encroaching on others, thereby mitigating the mutual influence between
+categories caused by uncontrollable shifts. Moreover, our method reduces the
+sensitivity of model to domain transformations, thereby minimizing the
+magnitude of category shifts. Extensive quantitative experiments demonstrate
+the effectiveness of our method, while qualitative analyses, such as t-SNE
+plots, confirm the theoretical validity of our approach.",cs.LG,['cs.LG']
+Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning,xin zhang · Jiawei Du · Weiying Xie · Yunsong Li · Joey Tianyi Zhou, ,https://arxiv.org/abs/2311.13613,,2311.13613.pdf,Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning,"Dataset pruning aims to construct a coreset capable of achieving performance
+comparable to the original, full dataset. Most existing dataset pruning methods
+rely on snapshot-based criteria to identify representative samples, often
+resulting in poor generalization across various pruning and cross-architecture
+scenarios. Recent studies have addressed this issue by expanding the scope of
+training dynamics considered, including factors such as forgetting event and
+probability change, typically using an averaging approach. However, these works
+struggle to integrate a broader range of training dynamics without overlooking
+well-generalized samples, which may not be sufficiently highlighted in an
+averaging manner. In this study, we propose a novel dataset pruning method
+termed as Temporal Dual-Depth Scoring (TDDS), to tackle this problem. TDDS
+utilizes a dual-depth strategy to achieve a balance between incorporating
+extensive training dynamics and identifying representative samples for dataset
+pruning. In the first depth, we estimate the series of each sample's individual
+contributions spanning the training progress, ensuring comprehensive
+integration of training dynamics. In the second depth, we focus on the
+variability of the sample-wise contributions identified in the first depth to
+highlight well-generalized samples. Extensive experiments conducted on CIFAR
+and ImageNet datasets verify the superiority of TDDS over previous SOTA
+methods. Specifically on CIFAR-100, our method achieves 54.51% accuracy with
+only 10% training data, surpassing random selection by 7.83% and other
+comparison methods by at least 12.69%.",cs.CV,"['cs.CV', 'cs.LG']"
+FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models,Ao Luo · XIN LI · Fan Yang · Jiangyu Liu · Haoqiang Fan · Shuaicheng Liu, ,https://arxiv.org/html/2312.01746v1,,2312.01746v1.pdf,Open-DDVM: A Reproduction and Extension of Diffusion Model for Optical Flow Estimation,"Recently, Google proposes DDVM which for the first time demonstrates that a
+general diffusion model for image-to-image translation task works impressively
+well on optical flow estimation task without any specific designs like RAFT.
+However, DDVM is still a closed-source model with the expensive and private
+Palette-style pretraining. In this technical report, we present the first
+open-source DDVM by reproducing it. We study several design choices and find
+those important ones. By training on 40k public data with 4 GPUs, our
+reproduction achieves comparable performance to the closed-source DDVM. The
+code and model have been released in
+https://github.com/DQiaole/FlowDiffusion_pytorch.",cs.CV,['cs.CV']
+Probabilistic Sampling of Balanced K-Means using Adiabatic Quantum Computing,Jan-Nico Zaech · Martin Danelljan · Tolga Birdal · Luc Van Gool, ,https://arxiv.org/abs/2310.12153,,2310.12153.pdf,Probabilistic Sampling of Balanced K-Means using Adiabatic Quantum Computing,"Adiabatic quantum computing (AQC) is a promising approach for discrete and
+often NP-hard optimization problems. Current AQCs allow to implement problems
+of research interest, which has sparked the development of quantum
+representations for many computer vision tasks. Despite requiring multiple
+measurements from the noisy AQC, current approaches only utilize the best
+measurement, discarding information contained in the remaining ones. In this
+work, we explore the potential of using this information for probabilistic
+balanced k-means clustering. Instead of discarding non-optimal solutions, we
+propose to use them to compute calibrated posterior probabilities with little
+additional compute cost. This allows us to identify ambiguous solutions and
+data points, which we demonstrate on a D-Wave AQC on synthetic tasks and real
+visual data.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CV']"
+OMG-Seg: Is One Model Good Enough For All Segmentation?,Xiangtai Li · Haobo Yuan · Wei Li · Henghui Ding · Size Wu · Wenwei Zhang · Yining Li · Kai Chen · Chen Change Loy, ,https://arxiv.org/abs/2401.10229,,2401.10229.pdf,OMG-Seg: Is One Model Good Enough For All Segmentation?,"In this work, we address various segmentation tasks, each traditionally
+tackled by distinct or partially unified models. We propose OMG-Seg, One Model
+that is Good enough to efficiently and effectively handle all the segmentation
+tasks, including image semantic, instance, and panoptic segmentation, as well
+as their video counterparts, open vocabulary settings, prompt-driven,
+interactive segmentation like SAM, and video object segmentation. To our
+knowledge, this is the first model to handle all these tasks in one model and
+achieve satisfactory performance. We show that OMG-Seg, a transformer-based
+encoder-decoder architecture with task-specific queries and outputs, can
+support over ten distinct segmentation tasks and yet significantly reduce
+computational and parameter overhead across various tasks and datasets. We
+rigorously evaluate the inter-task influences and correlations during
+co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.",cs.CV,['cs.CV']
+Towards Fairness-Aware Adversarial Learning,Yanghao Zhang · Tianle Zhang · Ronghui Mu · Xiaowei Huang · Wenjie Ruan, ,https://arxiv.org/abs/2402.17729,,2402.17729.pdf,Towards Fairness-Aware Adversarial Learning,"Although adversarial training (AT) has proven effective in enhancing the
+model's robustness, the recently revealed issue of fairness in robustness has
+not been well addressed, i.e. the robust accuracy varies significantly among
+different categories. In this paper, instead of uniformly evaluating the
+model's average class performance, we delve into the issue of robust fairness,
+by considering the worst-case distribution across various classes. We propose a
+novel learning paradigm, named Fairness-Aware Adversarial Learning (FAAL). As a
+generalization of conventional AT, we re-define the problem of adversarial
+training as a min-max-max framework, to ensure both robustness and fairness of
+the trained model. Specifically, by taking advantage of distributional robust
+optimization, our method aims to find the worst distribution among different
+categories, and the solution is guaranteed to obtain the upper bound
+performance with high probability. In particular, FAAL can fine-tune an unfair
+robust model to be fair within only two epochs, without compromising the
+overall clean and robust accuracies. Extensive experiments on various image
+datasets validate the superior performance and efficiency of the proposed FAAL
+compared to other state-of-the-art methods.",cs.CV,['cs.CV']
+Inter-X: Towards Versatile Human-Human Interaction Analysis,Liang Xu · Xintao Lv · Yichao Yan · Xin Jin · Wu Shuwen · Congsheng Xu · Yifan Liu · Yizhou Zhou · Fengyun Rao · Xingdong Sheng · Yunhui LIU · Wenjun Zeng · Xiaokang Yang, ,https://arxiv.org/abs/2312.16051,,2312.16051.pdf,Inter-X: Towards Versatile Human-Human Interaction Analysis,"The analysis of the ubiquitous human-human interactions is pivotal for
+understanding humans as social beings. Existing human-human interaction
+datasets typically suffer from inaccurate body motions, lack of hand gestures
+and fine-grained textual descriptions. To better perceive and generate
+human-human interactions, we propose Inter-X, a currently largest human-human
+interaction dataset with accurate body movements and diverse interaction
+patterns, together with detailed hand gestures. The dataset includes ~11K
+interaction sequences and more than 8.1M frames. We also equip Inter-X with
+versatile annotations of more than 34K fine-grained human part-level textual
+descriptions, semantic interaction categories, interaction order, and the
+relationship and personality of the subjects. Based on the elaborate
+annotations, we propose a unified benchmark composed of 4 categories of
+downstream tasks from both the perceptual and generative directions. Extensive
+experiments and comprehensive analysis show that Inter-X serves as a testbed
+for promoting the development of versatile human-human interaction analysis.
+Our dataset and benchmark will be publicly available for research purposes.",cs.CV,['cs.CV']
+ReGenNet: Towards Human Action-Reaction Synthesis,Liang Xu · Yizhou Zhou · Yichao Yan · Xin Jin · Wenhan Zhu · Fengyun Rao · Xiaokang Yang · Wenjun Zeng, ,https://arxiv.org/abs/2403.11882,,2403.11882.pdf,ReGenNet: Towards Human Action-Reaction Synthesis,"Humans constantly interact with their surrounding environments. Current
+human-centric generative models mainly focus on synthesizing humans plausibly
+interacting with static scenes and objects, while the dynamic human
+action-reaction synthesis for ubiquitous causal human-human interactions is
+less explored. Human-human interactions can be regarded as asymmetric with
+actors and reactors in atomic interaction periods. In this paper, we
+comprehensively analyze the asymmetric, dynamic, synchronous, and detailed
+nature of human-human interactions and propose the first multi-setting human
+action-reaction synthesis benchmark to generate human reactions conditioned on
+given human actions. To begin with, we propose to annotate the actor-reactor
+order of the interaction sequences for the NTU120, InterHuman, and Chi3D
+datasets. Based on them, a diffusion-based generative model with a Transformer
+decoder architecture called ReGenNet together with an explicit distance-based
+interaction loss is proposed to predict human reactions in an online manner,
+where the future states of actors are unavailable to reactors. Quantitative and
+qualitative results show that our method can generate instant and plausible
+human reactions compared to the baselines, and can generalize to unseen actor
+motions and viewpoint changes.",cs.CV,"['cs.CV', 'cs.AI']"
+Universal Novelty Detection through Adaptive Contrastive Learning,Hossein Mirzaei · Mojtaba Nafez · Mohammad Jafari · Mohammad Soltani · Mohammad Azizmalayeri · Jafar Habibi · Mohammad Sabokrou · Mohammad Rohban, ,,https://oist.mlds.jp/2024/02/27/two-papers-have-been-accepted-by-cvpr-2024/,,,,,nan
+Cross-dimension Affinity Distillation for 3D EM Neuron Segmentation,Xiaoyu Liu · Miaomiao Cai · Yinda Chen · Yueyi Zhang · Te Shi · Ruobing Zhang · Xuejin Chen · Zhiwei Xiong, ,https://arxiv.org/html/2401.03043v1,,2401.03043v1.pdf,Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing,"The current neuron reconstruction pipeline for electron microscopy (EM) data
+usually includes automatic image segmentation followed by extensive human
+expert proofreading. In this work, we aim to reduce human workload by
+predicting connectivity between over-segmented neuron pieces, taking both
+microscopy image and 3D morphology features into account, similar to human
+proofreading workflow. To this end, we first construct a dataset, named
+FlyTracing, that contains millions of pairwise connections of segments
+expanding the whole fly brain, which is three orders of magnitude larger than
+existing datasets for neuron segment connection. To learn sophisticated
+biological imaging features from the connectivity annotations, we propose a
+novel connectivity-aware contrastive learning method to generate dense
+volumetric EM image embedding. The learned embeddings can be easily
+incorporated with any point or voxel-based morphological representations for
+automatic neuron tracing. Extensive comparisons of different combination
+schemes of image and morphological representation in identifying split errors
+across the whole fly brain demonstrate the superiority of the proposed
+approach, especially for the locations that contain severe imaging artifacts,
+such as section missing and misalignment. The dataset and code are available at
+https://github.com/Levishery/Flywire-Neuron-Tracing.",cs.CV,['cs.CV']
+PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation,Ardian Umam · Cheng-Kun Yang · Min-Hung Chen · Jen-Hui Chuang · Yen-Yu Lin,https://ardianumam.github.io/partdistill/,https://arxiv.org/abs/2312.04016,,2312.04016.pdf,PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation,"This paper proposes a cross-modal distillation framework, PartDistill, which
+transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D
+shape part segmentation. PartDistill addresses three major challenges in this
+task: the lack of 3D segmentation in invisible or undetected regions in the 2D
+projections, inconsistent 2D predictions by VLMs, and the lack of knowledge
+accumulation across different 3D shapes. PartDistill consists of a teacher
+network that uses a VLM to make 2D predictions and a student network that
+learns from the 2D predictions while extracting geometrical features from
+multiple 3D shapes to carry out 3D part segmentation. A bi-directional
+distillation, including forward and backward distillations, is carried out
+within the framework, where the former forward distills the 2D predictions to
+the student network, and the latter improves the quality of the 2D predictions,
+which subsequently enhances the final 3D segmentation. Moreover, PartDistill
+can exploit generative models that facilitate effortless 3D shape creation for
+generating knowledge sources to be distilled. Through extensive experiments,
+PartDistill boosts the existing methods with substantial margins on widely used
+ShapeNetPart and PartNetE datasets, by more than 15% and 12% higher mIoU
+scores, respectively. The code for this work is available at
+https://github.com/ardianumam/PartDistill.",cs.CV,['cs.CV']
+Diffusion-FOF: Single-view Clothed Human Reconstruction via Diffusion-based Fourier Occupancy Field,Yuanzhen Li · Fei LUO · Chunxia Xiao,https://youtu.be/jm1CsLV_5XU,https://arxiv.org/abs/2311.15855,,,SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion,"A long-standing goal of 3D human reconstruction is to create lifelike and
+fully detailed 3D humans from single-view images. The main challenge lies in
+inferring unknown body shapes, appearances, and clothing details in areas not
+visible in the images. To address this, we propose SiTH, a novel pipeline that
+uniquely integrates an image-conditioned diffusion model into a 3D mesh
+reconstruction workflow. At the core of our method lies the decomposition of
+the challenging single-view reconstruction problem into generative
+hallucination and reconstruction subproblems. For the former, we employ a
+powerful generative diffusion model to hallucinate unseen back-view appearance
+based on the input images. For the latter, we leverage skinned body meshes as
+guidance to recover full-body texture meshes from the input and back-view
+images. SiTH requires as few as 500 3D human scans for training while
+maintaining its generality and robustness to diverse images. Extensive
+evaluations on two 3D human benchmarks, including our newly created one,
+highlighted our method's superior accuracy and perceptual quality in 3D
+textured human reconstruction. Our code and evaluation benchmark are available
+at https://ait.ethz.ch/sith",cs.CV,['cs.CV']
+Distilling Semantic Priors from SAM to Efficient Image Restoration Models,Quan Zhang · Xiaoyu Liu · Wei Li · Hanting Chen · Junchao Liu · Jie Hu · Zhiwei Xiong · Chun Yuan · Yunhe Wang, ,https://arxiv.org/abs/2403.16368,,2403.16368.pdf,Distilling Semantic Priors from SAM to Efficient Image Restoration Models,"In image restoration (IR), leveraging semantic priors from segmentation
+models has been a common approach to improve performance. The recent segment
+anything model (SAM) has emerged as a powerful tool for extracting advanced
+semantic priors to enhance IR tasks. However, the computational cost of SAM is
+prohibitive for IR, compared to existing smaller IR models. The incorporation
+of SAM for extracting semantic priors considerably hampers the model inference
+efficiency. To address this issue, we propose a general framework to distill
+SAM's semantic knowledge to boost exiting IR models without interfering with
+their inference process. Specifically, our proposed framework consists of the
+semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD)
+scheme. SPF fuses two kinds of information between the restored image predicted
+by the original IR model and the semantic mask predicted by SAM for the refined
+restored image. SPD leverages a self-distillation manner to distill the fused
+semantic priors to boost the performance of original IR models. Additionally,
+we design a semantic-guided relation (SGR) module for SPD, which ensures
+semantic feature representation space consistency to fully distill the priors.
+We demonstrate the effectiveness of our framework across multiple IR models and
+tasks, including deraining, deblurring, and denoising.",cs.CV,['cs.CV']
+Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features,Niladri Shekhar Dutt · Sanjeev Muralikrishnan · Niloy J. Mitra,https://diff3f.github.io/,https://arxiv.org/abs/2311.17024,,2311.17024.pdf,Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features,"We present Diff3F as a simple, robust, and class-agnostic feature descriptor
+that can be computed for untextured input shapes (meshes or point clouds). Our
+method distills diffusion features from image foundational models onto input
+shapes. Specifically, we use the input shapes to produce depth and normal maps
+as guidance for conditional image synthesis. In the process, we produce
+(diffusion) features in 2D that we subsequently lift and aggregate on the
+original surface. Our key observation is that even if the conditional image
+generations obtained from multi-view rendering of the input shapes are
+inconsistent, the associated image features are robust and, hence, can be
+directly aggregated across views. This produces semantic features on the input
+shapes, without requiring additional data or training. We perform extensive
+experiments on multiple benchmarks (SHREC'19, SHREC'20, FAUST, and TOSCA) and
+demonstrate that our features, being semantic instead of geometric, produce
+reliable correspondence across both isometric and non-isometrically related
+shape families. Code is available via the project page at
+https://diff3f.github.io/",cs.CV,"['cs.CV', 'cs.GR']"
+TutteNet: Injective 3D Deformations by Composition of 2D Mesh Deformations,Bo Sun · Thibault Groueix · Chen Song · Qixing Huang · Noam Aigerman, ,https://arxiv.org/abs/2307.09892,,2307.09892.pdf,3Deformer: A Common Framework for Image-Guided Mesh Deformation,"We propose 3Deformer, a general-purpose framework for interactive 3D shape
+editing. Given a source 3D mesh with semantic materials, and a user-specified
+semantic image, 3Deformer can accurately edit the source mesh following the
+shape guidance of the semantic image, while preserving the source topology as
+rigid as possible. Recent studies of 3D shape editing mostly focus on learning
+neural networks to predict 3D shapes, which requires high-cost 3D training
+datasets and is limited to handling objects involved in the datasets. Unlike
+these studies, our 3Deformer is a non-training and common framework, which only
+requires supervision of readily-available semantic images, and is compatible
+with editing various objects unlimited by datasets. In 3Deformer, the source
+mesh is deformed utilizing the differentiable renderer technique, according to
+the correspondences between semantic images and mesh materials. However,
+guiding complex 3D shapes with a simple 2D image incurs extra challenges, that
+is, the deform accuracy, surface smoothness, geometric rigidity, and global
+synchronization of the edited mesh should be guaranteed. To address these
+challenges, we propose a hierarchical optimization architecture to balance the
+global and local shape features, and propose further various strategies and
+losses to improve properties of accuracy, smoothness, rigidity, and so on.
+Extensive experiments show that our 3Deformer is able to produce impressive
+results and reaches the state-of-the-art level.",cs.CV,['cs.CV']
+MAGICK: A Large-scale Captioned Dataset from Matting Generated Images using Chroma Keying,Ryan Burgert · Brian Price · Jason Kuen · Yijun Li · Michael Ryoo,https://ryanndagreat.github.io/MAGICK,https://arxiv.org/abs/2307.10350,,2307.10350.pdf,Improving Multimodal Datasets with Image Captioning,"Massive web datasets play a key role in the success of large vision-language
+models like CLIP and Flamingo. However, the raw web data is noisy, and existing
+filtering methods to reduce noise often come at the expense of data diversity.
+Our work focuses on caption quality as one major source of noise, and studies
+how generated captions can increase the utility of web-scraped datapoints with
+nondescript text. Through exploring different mixing strategies for raw and
+generated captions, we outperform the best filtering method proposed by the
+DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
+candidate pool of 128M image-text pairs. Our best approach is also 2x better at
+Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
+effective source of text supervision. In experimenting with different image
+captioning models, we also demonstrate that the performance of a model on
+standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
+indicator of the utility of the captions it generates for multimodal training.
+Finally, our experiments with using generated captions at DataComp's large
+scale (1.28B image-text pairs) offer insights into the limitations of synthetic
+text, as well as the importance of image curation with increasing training data
+quantity. The synthetic captions used in our experiments are now available on
+HuggingFace.",cs.LG,"['cs.LG', 'cs.CV']"
+Generative Latent Coding for Ultra-Low Bitrate Image Compression,Zhaoyang Jia · Jiahao Li · Bin Li · Houqiang Li · Yan Lu, ,https://arxiv.org/abs/2403.03736,,2403.03736.pdf,Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer,"Recent progress in generative compression technology has significantly
+improved the perceptual quality of compressed data. However, these advancements
+primarily focus on producing high-frequency details, often overlooking the
+ability of generative models to capture the prior distribution of image
+content, thus impeding further bitrate reduction in extreme compression
+scenarios (<0.05 bpp). Motivated by the capabilities of predictive language
+models for lossless compression, this paper introduces a novel Unified Image
+Generation-Compression (UIGC) paradigm, merging the processes of generation and
+compression. A key feature of the UIGC framework is the adoption of
+vector-quantized (VQ) image models for tokenization, alongside a multi-stage
+transformer designed to exploit spatial contextual information for modeling the
+prior distribution. As such, the dual-purpose framework effectively utilizes
+the learned prior for entropy estimation and assists in the regeneration of
+lost tokens. Extensive experiments demonstrate the superiority of the proposed
+UIGC framework over existing codecs in perceptual quality and human perception,
+particularly in ultra-low bitrate scenarios (<=0.03 bpp), pioneering a new
+direction in generative compression.",cs.CV,"['cs.CV', 'cs.LG', 'eess.IV']"
+Makeup Prior Models for 3D Facial Makeup Estimation and Applications,Xingchao Yang · Takafumi Taketomi · Yuki Endo · Yoshihiro Kanamori,https://yangxingchao.github.io/makeup-priors-page/,https://arxiv.org/abs/2403.17761,,2403.17761.pdf,Makeup Prior Models for 3D Facial Makeup Estimation and Applications,"In this work, we introduce two types of makeup prior models to extend
+existing 3D face prior models: PCA-based and StyleGAN2-based priors. The
+PCA-based prior model is a linear model that is easy to construct and is
+computationally efficient. However, it retains only low-frequency information.
+Conversely, the StyleGAN2-based model can represent high-frequency information
+with relatively higher computational cost than the PCA-based model. Although
+there is a trade-off between the two models, both are applicable to 3D facial
+makeup estimation and related applications. By leveraging makeup prior models
+and designing a makeup consistency module, we effectively address the
+challenges that previous methods faced in robustly estimating makeup,
+particularly in the context of handling self-occluded faces. In experiments, we
+demonstrate that our approach reduces computational costs by several orders of
+magnitude, achieving speeds up to 180 times faster. In addition, by improving
+the accuracy of the estimated makeup, we confirm that our methods are highly
+advantageous for various 3D facial makeup applications such as 3D makeup face
+reconstruction, user-friendly makeup editing, makeup transfer, and
+interpolation.",cs.CV,"['cs.CV', 'cs.GR']"
+Asymmetric Masked Distillation for Pre-Training Small Foundation Models,Zhiyu Zhao · Bingkun Huang · Sen Xing · Gangshan Wu · Yu Qiao · Limin Wang, ,https://arxiv.org/abs/2311.03149,,,Asymmetric Masked Distillation for Pre-Training Small Foundation Models,"Self-supervised foundation models have shown great potential in computer
+vision thanks to the pre-training paradigm of masked autoencoding. Scale is a
+primary factor influencing the performance of these foundation models. However,
+these large foundation models often result in high computational cost. This
+paper focuses on pre-training relatively small vision transformer models that
+could be efficiently adapted to downstream tasks. Specifically, taking
+inspiration from knowledge distillation in model compression, we propose a new
+asymmetric masked distillation (AMD) framework for pre-training relatively
+small models with autoencoding. The core of AMD is to devise an asymmetric
+masking strategy, where the teacher model is enabled to see more context
+information with a lower masking ratio, while the student model is still
+equipped with a high masking ratio. We design customized multi-layer feature
+alignment between the teacher encoder and student encoder to regularize the
+pre-training of student MAE. To demonstrate the effectiveness and versatility
+of AMD, we apply it to both ImageMAE and VideoMAE for pre-training relatively
+small ViT models. AMD achieved 84.6% classification accuracy on IN1K using the
+ViT-B model. And AMD achieves 73.3% classification accuracy using the ViT-B
+model on the Something-in-Something V2 dataset, a 3.7% improvement over the
+original ViT-B model from VideoMAE. We also transfer AMD pre-trained models to
+downstream tasks and obtain consistent performance improvement over the
+original masked autoencoding. The code and models are available at
+https://github.com/MCG-NJU/AMD.",cs.CV,['cs.CV']
+Uncertainty-Aware Source-Free Adaptive Image Super-Resolution with Wavelet Augmentation Transformer,Yuang Ai · Xiaoqiang Zhou · Huaibo Huang · Lei Zhang · Ran He, ,https://arxiv.org/abs/2404.11273,,2404.11273.pdf,Training Transformer Models by Wavelet Losses Improves Quantitative and Visual Performance in Single Image Super-Resolution,"Transformer-based models have achieved remarkable results in low-level vision
+tasks including image super-resolution (SR). However, early Transformer-based
+approaches that rely on self-attention within non-overlapping windows encounter
+challenges in acquiring global information. To activate more input pixels
+globally, hybrid attention models have been proposed. Moreover, training by
+solely minimizing pixel-wise RGB losses, such as L1, have been found inadequate
+for capturing essential high-frequency details. This paper presents two
+contributions: i) We introduce convolutional non-local sparse attention (NLSA)
+blocks to extend the hybrid transformer architecture in order to further
+enhance its receptive field. ii) We employ wavelet losses to train Transformer
+models to improve quantitative and subjective performance. While wavelet losses
+have been explored previously, showing their power in training
+Transformer-based SR models is novel. Our experimental results demonstrate that
+the proposed model provides state-of-the-art PSNR results as well as superior
+visual performance across various benchmark datasets.",eess.IV,"['eess.IV', 'cs.CV']"
+AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval,Sixing Yan · William K. Cheung · Ivor Tsang · Wan Hang Keith Chiu · Tong Terence · Ka Chun Cheung · Simon See, ,,https://www.a-star.edu.sg/cfar/research/publications,,,,,nan
+Unveiling the Unknown: Unleashing the Power of Unknown to Known in Open-Set Source-Free Domain Adaptation,Fuli Wan · Han Zhao · Xu Yang · Cheng Deng, ,https://arxiv.org/abs/2312.03767,,2312.03767.pdf,Unknown Sample Discovery for Source Free Open Set Domain Adaptation,"Open Set Domain Adaptation (OSDA) aims to adapt a model trained on a source
+domain to a target domain that undergoes distribution shift and contains
+samples from novel classes outside the source domain. Source-free OSDA
+(SF-OSDA) techniques eliminate the need to access source domain samples, but
+current SF-OSDA methods utilize only the known classes in the target domain for
+adaptation, and require access to the entire target domain even during
+inference after adaptation, to make the distinction between known and unknown
+samples. In this paper, we introduce Unknown Sample Discovery (USD) as an
+SF-OSDA method that utilizes a temporally ensembled teacher model to conduct
+known-unknown target sample separation and adapts the student model to the
+target domain over all classes using co-training and temporal consistency
+between the teacher and the student. USD promotes Jensen-Shannon distance (JSD)
+as an effective measure for known-unknown sample separation. Our
+teacher-student framework significantly reduces error accumulation resulting
+from imperfect known-unknown sample separation, while curriculum guidance helps
+to reliably learn the distinction between target known and target unknown
+subspaces. USD appends the target model with an unknown class node, thus
+readily classifying a target sample into any of the known or unknown classes in
+subsequent post-adaptation inference stages. Empirical results show that USD is
+superior to existing SF-OSDA methods and is competitive with current OSDA
+models that utilize both source and target domains during adaptation.",cs.CV,"['cs.CV', 'cs.AI']"
+Classes Are Not Equal: An Empirical Study on Image Recognition Fairness,Jiequan Cui · Beier Zhu · Xin Wen · Xiaojuan Qi · Bei Yu · Hanwang Zhang, ,https://arxiv.org/abs/2402.18133,,2402.18133.pdf,Classes Are Not Equal: An Empirical Study on Image Recognition Fairness,"In this paper, we present an empirical study on image recognition fairness,
+i.e., extreme class accuracy disparity on balanced data like ImageNet. We
+experimentally demonstrate that classes are not equal and the fairness issue is
+prevalent for image classification models across various datasets, network
+architectures, and model capacities. Moreover, several intriguing properties of
+fairness are identified. First, the unfairness lies in problematic
+representation rather than classifier bias. Second, with the proposed concept
+of Model Prediction Bias, we investigate the origins of problematic
+representation during optimization. Our findings reveal that models tend to
+exhibit greater prediction biases for classes that are more challenging to
+recognize. It means that more other classes will be confused with harder
+classes. Then the False Positives (FPs) will dominate the learning in
+optimization, thus leading to their poor accuracy. Further, we conclude that
+data augmentation and representation learning algorithms improve overall
+performance by promoting fairness to some degree in image classification. The
+Code is available at
+https://github.com/dvlab-research/Parametric-Contrastive-Learning.",cs.LG,"['cs.LG', 'cs.CV']"
+The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding,Lorenzo Bianchi · Fabio Carrara · Nicola Messina · Claudio Gennaro · Fabrizio Falchi,https://lorebianchi98.github.io/FG-OVD/,https://arxiv.org/abs/2311.17518v2,,2311.17518v2.pdf,The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding,"Recent advancements in large vision-language models enabled visual object
+detection in open-vocabulary scenarios, where object classes are defined in
+free-text formats during inference. In this paper, we aim to probe the
+state-of-the-art methods for open-vocabulary object detection to determine to
+what extent they understand fine-grained properties of objects and their parts.
+To this end, we introduce an evaluation protocol based on dynamic vocabulary
+generation to test whether models detect, discern, and assign the correct
+fine-grained description to objects in the presence of hard-negative classes.
+We contribute with a benchmark suite of increasing difficulty and probing
+different properties like color, pattern, and material. We further enhance our
+investigation by evaluating several state-of-the-art open-vocabulary object
+detectors using the proposed protocol and find that most existing solutions,
+which shine in standard open-vocabulary benchmarks, struggle to accurately
+capture and distinguish finer object details. We conclude the paper by
+highlighting the limitations of current methodologies and exploring promising
+research directions to overcome the discovered drawbacks. Data and code are
+available at https://lorebianchi98.github.io/FG-OVD/.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Style Aligned Image Generation via Shared Attention,Amir Hertz · Andrey Voynov · Shlomi Fruchter · Daniel Cohen-Or,https://style-aligned-gen.github.io/,https://arxiv.org/abs/2312.02133v1,,2312.02133v1.pdf,Style Aligned Image Generation via Shared Attention,"Large-scale Text-to-Image (T2I) models have rapidly gained prominence across
+creative fields, generating visually compelling outputs from textual prompts.
+However, controlling these models to ensure consistent style remains
+challenging, with existing methods necessitating fine-tuning and manual
+intervention to disentangle content and style. In this paper, we introduce
+StyleAligned, a novel technique designed to establish style alignment among a
+series of generated images. By employing minimal `attention sharing' during the
+diffusion process, our method maintains style consistency across images within
+T2I models. This approach allows for the creation of style-consistent images
+using a reference style through a straightforward inversion operation. Our
+method's evaluation across diverse styles and text prompts demonstrates
+high-quality synthesis and fidelity, underscoring its efficacy in achieving
+consistent style across various inputs.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+"Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration",Yuang Ai · Huaibo Huang · Xiaoqiang Zhou · Jiexiang Wang · Ran He, ,https://arxiv.org/abs/2312.02918v2,,2312.02918v2.pdf,"Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration","Despite substantial progress, all-in-one image restoration (IR) grapples with
+persistent challenges in handling intricate real-world degradations. This paper
+introduces MPerceiver: a novel multimodal prompt learning approach that
+harnesses Stable Diffusion (SD) priors to enhance adaptiveness,
+generalizability and fidelity for all-in-one image restoration. Specifically,
+we develop a dual-branch module to master two types of SD prompts: textual for
+holistic representation and visual for multiscale detail representation. Both
+prompts are dynamically adjusted by degradation predictions from the CLIP image
+encoder, enabling adaptive responses to diverse unknown degradations. Moreover,
+a plug-in detail refinement module improves restoration fidelity via direct
+encoder-to-decoder information transformation. To assess our method, MPerceiver
+is trained on 9 tasks for all-in-one IR and outperforms state-of-the-art
+task-specific methods across most tasks. Post multitask pre-training,
+MPerceiver attains a generalized representation in low-level vision, exhibiting
+remarkable zero-shot and few-shot capabilities in unseen tasks. Extensive
+experiments on 16 IR tasks underscore the superiority of MPerceiver in terms of
+adaptiveness, generalizability and fidelity.",cs.CV,['cs.CV']
+FMA-Net: Flow Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring,Geunhyuk Youk · Jihyong Oh · Munchurl Kim,https://kaist-viclab.github.io/fmanet-site,https://arxiv.org/abs/2401.03707,,2401.03707.pdf,FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring,"We present a joint learning scheme of video super-resolution and deblurring,
+called VSRDB, to restore clean high-resolution (HR) videos from blurry
+low-resolution (LR) ones. This joint restoration problem has drawn much less
+attention compared to single restoration problems. In this paper, we propose a
+novel flow-guided dynamic filtering (FGDF) and iterative feature refinement
+with multi-attention (FRMA), which constitutes our VSRDB framework, denoted as
+FMA-Net. Specifically, our proposed FGDF enables precise estimation of both
+spatio-temporally-variant degradation and restoration kernels that are aware of
+motion trajectories through sophisticated motion representation learning.
+Compared to conventional dynamic filtering, the FGDF enables the FMA-Net to
+effectively handle large motions into the VSRDB. Additionally, the stacked FRMA
+blocks trained with our novel temporal anchor (TA) loss, which temporally
+anchors and sharpens features, refine features in a course-to-fine manner
+through iterative updates. Extensive experiments demonstrate the superiority of
+the proposed FMA-Net over state-of-the-art methods in terms of both
+quantitative and qualitative quality. Codes and pre-trained models are
+available at: https://kaist-viclab.github.io/fmanet-site",cs.CV,['cs.CV']
+Device-Wise Federated Network Pruning,Shangqian Gao · Junyi Li · Zeyu Zhang · Yanfu Zhang · Weidong Cai · Heng Huang, ,,https://lijunyi95.github.io/publications/,,,,,nan
+Differentiable Display Photometric Stereo,Seokjun Choi · Seungwoo Yoon · Giljoo Nam · Seungyong Lee · Seung-Hwan Baek, ,https://arxiv.org/abs/2306.13325,,2306.13325.pdf,Differentiable Display Photometric Stereo,"Photometric stereo leverages variations in illumination conditions to
+reconstruct surface normals. Display photometric stereo, which employs a
+conventional monitor as an illumination source, has the potential to overcome
+limitations often encountered in bulky and difficult-to-use conventional
+setups. In this paper, we present differentiable display photometric stereo
+(DDPS), addressing an often overlooked challenge in display photometric stereo:
+the design of display patterns. Departing from using heuristic display
+patterns, DDPS learns the display patterns that yield accurate normal
+reconstruction for a target system in an end-to-end manner. To this end, we
+propose a differentiable framework that couples basis-illumination image
+formation with analytic photometric-stereo reconstruction. The differentiable
+framework facilitates the effective learning of display patterns via
+auto-differentiation. Also, for training supervision, we propose to use 3D
+printing for creating a real-world training dataset, enabling accurate
+reconstruction on the target real-world setup. Finally, we exploit that
+conventional LCD monitors emit polarized light, which allows for the optical
+separation of diffuse and specular reflections when combined with a
+polarization camera, leading to accurate normal reconstruction. Extensive
+evaluation of DDPS shows improved normal-reconstruction accuracy compared to
+heuristic patterns and demonstrates compelling properties such as robustness to
+pattern initialization, calibration errors, and simplifications in image
+formation and reconstruction.",cs.CV,['cs.CV']
+"Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction",Yizhi Wang · Wallace Lira · Wenqi Wang · Ali Mahdavi Amiri · Hao Zhang,https://yizhiwang96.github.io/Slice3D/,https://arxiv.org/abs/2312.02221,,2312.02221.pdf,"Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction","We introduce multi-slice reasoning, a new notion for single-view 3D
+reconstruction which challenges the current and prevailing belief that
+multi-view synthesis is the most natural conduit between single-view and 3D.
+Our key observation is that object slicing is more advantageous than altering
+views to reveal occluded structures. Specifically, slicing is more
+occlusion-revealing since it can peel through any occluders without
+obstruction. In the limit, i.e., with infinitely many slices, it is guaranteed
+to unveil all hidden object parts. We realize our idea by developing Slice3D, a
+novel method for single-view 3D reconstruction which first predicts multi-slice
+images from a single RGB image and then integrates the slices into a 3D model
+using a coordinate-based transformer network for signed distance prediction.
+The slice images can be regressed or generated, both through a U-Net based
+network. For the former, we inject a learnable slice indicator code to
+designate each decoded image into a spatial slice location, while the slice
+generator is a denoising diffusion model operating on the entirety of slice
+images stacked on the input channels. We conduct extensive evaluation against
+state-of-the-art alternatives to demonstrate superiority of our method,
+especially in recovering complex and severely occluded shape structures, amid
+ambiguities. All Slice3D results were produced by networks trained on a single
+Nvidia A40 GPU, with an inference time less than 20 seconds.",cs.CV,"['cs.CV', 'cs.GR']"
+Cyclic Learning for Binaural Audio Generation and Localization,Zhaojian Li · Bin Zhao · Yuan Yuan, ,https://arxiv.org/abs/2311.07630,,2311.07630.pdf,Cross-modal Generative Model for Visual-Guided Binaural Stereo Generation,"Binaural stereo audio is recorded by imitating the way the human ear receives
+sound, which provides people with an immersive listening experience. Existing
+approaches leverage autoencoders and directly exploit visual spatial
+information to synthesize binaural stereo, resulting in a limited
+representation of visual guidance. For the first time, we propose a visually
+guided generative adversarial approach for generating binaural stereo audio
+from mono audio. Specifically, we develop a Stereo Audio Generation Model
+(SAGM), which utilizes shared spatio-temporal visual information to guide the
+generator and the discriminator to work separately. The shared visual
+information is updated alternately in the generative adversarial stage,
+allowing the generator and discriminator to deliver their respective guided
+knowledge while visually sharing. The proposed method learns bidirectional
+complementary visual information, which facilitates the expression of visual
+guidance in generation. In addition, spatial perception is a crucial attribute
+of binaural stereo audio, and thus the evaluation of stereo spatial perception
+is essential. However, previous metrics failed to measure the spatial
+perception of audio. To this end, a metric to measure the spatial perception of
+audio is proposed for the first time. The proposed metric is capable of
+measuring the magnitude and direction of spatial perception in the temporal
+dimension. Further, considering its function, it is feasible to utilize it
+instead of demanding user studies to some extent. The proposed method achieves
+state-of-the-art performance on 2 datasets and 5 evaluation metrics.
+Qualitative experiments and user studies demonstrate that the method generates
+space-realistic stereo audio.",cs.SD,"['cs.SD', 'cs.CV', 'cs.LG', 'eess.AS']"
+OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition,Tongjia Chen · Hongshan Yu · Zhengeng Yang · Zechuan Li · Wei Sun · Chen Chen,https://tomchen-ctj.github.io/OST/,https://arxiv.org/abs/2312.00096,,2312.00096.pdf,OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition,"Due to the resource-intensive nature of training vision-language models on
+expansive video data, a majority of studies have centered on adapting
+pre-trained image-language models to the video domain. Dominant pipelines
+propose to tackle the visual discrepancies with additional temporal learners
+while overlooking the substantial discrepancy for web-scaled descriptive
+narratives and concise action category names, leading to less distinct semantic
+space and potential performance limitations. In this work, we prioritize the
+refinement of text knowledge to facilitate generalizable video recognition. To
+address the limitations of the less distinct semantic space of category names,
+we prompt a large language model (LLM) to augment action class names into
+Spatio-Temporal Descriptors thus bridging the textual discrepancy and serving
+as a knowledge base for general recognition. Moreover, to assign the best
+descriptors with different video instances, we propose Optimal Descriptor
+Solver, forming the video recognition problem as solving the optimal matching
+flow across frame-level representations and descriptors. Comprehensive
+evaluations in zero-shot, few-shot, and fully supervised video recognition
+highlight the effectiveness of our approach. Our best model achieves a
+state-of-the-art zero-shot accuracy of 75.1% on Kinetics-600.",cs.CV,['cs.CV']
+Visual Objectification in Films: Towards a New AI Task for Video Interpretation,Julie Tores · Lucile Sassatelli · Hui-Yin Wu · Clement Bergman · Léa Andolfi · Victor Ecrement · Frederic Precioso · Thierry Devars · Magali GUARESI · Virginie Julliard · Sarah Lécossais, ,https://arxiv.org/abs/2401.13296,,2401.13296.pdf,Visual Objectification in Films: Towards a New AI Task for Video Interpretation,"In film gender studies, the concept of 'male gaze' refers to the way the
+characters are portrayed on-screen as objects of desire rather than subjects.
+In this article, we introduce a novel video-interpretation task, to detect
+character objectification in films. The purpose is to reveal and quantify the
+usage of complex temporal patterns operated in cinema to produce the cognitive
+perception of objectification. We introduce the ObyGaze12 dataset, made of 1914
+movie clips densely annotated by experts for objectification concepts
+identified in film studies and psychology. We evaluate recent vision models,
+show the feasibility of the task and where the challenges remain with concept
+bottleneck models. Our new dataset and code are made available to the
+community.",cs.CV,['cs.CV']
+Bilateral Event Mining and Complementary for Event Stream Super-Resolution,Zhilin Huang · Quanmin Liang · Yijie Yu · Chujun Qin · Xiawu Zheng · Kai Huang · Zikun Zhou · Wenming Yang, ,https://arxiv.org/abs/2405.10037v1,,2405.10037v1.pdf,Bilateral Event Mining and Complementary for Event Stream Super-Resolution,"Event Stream Super-Resolution (ESR) aims to address the challenge of
+insufficient spatial resolution in event streams, which holds great
+significance for the application of event cameras in complex scenarios.
+Previous works for ESR often process positive and negative events in a mixed
+paradigm. This paradigm limits their ability to effectively model the unique
+characteristics of each event and mutually refine each other by considering
+their correlations. In this paper, we propose a bilateral event mining and
+complementary network (BMCNet) to fully leverage the potential of each event
+and capture the shared information to complement each other simultaneously.
+Specifically, we resort to a two-stream network to accomplish comprehensive
+mining of each type of events individually. To facilitate the exchange of
+information between two streams, we propose a bilateral information exchange
+(BIE) module. This module is layer-wisely embedded between two streams,
+enabling the effective propagation of hierarchical global information while
+alleviating the impact of invalid information brought by inherent
+characteristics of events. The experimental results demonstrate that our
+approach outperforms the previous state-of-the-art methods in ESR, achieving
+performance improvements of over 11\% on both real and synthetic datasets.
+Moreover, our method significantly enhances the performance of event-based
+downstream tasks such as object recognition and video reconstruction. Our code
+is available at https://github.com/Lqm26/BMCNet-ESR.",cs.CV,['cs.CV']
+Instance-Aware Group Quantization for Vision Transformers,Jaehyeon Moon · Dohyung Kim · Jun Yong Cheon · Bumsub Ham,https://cvlab.yonsei.ac.kr/projects/IGQ-ViT/,https://arxiv.org/abs/2404.00928,,2404.00928.pdf,Instance-Aware Group Quantization for Vision Transformers,"Post-training quantization (PTQ) is an efficient model compression technique
+that quantizes a pretrained full-precision model using only a small calibration
+set of unlabeled samples without retraining. PTQ methods for convolutional
+neural networks (CNNs) provide quantization results comparable to
+full-precision counterparts. Directly applying them to vision transformers
+(ViTs), however, incurs severe performance degradation, mainly due to the
+differences in architectures between CNNs and ViTs. In particular, the
+distribution of activations for each channel vary drastically according to
+input instances, making PTQ methods for CNNs inappropriate for ViTs. To address
+this, we introduce instance-aware group quantization for ViTs (IGQ-ViT). To
+this end, we propose to split the channels of activation maps into multiple
+groups dynamically for each input instance, such that activations within each
+group share similar statistical properties. We also extend our scheme to
+quantize softmax attentions across tokens. In addition, the number of groups
+for each layer is adjusted to minimize the discrepancies between predictions
+from quantized and full-precision models, under a bit-operation (BOP)
+constraint. We show extensive experimental results on image classification,
+object detection, and instance segmentation, with various transformer
+architectures, demonstrating the effectiveness of our approach.",cs.CV,"['cs.CV', 'cs.LG']"
+MLP Can Be A Good Transformer Learner,Sihao Lin · Pumeng Lyu · Dongrui Liu · Tao Tang · Xiaodan Liang · Andy Song · Xiaojun Chang, ,https://arxiv.org/abs/2404.05657,,2404.05657.pdf,MLP Can Be A Good Transformer Learner,"Self-attention mechanism is the key of the Transformer but often criticized
+for its computation demands. Previous token pruning works motivate their
+methods from the view of computation redundancy but still need to load the full
+network and require same memory costs. This paper introduces a novel strategy
+that simplifies vision transformers and reduces computational load through the
+selective removal of non-essential attention layers, guided by entropy
+considerations. We identify that regarding the attention layer in bottom
+blocks, their subsequent MLP layers, i.e. two feed-forward layers, can elicit
+the same entropy quantity. Meanwhile, the accompanied MLPs are under-exploited
+since they exhibit smaller feature entropy compared to those MLPs in the top
+blocks. Therefore, we propose to integrate the uninformative attention layers
+into their subsequent counterparts by degenerating them into identical mapping,
+yielding only MLP in certain transformer blocks. Experimental results on
+ImageNet-1k show that the proposed method can remove 40% attention layer of
+DeiT-B, improving throughput and memory bound without performance compromise.
+Code is available at https://github.com/sihaoevery/lambda_vit.",cs.CV,['cs.CV']
+BiPer: Binary Neural Networks using a Periodic Function,Edwin Vargas · Claudia Correa · Carlos Hinojosa · Henry Arguello, ,https://arxiv.org/abs/2404.01278,,2404.01278.pdf,BiPer: Binary Neural Networks using a Periodic Function,"Quantized neural networks employ reduced precision representations for both
+weights and activations. This quantization process significantly reduces the
+memory requirements and computational complexity of the network. Binary Neural
+Networks (BNNs) are the extreme quantization case, representing values with
+just one bit. Since the sign function is typically used to map real values to
+binary values, smooth approximations are introduced to mimic the gradients
+during error backpropagation. Thus, the mismatch between the forward and
+backward models corrupts the direction of the gradient, causing training
+inconsistency problems and performance degradation. In contrast to current BNN
+approaches, we propose to employ a binary periodic (BiPer) function during
+binarization. Specifically, we use a square wave for the forward pass to obtain
+the binary values and employ the trigonometric sine function with the same
+period of the square wave as a differentiable surrogate during the backward
+pass. We demonstrate that this approach can control the quantization error by
+using the frequency of the periodic function and improves network performance.
+Extensive experiments validate the effectiveness of BiPer in benchmark datasets
+and network architectures, with improvements of up to 1% and 0.69% with respect
+to state-of-the-art methods in the classification task over CIFAR-10 and
+ImageNet, respectively. Our code is publicly available at
+https://github.com/edmav4/BiPer.",cs.CV,['cs.CV']
+Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations,Daan de Geus · Gijs Dubbelman,https://www.tue-mps.org/tapps/,https://arxiv.org/abs/2311.18618,,2311.18618.pdf,JPPF: Multi-task Fusion for Consistent Panoptic-Part Segmentation,"Part-aware panoptic segmentation is a problem of computer vision that aims to
+provide a semantic understanding of the scene at multiple levels of
+granularity. More precisely, semantic areas, object instances, and semantic
+parts are predicted simultaneously. In this paper, we present our Joint
+Panoptic Part Fusion (JPPF) that combines the three individual segmentations
+effectively to obtain a panoptic-part segmentation. Two aspects are of utmost
+importance for this: First, a unified model for the three problems is desired
+that allows for mutually improved and consistent representation learning.
+Second, balancing the combination so that it gives equal importance to all
+individual results during fusion. Our proposed JPPF is parameter-free and
+dynamically balances its input. The method is evaluated and compared on the
+Cityscapes Panoptic Parts (CPP) and Pascal Panoptic Parts (PPP) datasets in
+terms of PartPQ and Part-Whole Quality (PWQ). In extensive experiments, we
+verify the importance of our fair fusion, highlight its most significant impact
+for areas that can be further segmented into parts, and demonstrate the
+generalization capabilities of our design without fine-tuning on 5 additional
+datasets.",cs.CV,['cs.CV']
+CaKDP: Category-aware Knowledge Distillation and Pruning Framework for Lightweight 3D Object Detection,Haonan Zhang · Longjun Liu · Yuqi Huang · YangZhao · Xinyu Lei · Bihan Wen, ,,https://github.com/zhnxjtu/CaKDP,,,,,nan
+Bilateral Propagation Network for Depth Completion,Jie Tang · Fei-Peng Tian · Boshi An · Jian Li · Ping Tan, ,https://arxiv.org/abs/2403.11270,,2403.11270.pdf,Bilateral Propagation Network for Depth Completion,"Depth completion aims to derive a dense depth map from sparse depth
+measurements with a synchronized color image. Current state-of-the-art (SOTA)
+methods are predominantly propagation-based, which work as an iterative
+refinement on the initial estimated dense depth. However, the initial depth
+estimations mostly result from direct applications of convolutional layers on
+the sparse depth map. In this paper, we present a Bilateral Propagation Network
+(BP-Net), that propagates depth at the earliest stage to avoid directly
+convolving on sparse data. Specifically, our approach propagates the target
+depth from nearby depth measurements via a non-linear model, whose coefficients
+are generated through a multi-layer perceptron conditioned on both
+\emph{radiometric difference} and \emph{spatial distance}. By integrating
+bilateral propagation with multi-modal fusion and depth refinement in a
+multi-scale framework, our BP-Net demonstrates outstanding performance on both
+indoor and outdoor scenes. It achieves SOTA on the NYUv2 dataset and ranks 1st
+on the KITTI depth completion benchmark at the time of submission. Experimental
+results not only show the effectiveness of bilateral propagation but also
+emphasize the significance of early-stage propagation in contrast to the
+refinement stage. Our code and trained models will be available on the project
+page.",cs.CV,['cs.CV']
+SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection,Peng Qi · Zehong Yan · Wynne Hsu · Mong Li Lee,https://pengqi.site/Sniffer/,https://arxiv.org/abs/2403.03170,,2403.03170.pdf,SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection,"Misinformation is a prevalent societal issue due to its potential high risks.
+Out-of-context (OOC) misinformation, where authentic images are repurposed with
+false text, is one of the easiest and most effective ways to mislead audiences.
+Current methods focus on assessing image-text consistency but lack convincing
+explanations for their judgments, which is essential for debunking
+misinformation. While Multimodal Large Language Models (MLLMs) have rich
+knowledge and innate capability for visual reasoning and explanation
+generation, they still lack sophistication in understanding and discovering the
+subtle crossmodal differences. In this paper, we introduce SNIFFER, a novel
+multimodal large language model specifically engineered for OOC misinformation
+detection and explanation. SNIFFER employs two-stage instruction tuning on
+InstructBLIP. The first stage refines the model's concept alignment of generic
+objects with news-domain entities and the second stage leverages language-only
+GPT-4 generated OOC-specific instruction data to fine-tune the model's
+discriminatory powers. Enhanced by external tools and retrieval, SNIFFER not
+only detects inconsistencies between text and image but also utilizes external
+knowledge for contextual verification. Our experiments show that SNIFFER
+surpasses the original MLLM by over 40% and outperforms state-of-the-art
+methods in detection accuracy. SNIFFER also provides accurate and persuasive
+explanations as validated by quantitative and human evaluations.",cs.MM,"['cs.MM', 'cs.AI', 'cs.CL', 'cs.CV', 'cs.CY']"
+Semantic-aware SAM for Point-Prompted Instance Segmentation,Zhaoyang Wei · Pengfei Chen · Xuehui Yu · Guorong Li · Jianbin Jiao · Zhenjun Han, ,https://arxiv.org/abs/2312.15895,,2312.15895.pdf,Semantic-aware SAM for Point-Prompted Instance Segmentation,"Single-point annotation in visual tasks, with the goal of minimizing
+labelling costs, is becoming increasingly prominent in research. Recently,
+visual foundation models, such as Segment Anything (SAM), have gained
+widespread usage due to their robust zero-shot capabilities and exceptional
+annotation performance. However, SAM's class-agnostic output and high
+confidence in local segmentation introduce 'semantic ambiguity', posing a
+challenge for precise category-specific segmentation. In this paper, we
+introduce a cost-effective category-specific segmenter using SAM. To tackle
+this challenge, we have devised a Semantic-Aware Instance Segmentation Network
+(SAPNet) that integrates Multiple Instance Learning (MIL) with matching
+capability and SAM with point prompts. SAPNet strategically selects the most
+representative mask proposals generated by SAM to supervise segmentation, with
+a specific focus on object category information. Moreover, we introduce the
+Point Distance Guidance and Box Mining Strategy to mitigate inherent
+challenges: 'group' and 'local' issues in weakly supervised segmentation. These
+strategies serve to further enhance the overall segmentation performance. The
+experimental results on Pascal VOC and COCO demonstrate the promising
+performance of our proposed SAPNet, emphasizing its semantic matching
+capabilities and its potential to advance point-prompted instance segmentation.
+The code will be made publicly available.",cs.CV,['cs.CV']
+Loopy-SLAM: Dense Neural SLAM with Loop Closures,Lorenzo Liso · Erik Sandström · Vladimir Yugay · Luc Van Gool · Martin R. Oswald, ,https://arxiv.org/abs/2402.09944,,2402.09944.pdf,Loopy-SLAM: Dense Neural SLAM with Loop Closures,"Neural RGBD SLAM techniques have shown promise in dense Simultaneous
+Localization And Mapping (SLAM), yet face challenges such as error accumulation
+during camera tracking resulting in distorted maps. In response, we introduce
+Loopy-SLAM that globally optimizes poses and the dense 3D model. We use
+frame-to-model tracking using a data-driven point-based submap generation
+method and trigger loop closures online by performing global place recognition.
+Robust pose graph optimization is used to rigidly align the local submaps. As
+our representation is point based, map corrections can be performed efficiently
+without the need to store the entire history of input frames used for mapping
+as typically required by methods employing a grid based mapping structure.
+Evaluation on the synthetic Replica and real-world TUM-RGBD and ScanNet
+datasets demonstrate competitive or superior performance in tracking, mapping,
+and rendering accuracy when compared to existing dense neural RGBD SLAM
+methods. Project page: notchla.github.io/Loopy-SLAM.",cs.CV,['cs.CV']
+Aligning Logits Generatively for Principled Black-Box Knowledge Distillation,Jing Ma · Xiang Xiang · Ke Wang · Yuchuan Wu · Yongbin Li, ,https://arxiv.org/abs/2403.01427,,,Logit Standardization in Knowledge Distillation,"Knowledge distillation involves transferring soft labels from a teacher to a
+student using a shared temperature-based softmax function. However, the
+assumption of a shared temperature between teacher and student implies a
+mandatory exact match between their logits in terms of logit range and
+variance. This side-effect limits the performance of student, considering the
+capacity discrepancy between them and the finding that the innate logit
+relations of teacher are sufficient for student to learn. To address this
+issue, we propose setting the temperature as the weighted standard deviation of
+logit and performing a plug-and-play Z-score pre-process of logit
+standardization before applying softmax and Kullback-Leibler divergence. Our
+pre-process enables student to focus on essential logit relations from teacher
+rather than requiring a magnitude match, and can improve the performance of
+existing logit-based distillation methods. We also show a typical case where
+the conventional setting of sharing temperature between teacher and student
+cannot reliably yield the authentic distillation evaluation; nonetheless, this
+challenge is successfully alleviated by our Z-score. We extensively evaluate
+our method for various student and teacher models on CIFAR-100 and ImageNet,
+showing its significant superiority. The vanilla knowledge distillation powered
+by our pre-process can achieve favorable performance against state-of-the-art
+methods, and other distillation variants can obtain considerable gain with the
+assistance of our pre-process.",cs.CV,['cs.CV']
+Grid Diffusion Models for Text-to-Video Generation,Taegyeong Lee · Soyeong Kwon · Taehwan Kim,https://taegyeong-lee.github.io/text2video,https://arxiv.org/abs/2404.00234v1,,2404.00234v1.pdf,Grid Diffusion Models for Text-to-Video Generation,"Recent advances in the diffusion models have significantly improved
+text-to-image generation. However, generating videos from text is a more
+challenging task than generating images from text, due to the much larger
+dataset and higher computational cost required. Most existing video generation
+methods use either a 3D U-Net architecture that considers the temporal
+dimension or autoregressive generation. These methods require large datasets
+and are limited in terms of computational costs compared to text-to-image
+generation. To tackle these challenges, we propose a simple but effective novel
+grid diffusion for text-to-video generation without temporal dimension in
+architecture and a large text-video paired dataset. We can generate a
+high-quality video using a fixed amount of GPU memory regardless of the number
+of frames by representing the video as a grid image. Additionally, since our
+method reduces the dimensions of the video to the dimensions of the image,
+various image-based methods can be applied to videos, such as text-guided video
+manipulation from image manipulation. Our proposed method outperforms the
+existing methods in both quantitative and qualitative evaluations,
+demonstrating the suitability of our model for real-world video generation.",cs.CV,['cs.CV']
+Wonder3D: Single Image to 3D using Cross-Domain Diffusion,Xiaoxiao Long · Yuan-Chen Guo · Cheng Lin · Yuan Liu · Zhiyang Dou · Lingjie Liu · Yuexin Ma · Song-Hai Zhang · Marc Habermann · Christian Theobalt · Wenping Wang, ,https://arxiv.org/abs/2310.15008,,2310.15008.pdf,Wonder3D: Single Image to 3D using Cross-Domain Diffusion,"In this work, we introduce Wonder3D, a novel method for efficiently
+generating high-fidelity textured meshes from single-view images.Recent methods
+based on Score Distillation Sampling (SDS) have shown the potential to recover
+3D geometry from 2D diffusion priors, but they typically suffer from
+time-consuming per-shape optimization and inconsistent geometry. In contrast,
+certain works directly produce 3D information via fast network inferences, but
+their results are often of low quality and lack geometric details. To
+holistically improve the quality, consistency, and efficiency of image-to-3D
+tasks, we propose a cross-domain diffusion model that generates multi-view
+normal maps and the corresponding color images. To ensure consistency, we
+employ a multi-view cross-domain attention mechanism that facilitates
+information exchange across views and modalities. Lastly, we introduce a
+geometry-aware normal fusion algorithm that extracts high-quality surfaces from
+the multi-view 2D representations. Our extensive evaluations demonstrate that
+our method achieves high-quality reconstruction results, robust generalization,
+and reasonably good efficiency compared to prior works.",cs.CV,['cs.CV']
+Towards High-fidelity Artistic Image Vectorization via Texture-Encapsulated Shape Parameterization,Ye Chen · Bingbing Ni · Jinfan Liu · Xiaoyang Huang · Xuanhong Chen, ,https://arxiv.org/abs/2308.13628,,2308.13628.pdf,HiFiHR: Enhancing 3D Hand Reconstruction from a Single Image via High-Fidelity Texture,"We present HiFiHR, a high-fidelity hand reconstruction approach that utilizes
+render-and-compare in the learning-based framework from a single image, capable
+of generating visually plausible and accurate 3D hand meshes while recovering
+realistic textures. Our method achieves superior texture reconstruction by
+employing a parametric hand model with predefined texture assets, and by
+establishing a texture reconstruction consistency between the rendered and
+input images during training. Moreover, based on pretraining the network on an
+annotated dataset, we apply varying degrees of supervision using our pipeline,
+i.e., self-supervision, weak supervision, and full supervision, and discuss the
+various levels of contributions of the learned high-fidelity textures in
+enhancing hand pose and shape estimation. Experimental results on public
+benchmarks including FreiHAND and HO-3D demonstrate that our method outperforms
+the state-of-the-art hand reconstruction methods in texture reconstruction
+quality while maintaining comparable accuracy in pose and shape estimation. Our
+code is available at https://github.com/viridityzhu/HiFiHR.",cs.CV,"['cs.CV', 'cs.AI']"
+Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks,Yuhao Liu · Zhanghan Ke · Fang Liu · Nanxuan Zhao · Rynson W.H. Lau, ,https://arxiv.org/abs/2403.00644,,2403.00644.pdf,Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks,"Diffusion models trained on large-scale datasets have achieved remarkable
+progress in image synthesis. However, due to the randomness in the diffusion
+process, they often struggle with handling diverse low-level tasks that require
+details preservation. To overcome this limitation, we present a new Diff-Plugin
+framework to enable a single pre-trained diffusion model to generate
+high-fidelity results across a variety of low-level tasks. Specifically, we
+first propose a lightweight Task-Plugin module with a dual branch design to
+provide task-specific priors, guiding the diffusion process in preserving image
+content. We then propose a Plugin-Selector that can automatically select
+different Task-Plugins based on the text instruction, allowing users to edit
+images by indicating multiple low-level tasks with natural language. We conduct
+extensive experiments on 8 low-level vision tasks. The results demonstrate the
+superiority of Diff-Plugin over existing methods, particularly in real-world
+scenarios. Our ablations further validate that Diff-Plugin is stable,
+schedulable, and supports robust training across different dataset sizes.",cs.CV,['cs.CV']
+Hunting Attributes:  Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation,Feilong Tang · Zhongxing Xu · Zhaojun QU · Wei Feng · xingjian jiang · Zongyuan Ge, ,https://arxiv.org/abs/2403.07630,,2403.07630.pdf,Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation,"Recent weakly supervised semantic segmentation (WSSS) methods strive to
+incorporate contextual knowledge to improve the completeness of class
+activation maps (CAM). In this work, we argue that the knowledge bias between
+instances and contexts affects the capability of the prototype to sufficiently
+understand instance semantics. Inspired by prototype learning theory, we
+propose leveraging prototype awareness to capture diverse and fine-grained
+feature attributes of instances. The hypothesis is that contextual prototypes
+might erroneously activate similar and frequently co-occurring object
+categories due to this knowledge bias. Therefore, we propose to enhance the
+prototype representation ability by mitigating the bias to better capture
+spatial coverage in semantic object regions. With this goal, we present a
+Context Prototype-Aware Learning (CPAL) strategy, which leverages semantic
+context to enrich instance comprehension. The core of this method is to
+accurately capture intra-class variations in object features through
+context-aware prototypes, facilitating the adaptation to the semantic
+attributes of various instances. We design feature distribution alignment to
+optimize prototype awareness, aligning instance feature distributions with
+dense features. In addition, a unified training framework is proposed to
+combine label-guided classification supervision and prototypes-guided
+self-supervision. Experimental results on PASCAL VOC 2012 and MS COCO 2014 show
+that CPAL significantly improves off-the-shelf methods and achieves
+state-of-the-art performance. The project is available at
+https://github.com/Barrett-python/CPAL.",cs.CV,"['cs.CV', 'cs.AI']"
+Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation,Luca Barsellotti · Roberto Amoroso · Marcella Cornia · Lorenzo Baraldi · Rita Cucchiara,https://aimagelab.github.io/freeda/,https://arxiv.org/abs/2404.06542,,2404.06542.pdf,Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation,"Open-vocabulary semantic segmentation aims at segmenting arbitrary categories
+expressed in textual form. Previous works have trained over large amounts of
+image-caption pairs to enforce pixel-level multimodal alignments. However,
+captions provide global information about the semantics of a given image but
+lack direct localization of individual concepts. Further, training on
+large-scale datasets inevitably brings significant computational costs. In this
+paper, we propose FreeDA, a training-free diffusion-augmented method for
+open-vocabulary semantic segmentation, which leverages the ability of diffusion
+models to visually localize generated concepts and local-global similarities to
+match class-agnostic regions with semantic classes. Our approach involves an
+offline stage in which textual-visual reference embeddings are collected,
+starting from a large set of captions and leveraging visual and semantic
+contexts. At test time, these are queried to support the visual matching
+process, which is carried out by jointly considering class-agnostic regions and
+global semantic similarities. Extensive analyses demonstrate that FreeDA
+achieves state-of-the-art performance on five datasets, surpassing previous
+methods by more than 7.0 average points in terms of mIoU and without requiring
+any training.",cs.CV,['cs.CV']
+TULIP: Multi-camera 3D Precision Assessment of Parkinson's Disease,Kyungdo Kim · Sihan Lyu · Sneha Mantri · Timothy DUNN, ,,https://www.nature.com/articles/s41746-023-00905-9,,,,,nan
+ControlRoom3D: Room Generation using Semantic Controls,Jonas Schult · Sam Tsai · Lukas Höllein · Bichen Wu · Jialiang Wang · Chih-Yao Ma · Kunpeng Li · Xiaofang Wang · Felix Wimbauer · Zijian He · Peizhao Zhang · Bastian Leibe · Peter Vajda · Ji Hou,https://jonasschult.github.io/ControlRoom3D/,https://arxiv.org/abs/2312.05208,,2312.05208.pdf,ControlRoom3D: Room Generation using Semantic Proxy Rooms,"Manually creating 3D environments for AR/VR applications is a complex process
+requiring expert knowledge in 3D modeling software. Pioneering works facilitate
+this process by generating room meshes conditioned on textual style
+descriptions. Yet, many of these automatically generated 3D meshes do not
+adhere to typical room layouts, compromising their plausibility, e.g., by
+placing several beds in one bedroom. To address these challenges, we present
+ControlRoom3D, a novel method to generate high-quality room meshes. Central to
+our approach is a user-defined 3D semantic proxy room that outlines a rough
+room layout based on semantic bounding boxes and a textual description of the
+overall room style. Our key insight is that when rendered to 2D, this 3D
+representation provides valuable geometric and semantic information to control
+powerful 2D models to generate 3D consistent textures and geometry that aligns
+well with the proxy room. Backed up by an extensive study including
+quantitative metrics and qualitative user evaluations, our method generates
+diverse and globally plausible 3D room meshes, thus empowering users to design
+3D rooms effortlessly without specialized knowledge.",cs.CV,['cs.CV']
+Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging,Bhargav Ghanekar · Salman Siddique Khan · Pranav Sharma · Shreyas Singh · Vivek Boominathan · Kaushik Mitra · Ashok Veeraraghavan,https://shadowfax11.github.io/cads/,https://arxiv.org/abs/2402.18102,,2402.18102.pdf,Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging,"Passive, compact, single-shot 3D sensing is useful in many application areas
+such as microscopy, medical imaging, surgical navigation, and autonomous
+driving where form factor, time, and power constraints can exist. Obtaining
+RGB-D scene information over a short imaging distance, in an ultra-compact form
+factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP)
+sensors are a potential solution to achieve the same. DP sensors collect light
+rays from two different halves of the lens in two interleaved pixel arrays,
+thus capturing two slightly different views of the scene, like a stereo camera
+system. However, imaging with a DP sensor implies that the defocus blur size is
+directly proportional to the disparity seen between the views. This creates a
+trade-off between disparity estimation vs. deblurring accuracy. To improve this
+trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which
+we use a coded aperture in the imaging lens along with a DP sensor. In our
+approach, we jointly learn an optimal coded pattern and the reconstruction
+algorithm in an end-to-end optimization setting. Our resulting CADS imaging
+system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates
+and 5-6% in depth estimation quality over naive DP sensing for a wide range of
+aperture settings. Furthermore, we build the proposed CADS prototypes for DSLR
+photography settings and in an endoscope and a dermoscope form factor. Our
+novel coded dual-pixel sensing approach demonstrates accurate RGB-D
+reconstruction results in simulations and real-world experiments in a passive,
+snapshot, and compact manner.",eess.IV,"['eess.IV', 'cs.CV']"
+Real-time 3D-aware Portrait Video Relighting,Ziqi Cai · Kaiwen Jiang · Shu-Yu Chen · Yu-Kun Lai · Hongbo Fu · Boxin Shi · Lin Gao,http://geometrylearning.com/VideoRelighting/,https://arxiv.org/html/2402.14000v1,,2402.14000v1.pdf,Real-time 3D-aware Portrait Editing from a Single Image,"This work presents 3DPE, a practical tool that can efficiently edit a face
+image following given prompts, like reference images or text descriptions, in
+the 3D-aware manner. To this end, a lightweight module is distilled from a 3D
+portrait generator and a text-to-image model, which provide prior knowledge of
+face geometry and open-vocabulary editing capability, respectively. Such a
+design brings two compelling advantages over existing approaches. First, our
+system achieves real-time editing with a feedforward network (i.e., ~0.04s per
+image), over 100x faster than the second competitor. Second, thanks to the
+powerful priors, our module could focus on the learning of editing-related
+variations, such that it manages to handle various types of editing
+simultaneously in the training phase and further supports fast adaptation to
+user-specified novel types of editing during inference (e.g., with ~5min
+fine-tuning per case). The code, the model, and the interface will be made
+publicly available to facilitate future research.",cs.CV,['cs.CV']
+DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes,Xiaoyu Zhou · Zhiwei Lin · Xiaojun Shan · Yongtao Wang · Deqing Sun · Ming-Hsuan Yang, ,https://arxiv.org/abs/2312.07920,,2312.07920.pdf,DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes,"We present DrivingGaussian, an efficient and effective framework for
+surrounding dynamic autonomous driving scenes. For complex scenes with moving
+objects, we first sequentially and progressively model the static background of
+the entire scene with incremental static 3D Gaussians. We then leverage a
+composite dynamic Gaussian graph to handle multiple moving objects,
+individually reconstructing each object and restoring their accurate positions
+and occlusion relationships within the scene. We further use a LiDAR prior for
+Gaussian Splatting to reconstruct scenes with greater details and maintain
+panoramic consistency. DrivingGaussian outperforms existing methods in dynamic
+driving scene reconstruction and enables photorealistic surround-view synthesis
+with high-fidelity and multi-camera consistency. Our project page is at:
+https://github.com/VDIGPKU/DrivingGaussian.",cs.CV,['cs.CV']
+A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition,Yusheng Dai · HangChen · Jun Du · Ruoyu Wang · shihao chen · Haotian Wang · Chin-Hui Lee,https://github.com/dalision/ModalBiasAVSR,https://arxiv.org/abs/2403.04245,,2403.04245.pdf,A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition,"Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to
+be sensitive to missing video frames, performing even worse than
+single-modality models. While applying the dropout technique to the video
+modality enhances robustness to missing frames, it simultaneously results in a
+performance loss when dealing with complete data input. In this paper, we
+investigate this contrasting phenomenon from the perspective of modality bias
+and reveal that an excessive modality bias on the audio caused by dropout is
+the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH)
+to systematically describe the relationship between modality bias and
+robustness against missing modality in multimodal systems. Building on these
+findings, we propose a novel Multimodal Distribution Approximation with
+Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio
+modality and to maintain performance and robustness simultaneously. Finally, to
+address an entirely missing modality, we adopt adapters to dynamically switch
+decision strategies. The effectiveness of our proposed approach is evaluated
+and validated through a series of comprehensive experiments using the MISP2021
+and MISP2022 datasets. Our code is available at
+https://github.com/dalision/ModalBiasAVSR",cs.SD,"['cs.SD', 'cs.CV', 'cs.LG', 'cs.MM', 'eess.AS']"
+GPLD3D: Latent Diffusion of 3D Shape Generative Models by Enforcing Geometric and Physical Priors,Yuan Dong · Qi Zuo · Xiaodong Gu · Weihao Yuan · zhengyi zhao · Zilong Dong · Liefeng Bo · Qixing Huang, ,https://arxiv.org/abs/2401.17603,,2401.17603.pdf,Topology-Aware Latent Diffusion for 3D Shape Generation,"We introduce a new generative model that combines latent diffusion with
+persistent homology to create 3D shapes with high diversity, with a special
+emphasis on their topological characteristics. Our method involves representing
+3D shapes as implicit fields, then employing persistent homology to extract
+topological features, including Betti numbers and persistence diagrams. The
+shape generation process consists of two steps. Initially, we employ a
+transformer-based autoencoding module to embed the implicit representation of
+each 3D shape into a set of latent vectors. Subsequently, we navigate through
+the learned latent space via a diffusion model. By strategically incorporating
+topological features into the diffusion process, our generative module is able
+to produce a richer variety of 3D shapes with different topological structures.
+Furthermore, our framework is flexible, supporting generation tasks constrained
+by a variety of inputs, including sparse and partial point clouds, as well as
+sketches. By modifying the persistence diagrams, we can alter the topology of
+the shapes generated from these input modalities.",cs.CV,"['cs.CV', 'I.3.5; I.2.10']"
+Shallow-Deep Collaborative Learning for Unsupervised Visible-Infrared Person Re-Identification,Bin Yang · Jun Chen · Mang Ye, ,,https://dl.acm.org/doi/10.1145/3581783.3612077,,,,,nan
+Benchmarking Implicit Neural Representation and Geometric Rendering in Real-Time RGB-D SLAM,"Tongyan Hua · Addison, Lin Wang",https://vlis2022.github.io/nerf-slam-benchmark/,https://arxiv.org/abs/2403.19473,,2403.19473.pdf,Benchmarking Implicit Neural Representation and Geometric Rendering in Real-Time RGB-D SLAM,"Implicit neural representation (INR), in combination with geometric
+rendering, has recently been employed in real-time dense RGB-D SLAM. Despite
+active research endeavors being made, there lacks a unified protocol for fair
+evaluation, impeding the evolution of this area. In this work, we establish, to
+our knowledge, the first open-source benchmark framework to evaluate the
+performance of a wide spectrum of commonly used INRs and rendering functions
+for mapping and localization. The goal of our benchmark is to 1) gain an
+intuition of how different INRs and rendering functions impact mapping and
+localization and 2) establish a unified evaluation protocol w.r.t. the design
+choices that may impact the mapping and localization. With the framework, we
+conduct a large suite of experiments, offering various insights in choosing the
+INRs and geometric rendering functions: for example, the dense feature grid
+outperforms other INRs (e.g. tri-plane and hash grid), even when geometric and
+color features are jointly encoded for memory efficiency. To extend the
+findings into the practical scenario, a hybrid encoding strategy is proposed to
+bring the best of the accuracy and completion from the grid-based and
+decomposition-based INRs. We further propose explicit hybrid encoding for
+high-fidelity dense grid mapping to comply with the RGB-D SLAM system that puts
+the premise on robustness and computation efficiency.",cs.CV,['cs.CV']
+Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion,"Hao Ai · Addison, Lin Wang", ,http://export.arxiv.org/abs/2403.16376,,2403.16376.pdf,Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion,"360 depth estimation has recently received great attention for 3D
+reconstruction owing to its omnidirectional field of view (FoV). Recent
+approaches are predominantly focused on cross-projection fusion with
+geometry-based re-projection: they fuse 360 images with equirectangular
+projection (ERP) and another projection type, e.g., cubemap projection to
+estimate depth with the ERP format. However, these methods suffer from 1)
+limited local receptive fields, making it hardly possible to capture large FoV
+scenes, and 2) prohibitive computational cost, caused by the complex
+cross-projection fusion module design. In this paper, we propose Elite360D, a
+novel framework that inputs the ERP image and icosahedron projection (ICOSAP)
+point set, which is undistorted and spatially continuous. Elite360D is superior
+in its capacity in learning a representation from a local-with-global
+perspective. With a flexible ERP image encoder, it includes an ICOSAP point
+encoder, and a Bi-projection Bi-attention Fusion (B2F) module (totally ~1M
+parameters). Specifically, the ERP image encoder can take various perspective
+image-trained backbones (e.g., ResNet, Transformer) to extract local features.
+The point encoder extracts the global features from the ICOSAP. Then, the B2F
+module captures the semantic- and distance-aware dependencies between each
+pixel of the ERP feature and the entire ICOSAP feature set. Without specific
+backbone design and obvious computational cost increase, Elite360D outperforms
+the prior arts on several benchmark datasets.",cs.CV,['cs.CV']
+EventDance: Unsupervised Cross-modal Source-free Adaptation for Event-based Object Recognition,"Xu Zheng · Addison, Lin Wang", ,https://arxiv.org/abs/2403.14082,,2403.14082.pdf,EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition,"In this paper, we make the first attempt at achieving the cross-modal (i.e.,
+image-to-events) adaptation for event-based object recognition without
+accessing any labeled source image data owning to privacy and commercial
+issues. Tackling this novel problem is non-trivial due to the novelty of event
+cameras and the distinct modality gap between images and events. In particular,
+as only the source model is available, a hurdle is how to extract the knowledge
+from the source model by only using the unlabeled target event data while
+achieving knowledge transfer. To this end, we propose a novel framework, dubbed
+EventDance for this unsupervised source-free cross-modal adaptation problem.
+Importantly, inspired by event-to-video reconstruction methods, we propose a
+reconstruction-based modality bridging (RMB) module, which reconstructs
+intensity frames from events in a self-supervised manner. This makes it
+possible to build up the surrogate images to extract the knowledge (i.e.,
+labels) from the source model. We then propose a multi-representation knowledge
+adaptation (MKA) module that transfers the knowledge to target models learning
+events with multiple representation types for fully exploring the
+spatiotemporal information of events. The two modules connecting the source and
+target models are mutually updated so as to achieve the best performance.
+Experiments on three benchmark datasets with two adaption settings show that
+EventDance is on par with prior methods utilizing the source data.",cs.CV,['cs.CV']
+GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation,"WEIMING ZHANG · Yexin Liu · Xu Zheng · Addison, Lin Wang", ,https://arxiv.org/abs/2403.16370,,2403.16370.pdf,GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation,"This paper tackles a novel yet challenging problem: how to transfer knowledge
+from the emerging Segment Anything Model (SAM) -- which reveals impressive
+zero-shot instance segmentation capacity -- to learn a compact panoramic
+semantic segmentation model, i.e., student, without requiring any labeled data.
+This poses considerable challenges due to SAM's inability to provide semantic
+labels and the large capacity gap between SAM and the student. To this end, we
+propose a novel framework, called GoodSAM, that introduces a teacher assistant
+(TA) to provide semantic information, integrated with SAM to generate ensemble
+logits to achieve knowledge transfer. Specifically, we propose a
+Distortion-Aware Rectification (DAR) module that first addresses the distortion
+problem of panoramic images by imposing prediction-level consistency and
+boundary enhancement. This subtly enhances TA's prediction capacity on
+panoramic images. DAR then incorporates a cross-task complementary fusion block
+to adaptively merge the predictions of SAM and TA to obtain more reliable
+ensemble logits. Moreover, we introduce a Multi-level Knowledge Adaptation
+(MKA) module to efficiently transfer the multi-level feature knowledge from TA
+and ensemble logits to learn a compact student model. Extensive experiments on
+two benchmarks show that our GoodSAM achieves a remarkable +3.75\% mIoU
+improvement over the state-of-the-art (SOTA) domain adaptation methods. Also,
+our most lightweight model achieves comparable performance to the SOTA methods
+with only 3.7M parameters.",cs.CV,['cs.CV']
+ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More,"Jiazhou Zhou · Xu Zheng · Yuanhuiyi Lyu · Addison, Lin Wang", ,https://arxiv.org/abs/2403.12534,,2403.12534.pdf,ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More,"Event cameras have recently been shown beneficial for practical vision tasks,
+such as action recognition, thanks to their high temporal resolution, power
+efficiency, and reduced privacy concerns. However, current research is hindered
+by 1) the difficulty in processing events because of their prolonged duration
+and dynamic actions with complex and ambiguous semantics and 2) the redundant
+action depiction of the event frame representation with fixed stacks. We find
+language naturally conveys abundant semantic information, rendering it
+stunningly superior in reducing semantic uncertainty. In light of this, we
+propose ExACT, a novel approach that, for the first time, tackles event-based
+action recognition from a cross-modal conceptualizing perspective. Our ExACT
+brings two technical contributions. Firstly, we propose an adaptive
+fine-grained event (AFE) representation to adaptively filter out the repeated
+events for the stationary objects while preserving dynamic ones. This subtly
+enhances the performance of ExACT without extra computational cost. Then, we
+propose a conceptual reasoning-based uncertainty estimation module, which
+simulates the recognition process to enrich the semantic representation. In
+particular, conceptual reasoning builds the temporal relation based on the
+action semantics, and uncertainty estimation tackles the semantic uncertainty
+of actions based on the distributional representation. Experiments show that
+our ExACT achieves superior recognition accuracy of 94.83%(+2.23%),
+90.10%(+37.47%) and 67.24% on PAF, HARDVS and our SeAct datasets respectively.",cs.CV,['cs.CV']
+"Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation","Xu Zheng · Pengyuan Zhou · ATHANASIOS · Addison, Lin Wang",https://vlislab22.github.io/360SFUDA/,https://arxiv.org/abs/2403.12505,,2403.12505.pdf,"Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation","This paper addresses an interesting yet challenging problem -- source-free
+unsupervised domain adaptation (SFUDA) for pinhole-to-panoramic semantic
+segmentation -- given only a pinhole image-trained model (i.e., source) and
+unlabeled panoramic images (i.e., target). Tackling this problem is nontrivial
+due to the semantic mismatches, style discrepancies, and inevitable distortion
+of panoramic images. To this end, we propose a novel method that utilizes
+Tangent Projection (TP) as it has less distortion and meanwhile slits the
+equirectangular projection (ERP) with a fixed FoV to mimic the pinhole images.
+Both projections are shown effective in extracting knowledge from the source
+model. However, the distinct projection discrepancies between source and target
+domains impede the direct knowledge transfer; thus, we propose a panoramic
+prototype adaptation module (PPAM) to integrate panoramic prototypes from the
+extracted knowledge for adaptation. We then impose the loss constraints on both
+predictions and prototypes and propose a cross-dual attention module (CDAM) at
+the feature level to better align the spatial and channel characteristics
+across the domains and projections. Both knowledge extraction and transfer
+processes are synchronously updated to reach the best performance. Extensive
+experiments on the synthetic and real-world benchmarks, including outdoor and
+indoor scenarios, demonstrate that our method achieves significantly better
+performance than prior SFUDA methods for pinhole-to-panoramic adaptation.",cs.CV,['cs.CV']
+UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All,"Yuanhuiyi Lyu · Xu Zheng · Jiazhou Zhou · Addison, Lin Wang", ,https://arxiv.org/abs/2405.16108,,2405.16108.pdf,OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All,"Research on multi-modal learning dominantly aligns the modalities in a
+unified space at training, and only a single one is taken for prediction at
+inference. However, for a real machine, e.g., a robot, sensors could be added
+or removed at any time. Thus, it is crucial to enable the machine to tackle the
+mismatch and unequal-scale problems of modality combinations between training
+and inference. In this paper, we tackle these problems from a new perspective:
+""Modalities Help Modalities"". Intuitively, we present OmniBind, a novel
+two-stage learning framework that can achieve any modality combinations and
+interaction. It involves teaching data-constrained, a.k.a, student, modalities
+to be aligned with the well-trained data-abundant, a.k.a, teacher, modalities.
+This subtly enables the adaptive fusion of any modalities to build a unified
+representation space for any combinations. Specifically, we propose Cross-modal
+Alignment Distillation (CAD) to address the unequal-scale problem between
+student and teacher modalities and effectively align student modalities into
+the teacher modalities' representation space in stage one. We then propose an
+Adaptive Fusion (AF) module to fuse any modality combinations and learn a
+unified representation space in stage two. To address the mismatch problem, we
+aggregate existing datasets and combine samples from different modalities by
+the same semantics. This way, we build the first dataset for training and
+evaluation that consists of teacher (image, text) and student (touch, thermal,
+event, point cloud, audio) modalities and enables omni-bind for any of them.
+Extensive experiments on the recognition task show performance gains over prior
+arts by an average of 4.05 % on the arbitrary modality combination setting. It
+also achieves state-of-the-art performance for a single modality, e.g., touch,
+with a 4.34 % gain.",cs.CV,['cs.CV']
+C3Net: Compound Conditioned ControlNet for Multimodal Content Generation,Juntao Zhang · Yuehuai LIU · Yu-Wing Tai · Chi-Keung Tang, ,https://arxiv.org/abs/2311.17951,,2311.17951.pdf,C3Net: Compound Conditioned ControlNet for Multimodal Content Generation,"We present Compound Conditioned ControlNet, C3Net, a novel generative neural
+architecture taking conditions from multiple modalities and synthesizing
+multimodal contents simultaneously (e.g., image, text, audio). C3Net adapts the
+ControlNet architecture to jointly train and make inferences on a
+production-ready diffusion model and its trainable copies. Specifically, C3Net
+first aligns the conditions from multi-modalities to the same semantic latent
+space using modality-specific encoders based on contrastive training. Then, it
+generates multimodal outputs based on the aligned latent space, whose semantic
+information is combined using a ControlNet-like architecture called Control
+C3-UNet. Correspondingly, with this system design, our model offers an improved
+solution for joint-modality generation through learning and explaining
+multimodal conditions instead of simply taking linear interpolations on the
+latent space. Meanwhile, as we align conditions to a unified latent space,
+C3Net only requires one trainable Control C3-UNet to work on multimodal
+semantic information. Furthermore, our model employs unimodal pretraining on
+the condition alignment stage, outperforming the non-pretrained alignment even
+on relatively scarce training data and thus demonstrating high-quality compound
+condition generation. We contribute the first high-quality tri-modal validation
+set to validate quantitatively that C3Net outperforms or is on par with first
+and contemporary state-of-the-art multimodal generation. Our codes and
+tri-modal dataset will be released.",cs.LG,['cs.LG']
+Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach,"Guoqiang Liang · Kanghao Chen · Hangyu Li · Yunfan Lu · Addison, Lin Wang",https://vlislab22.github.io/eg-lowlight/.,https://arxiv.org/abs/2404.00834v1,,2404.00834v1.pdf,Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach,"Event camera has recently received much attention for low-light image
+enhancement (LIE) thanks to their distinct advantages, such as high dynamic
+range. However, current research is prohibitively restricted by the lack of
+large-scale, real-world, and spatial-temporally aligned event-image datasets.
+To this end, we propose a real-world (indoor and outdoor) dataset comprising
+over 30K pairs of images and events under both low and normal illumination
+conditions. To achieve this, we utilize a robotic arm that traces a consistent
+non-linear trajectory to curate the dataset with spatial alignment precision
+under 0.03mm. We then introduce a matching alignment strategy, rendering 90% of
+our dataset with errors less than 0.01s. Based on the dataset, we propose a
+novel event-guided LIE approach, called EvLight, towards robust performance in
+real-world low-light scenes. Specifically, we first design the multi-scale
+holistic fusion branch to extract holistic structural and textural information
+from both events and images. To ensure robustness against variations in the
+regional illumination and noise, we then introduce a Signal-to-Noise-Ratio
+(SNR)-guided regional feature selection to selectively fuse features of images
+from regions with high SNR and enhance those with low SNR by extracting
+regional structure information from events. Extensive experiments on our
+dataset and the synthetic SDSD dataset demonstrate our EvLight significantly
+surpasses the frame-based methods. Code and datasets are available at
+https://vlislab22.github.io/eg-lowlight/.",cs.CV,['cs.CV']
+Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning,Yixiong Zou · Yicong Liu · Yiman Hu · Yuhua Li · Ruixuan Li, ,https://arxiv.org/abs/2403.00567,,2403.00567.pdf,Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning,"Cross-domain few-shot learning (CDFSL) aims to acquire knowledge from limited
+training data in the target domain by leveraging prior knowledge transferred
+from source domains with abundant training samples. CDFSL faces challenges in
+transferring knowledge across dissimilar domains and fine-tuning models with
+limited training data. To address these challenges, we initially extend the
+analysis of loss landscapes from the parameter space to the representation
+space, which allows us to simultaneously interpret the transferring and
+fine-tuning difficulties of CDFSL models. We observe that sharp minima in the
+loss landscapes of the representation space result in representations that are
+hard to transfer and fine-tune. Moreover, existing flatness-based methods have
+limited generalization ability due to their short-range flatness. To enhance
+the transferability and facilitate fine-tuning, we introduce a simple yet
+effective approach to achieve long-range flattening of the minima in the loss
+landscape. This approach considers representations that are differently
+normalized as minima in the loss landscape and flattens the high-loss region in
+the middle by randomly sampling interpolated representations. We implement this
+method as a new normalization layer that replaces the original one in both CNNs
+and ViTs. This layer is simple and lightweight, introducing only a minimal
+number of additional parameters. Experimental results on 8 datasets demonstrate
+that our approach outperforms state-of-the-art methods in terms of average
+accuracy. Moreover, our method achieves performance improvements of up to 9\%
+compared to the current best approaches on individual datasets. Our code will
+be released.",cs.CV,"['cs.CV', 'cs.AI']"
+PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization,Xu Peng · Junwei Zhu · Boyuan Jiang · Ying Tai · Donghao Luo · Jiangning Zhang · Wei Lin · Taisong Jin · Chengjie Wang · Rongrong Ji, ,https://arxiv.org/abs/2312.06354,,2312.06354.pdf,PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization,"Recent advancements in personalized image generation using diffusion models
+have been noteworthy. However, existing methods suffer from inefficiencies due
+to the requirement for subject-specific fine-tuning. This computationally
+intensive process hinders efficient deployment, limiting practical usability.
+Moreover, these methods often grapple with identity distortion and limited
+expression diversity. In light of these challenges, we propose PortraitBooth,
+an innovative approach designed for high efficiency, robust identity
+preservation, and expression-editable text-to-image generation, without the
+need for fine-tuning. PortraitBooth leverages subject embeddings from a face
+recognition model for personalized image generation without fine-tuning. It
+eliminates computational overhead and mitigates identity distortion. The
+introduced dynamic identity preservation strategy further ensures close
+resemblance to the original image identity. Moreover, PortraitBooth
+incorporates emotion-aware cross-attention control for diverse facial
+expressions in generated images, supporting text-driven expression editing. Its
+scalability enables efficient and high-quality image creation, including
+multi-subject generation. Extensive results demonstrate superior performance
+over other state-of-the-art methods in both single and multiple image
+generation scenarios.",cs.CV,['cs.CV']
+Discriminability-Driven Channel Selection for Out-of-Distribution Detection,Yue Yuan · Rundong He · Yicong Dong · Zhongyi Han · Yilong Yin, ,,https://www.semanticscholar.org/paper/Exploring-Channel-Aware-Typical-Features-for-He-Yuan/755390c365c4a39445f73ed09fe673f2b823876d,,,,,nan
+Unmixing Diffusion for Self-Supervised Hyperspectral Image Denoising,Haijin Zeng · Jiezhang Cao · Yongyong Chen · Kai Zhang · Hiep Luong · Wilfried Philips, ,https://arxiv.org/abs/2311.11417,,,DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model,"This paper endeavors to advance the precision of snapshot compressive imaging
+(SCI) reconstruction for multispectral image (MSI). To achieve this, we
+integrate the advantageous attributes of established SCI techniques and an
+image generative model, propose a novel structured zero-shot diffusion model,
+dubbed DiffSCI. DiffSCI leverages the structural insights from the deep prior
+and optimization-based methodologies, complemented by the generative
+capabilities offered by the contemporary denoising diffusion model.
+Specifically, firstly, we employ a pre-trained diffusion model, which has been
+trained on a substantial corpus of RGB images, as the generative denoiser
+within the Plug-and-Play framework for the first time. This integration allows
+for the successful completion of SCI reconstruction, especially in the case
+that current methods struggle to address effectively. Secondly, we
+systematically account for spectral band correlations and introduce a robust
+methodology to mitigate wavelength mismatch, thus enabling seamless adaptation
+of the RGB diffusion model to MSIs. Thirdly, an accelerated algorithm is
+implemented to expedite the resolution of the data subproblem. This
+augmentation not only accelerates the convergence rate but also elevates the
+quality of the reconstruction process. We present extensive testing to show
+that DiffSCI exhibits discernible performance enhancements over prevailing
+self-supervised and zero-shot approaches, surpassing even supervised
+transformer counterparts across both simulated and real datasets. Our code will
+be available.",cs.CV,['cs.CV']
+Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection,Huan Liu · Zichang Tan · Chuangchuang Tan · Yunchao Wei · Jingdong Wang · Yao Zhao,https://github.com/Michel-liu/FatFormer,https://arxiv.org/abs/2312.16649,,2312.16649.pdf,Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection,"In this paper, we study the problem of generalizable synthetic image
+detection, aiming to detect forgery images from diverse generative methods,
+e.g., GANs and diffusion models. Cutting-edge solutions start to explore the
+benefits of pre-trained models, and mainly follow the fixed paradigm of solely
+training an attached classifier, e.g., combining frozen CLIP-ViT with a
+learnable linear layer in UniFD. However, our analysis shows that such a fixed
+paradigm is prone to yield detectors with insufficient learning regarding
+forgery representations. We attribute the key challenge to the lack of forgery
+adaptation, and present a novel forgery-aware adaptive transformer approach,
+namely FatFormer. Based on the pre-trained vision-language spaces of CLIP,
+FatFormer introduces two core designs for the adaption to build generalized
+forgery representations. First, motivated by the fact that both image and
+frequency analysis are essential for synthetic image detection, we develop a
+forgery-aware adapter to adapt image features to discern and integrate local
+forgery traces within image and frequency domains. Second, we find that
+considering the contrastive objectives between adapted image features and text
+prompt embeddings, a previously overlooked aspect, results in a nontrivial
+generalization improvement. Accordingly, we introduce language-guided alignment
+to supervise the forgery adaptation with image and text prompts in FatFormer.
+Experiments show that, by coupling these two designs, our approach tuned on
+4-class ProGAN data attains a remarkable detection performance, achieving an
+average of 98% accuracy to unseen GANs, and surprisingly generalizes to unseen
+diffusion models with 95% accuracy.",cs.CV,['cs.CV']
+"What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions",Brian Chen · Nina Shvetsova · Andrew Rouditchenko · Daniel Kondermann · Samuel Thomas · Shih-Fu Chang · Rogerio Feris · James Glass · Hilde Kuehne, ,,https://openreview.net/forum?id=eEtfBIjzWi,,,,,nan
+CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization,Yao Ni · Piotr Koniusz, ,https://arxiv.org/abs/2404.00521,,2404.00521.pdf,CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization,"Generative Adversarial Networks (GANs) significantly advanced image
+generation but their performance heavily depends on abundant training data. In
+scenarios with limited data, GANs often struggle with discriminator overfitting
+and unstable training. Batch Normalization (BN), despite being known for
+enhancing generalization and training stability, has rarely been used in the
+discriminator of Data-Efficient GANs. Our work addresses this gap by
+identifying a critical flaw in BN: the tendency for gradient explosion during
+the centering and scaling steps. To tackle this issue, we present CHAIN
+(lipsCHitz continuity constrAIned Normalization), which replaces the
+conventional centering step with zero-mean regularization and integrates a
+Lipschitz continuity constraint in the scaling step. CHAIN further enhances GAN
+training by adaptively interpolating the normalized and unnormalized features,
+effectively avoiding discriminator overfitting. Our theoretical analyses firmly
+establishes CHAIN's effectiveness in reducing gradients in latent features and
+weights, improving stability and generalization in GAN training. Empirical
+evidence supports our theory. CHAIN achieves state-of-the-art results in
+data-limited scenarios on CIFAR-10/100, ImageNet, five low-shot and seven
+high-resolution few-shot image datasets. Code:
+https://github.com/MaxwellYaoNi/CHAIN",cs.LG,"['cs.LG', 'cs.CV']"
+Improving Plasticity in Online Continual Learning via Collaborative Learning,Maorong Wang · Nicolas Michel · Ling Xiao · Toshihiko Yamasaki, ,https://arxiv.org/abs/2312.00600,,2312.00600.pdf,Improving Plasticity in Online Continual Learning via Collaborative Learning,"Online Continual Learning (CL) solves the problem of learning the
+ever-emerging new classification tasks from a continuous data stream. Unlike
+its offline counterpart, in online CL, the training data can only be seen once.
+Most existing online CL research regards catastrophic forgetting (i.e., model
+stability) as almost the only challenge. In this paper, we argue that the
+model's capability to acquire new knowledge (i.e., model plasticity) is another
+challenge in online CL. While replay-based strategies have been shown to be
+effective in alleviating catastrophic forgetting, there is a notable gap in
+research attention toward improving model plasticity. To this end, we propose
+Collaborative Continual Learning (CCL), a collaborative learning based strategy
+to improve the model's capability in acquiring new concepts. Additionally, we
+introduce Distillation Chain (DC), a collaborative learning scheme to boost the
+training of the models. We adapt CCL-DC to existing representative online CL
+works. Extensive experiments demonstrate that even if the learners are
+well-trained with state-of-the-art online CL methods, our strategy can still
+improve model plasticity dramatically, and thereby improve the overall
+performance by a large margin. The source code of our work is available at
+https://github.com/maorong-wang/CCL-DC.",cs.LG,['cs.LG']
+Bi-SSC: Geometric-Semantic Bidirectional Fusion for Camera-based 3D Semantic Scene Completion,Yujie Xue · Ruihui Li · F anWu · Zhuo Tang · Kenli Li · Duan Mingxing, ,https://arxiv.org/abs/2312.05752,,2312.05752.pdf,Camera-based 3D Semantic Scene Completion with Sparse Guidance Network,"Semantic scene completion (SSC) aims to predict the semantic occupancy of
+each voxel in the entire 3D scene from limited observations, which is an
+emerging and critical task for autonomous driving. Recently, many studies have
+turned to camera-based SSC solutions due to the richer visual cues and
+cost-effectiveness of cameras. However, existing methods usually rely on
+sophisticated and heavy 3D models to directly process the lifted 3D features
+that are not discriminative enough for clear segmentation boundaries. In this
+paper, we adopt the dense-sparse-dense design and propose an end-to-end
+camera-based SSC framework, termed SGN, to diffuse semantics from the semantic-
+and occupancy-aware seed voxels to the whole scene based on geometry prior and
+occupancy information. By designing hybrid guidance (sparse semantic and
+geometry guidance) and effective voxel aggregation for spatial occupancy and
+geometry priors, we enhance the feature separation between different categories
+and expedite the convergence of semantic diffusion. Extensive experimental
+results on the SemanticKITTI dataset demonstrate the superiority of our SGN
+over existing state-of-the-art methods.",cs.CV,['cs.CV']
+ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object,Chenshuang Zhang · Fei Pan · Junmo Kim · In So Kweon · Chengzhi Mao,https://github.com/chenshuang-zhang/imagenet_d,https://arxiv.org/abs/2403.18775,,2403.18775.pdf,ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object,"We establish rigorous benchmarks for visual perception robustness. Synthetic
+images such as ImageNet-C, ImageNet-9, and Stylized ImageNet provide specific
+type of evaluation over synthetic corruptions, backgrounds, and textures, yet
+those robustness benchmarks are restricted in specified variations and have low
+synthetic quality. In this work, we introduce generative model as a data source
+for synthesizing hard images that benchmark deep models' robustness. Leveraging
+diffusion models, we are able to generate images with more diversified
+backgrounds, textures, and materials than any prior work, where we term this
+benchmark as ImageNet-D. Experimental results show that ImageNet-D results in a
+significant accuracy drop to a range of vision models, from the standard ResNet
+visual classifier to the latest foundation models like CLIP and MiniGPT-4,
+significantly reducing their accuracy by up to 60\%. Our work suggests that
+diffusion models can be an effective source to test vision models. The code and
+dataset are available at https://github.com/chenshuang-zhang/imagenet_d.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Object Pose Estimation via the Aggregation of Diffusion Features,Tianfu Wang · Guosheng Hu · Hongguang Wang,https://github.com/Tianfu18/diff-feats-pose,https://arxiv.org/abs/2403.18791,,2403.18791.pdf,Object Pose Estimation via the Aggregation of Diffusion Features,"Estimating the pose of objects from images is a crucial task of 3D scene
+understanding, and recent approaches have shown promising results on very large
+benchmarks. However, these methods experience a significant performance drop
+when dealing with unseen objects. We believe that it results from the limited
+generalizability of image features. To address this problem, we have an
+in-depth analysis on the features of diffusion models, e.g. Stable Diffusion,
+which hold substantial potential for modeling unseen objects. Based on this
+analysis, we then innovatively introduce these diffusion features for object
+pose estimation. To achieve this, we propose three distinct architectures that
+can effectively capture and aggregate diffusion features of different
+granularity, greatly improving the generalizability of object pose estimation.
+Our approach outperforms the state-of-the-art methods by a considerable margin
+on three popular benchmark datasets, LM, O-LM, and T-LESS. In particular, our
+method achieves higher accuracy than the previous best arts on unseen objects:
+98.2% vs. 93.5% on Unseen LM, 85.9% vs. 76.3% on Unseen O-LM, showing the
+strong generalizability of our method. Our code is released at
+https://github.com/Tianfu18/diff-feats-pose.",cs.CV,['cs.CV']
+Efficient Meshflow and Optical Flow Estimation from Event Cameras,Xinglong Luo · Ao Luo · Zhengning Wang · Chunyu Lin · Bing Zeng · Shuaicheng Liu,https://github.com/boomluo02/EEMFlow,https://arxiv.org/abs/2307.05033,,2307.05033.pdf,Towards Anytime Optical Flow Estimation with Event Cameras,"Optical flow estimation is a fundamental task in the field of autonomous
+driving. Event cameras are capable of responding to log-brightness changes in
+microseconds. Its characteristic of producing responses only to the changing
+region is particularly suitable for optical flow estimation. In contrast to the
+super low-latency response speed of event cameras, existing datasets collected
+via event cameras, however, only provide limited frame rate optical flow ground
+truth, (e.g., at 10Hz), greatly restricting the potential of event-driven
+optical flow. To address this challenge, we put forward a high-frame-rate,
+low-latency event representation Unified Voxel Grid, sequentially fed into the
+network bin by bin. We then propose EVA-Flow, an EVent-based Anytime Flow
+estimation network to produce high-frame-rate event optical flow with only
+low-frame-rate optical flow ground truth for supervision. The key component of
+our EVA-Flow is the stacked Spatiotemporal Motion Refinement (SMR) module,
+which predicts temporally dense optical flow and enhances the accuracy via
+spatial-temporal motion refinement. The time-dense feature warping utilized in
+the SMR module provides implicit supervision for the intermediate optical flow.
+Additionally, we introduce the Rectified Flow Warp Loss (RFWL) for the
+unsupervised evaluation of intermediate optical flow in the absence of ground
+truth. This is, to the best of our knowledge, the first work focusing on
+anytime optical flow estimation via event cameras. A comprehensive variety of
+experiments on MVSEC, DESC, and our EVA-FlowSet demonstrates that EVA-Flow
+achieves competitive performance, super-low-latency (5ms), fastest inference
+(9.2ms), time-dense motion estimation (200Hz), and strong generalization. Our
+code will be available at https://github.com/Yaozhuwa/EVA-Flow.",cs.CV,"['cs.CV', 'cs.RO', 'eess.IV']"
+MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior,Honghua Chen · Chen Change Loy · Xingang Pan, ,https://arxiv.org/abs/2405.02859,,2405.02859.pdf,MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior,"Despite the emergence of successful NeRF inpainting methods built upon
+explicit RGB and depth 2D inpainting supervisions, these methods are inherently
+constrained by the capabilities of their underlying 2D inpainters. This is due
+to two key reasons: (i) independently inpainting constituent images results in
+view-inconsistent imagery, and (ii) 2D inpainters struggle to ensure
+high-quality geometry completion and alignment with inpainted RGB images.
+  To overcome these limitations, we propose a novel approach called MVIP-NeRF
+that harnesses the potential of diffusion priors for NeRF inpainting,
+addressing both appearance and geometry aspects. MVIP-NeRF performs joint
+inpainting across multiple views to reach a consistent solution, which is
+achieved via an iterative optimization process based on Score Distillation
+Sampling (SDS). Apart from recovering the rendered RGB images, we also extract
+normal maps as a geometric representation and define a normal SDS loss that
+motivates accurate geometry inpainting and alignment with the appearance.
+Additionally, we formulate a multi-view SDS score function to distill
+generative priors simultaneously from different view images, ensuring
+consistent visual completion when dealing with large view variations. Our
+experimental results show better appearance and geometry recovery than previous
+NeRF inpainting methods.",cs.CV,['cs.CV']
+Functional Diffusion,Biao Zhang · Peter Wonka, ,https://arxiv.org/abs/2311.15435,,2311.15435.pdf,Functional Diffusion,"We propose a new class of generative diffusion models, called functional
+diffusion. In contrast to previous work, functional diffusion works on samples
+that are represented by functions with a continuous domain. Functional
+diffusion can be seen as an extension of classical diffusion models to an
+infinite-dimensional domain. Functional diffusion is very versatile as images,
+videos, audio, 3D shapes, deformations, \etc, can be handled by the same
+framework with minimal changes. In addition, functional diffusion is especially
+suited for irregular data or data defined in non-standard domains. In our work,
+we derive the necessary foundations for functional diffusion and propose a
+first implementation based on the transformer architecture. We show generative
+results on complicated signed distance functions and deformation functions
+defined on 3D surfaces.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping,Hyeongjun Kwon · Jinhyun Jang · Jin Kim · Kwonyoung Kim · Kwanghoon Sohn, ,https://arxiv.org/abs/2404.00974v1,,2404.00974v1.pdf,Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping,"Visual scenes are naturally organized in a hierarchy, where a coarse semantic
+is recursively comprised of several fine details. Exploring such a visual
+hierarchy is crucial to recognize the complex relations of visual elements,
+leading to a comprehensive scene understanding. In this paper, we propose a
+Visual Hierarchy Mapper (Hi-Mapper), a novel approach for enhancing the
+structured understanding of the pre-trained Deep Neural Networks (DNNs).
+Hi-Mapper investigates the hierarchical organization of the visual scene by 1)
+pre-defining a hierarchy tree through the encapsulation of probability
+densities; and 2) learning the hierarchical relations in hyperbolic space with
+a novel hierarchical contrastive loss. The pre-defined hierarchy tree
+recursively interacts with the visual features of the pre-trained DNNs through
+hierarchy decomposition and encoding procedures, thereby effectively
+identifying the visual hierarchy and enhancing the recognition of an entire
+scene. Extensive experiments demonstrate that Hi-Mapper significantly enhances
+the representation capability of DNNs, leading to an improved performance on
+various tasks, including image classification and dense prediction tasks.",cs.CV,['cs.CV']
+Neural Underwater Scene Representation,Yunkai Tang · Chengxuan Zhu · Renjie Wan · Chao Xu · Boxin Shi, ,,https://freebutuselesssoul.github.io/publications/cvpr2024a,,,,,nan
+ViewFusion: Towards Multi-View Consistency via Interpolated Denoising,Xianghui Yang · Gil Avraham · Yan Zuo · Sameera Ramasinghe · Loris Bazzani · Anton van den Hengel, ,https://arxiv.org/abs/2402.18842,,2402.18842.pdf,ViewFusion: Towards Multi-View Consistency via Interpolated Denoising,"Novel-view synthesis through diffusion models has demonstrated remarkable
+potential for generating diverse and high-quality images. Yet, the independent
+process of image generation in these prevailing methods leads to challenges in
+maintaining multiple-view consistency. To address this, we introduce
+ViewFusion, a novel, training-free algorithm that can be seamlessly integrated
+into existing pre-trained diffusion models. Our approach adopts an
+auto-regressive method that implicitly leverages previously generated views as
+context for the next view generation, ensuring robust multi-view consistency
+during the novel-view generation process. Through a diffusion process that
+fuses known-view information via interpolated denoising, our framework
+successfully extends single-view conditioned models to work in multiple-view
+conditional settings without any additional fine-tuning. Extensive experimental
+results demonstrate the effectiveness of ViewFusion in generating consistent
+and detailed novel views.",cs.CV,['cs.CV']
+Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimation,Ruicong Liu · Takehiko Ohkawa · Mingfang Zhang · Yoichi Sato, ,https://arxiv.org/abs/2403.04381,,2403.04381.pdf,Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimation,"The pursuit of accurate 3D hand pose estimation stands as a keystone for
+understanding human activity in the realm of egocentric vision. The majority of
+existing estimation methods still rely on single-view images as input, leading
+to potential limitations, e.g., limited field-of-view and ambiguity in depth.
+To address these problems, adding another camera to better capture the shape of
+hands is a practical direction. However, existing multi-view hand pose
+estimation methods suffer from two main drawbacks: 1) Requiring multi-view
+annotations for training, which are expensive. 2) During testing, the model
+becomes inapplicable if camera parameters/layout are not the same as those used
+in training. In this paper, we propose a novel Single-to-Dual-view adaptation
+(S2DHand) solution that adapts a pre-trained single-view estimator to dual
+views. Compared with existing multi-view training methods, 1) our adaptation
+process is unsupervised, eliminating the need for multi-view annotation. 2)
+Moreover, our method can handle arbitrary dual-view pairs with unknown camera
+parameters, making the model applicable to diverse camera settings.
+Specifically, S2DHand is built on certain stereo constraints, including
+pair-wise cross-view consensus and invariance of transformation between both
+views. These two stereo constraints are used in a complementary manner to
+generate pseudo-labels, allowing reliable adaptation. Evaluation results reveal
+that S2DHand achieves significant improvements on arbitrary camera pairs under
+both in-dataset and cross-dataset settings, and outperforms existing adaptation
+methods with leading performance. Project page:
+https://github.com/MickeyLLG/S2DHand.",cs.CV,['cs.CV']
+Backpropagation-free Network for 3D Test-time Adaptation,YANSHUO WANG · Ali Cheraghian · Zeeshan Hayder · JIE HONG · Sameera Ramasinghe · Shafin Rahman · David Ahmedt-Aristizabal · Xuesong Li · Lars Petersson · Mehrtash Harandi, ,https://arxiv.org/abs/2403.18442,,2403.18442.pdf,Backpropagation-free Network for 3D Test-time Adaptation,"Real-world systems often encounter new data over time, which leads to
+experiencing target domain shifts. Existing Test-Time Adaptation (TTA) methods
+tend to apply computationally heavy and memory-intensive backpropagation-based
+approaches to handle this. Here, we propose a novel method that uses a
+backpropagation-free approach for TTA for the specific case of 3D data. Our
+model uses a two-stream architecture to maintain knowledge about the source
+domain as well as complementary target-domain-specific information. The
+backpropagation-free property of our model helps address the well-known
+forgetting problem and mitigates the error accumulation issue. The proposed
+method also eliminates the need for the usually noisy process of
+pseudo-labeling and reliance on costly self-supervised training. Moreover, our
+method leverages subspace learning, effectively reducing the distribution
+variance between the two domains. Furthermore, the source-domain-specific and
+the target-domain-specific streams are aligned using a novel entropy-based
+adaptive fusion strategy. Extensive experiments on popular benchmarks
+demonstrate the effectiveness of our method. The code will be available at
+\url{https://github.com/abie-e/BFTT3D}.",cs.CV,['cs.CV']
+Improving Single Domain-Generalized Object Detection: A Focus on Diversification and Alignment,Muhammad Sohail Danish · Muhammad Haris Khan · Muhammad Akhtar Munir · M. Sarfraz · Mohsen Ali, ,https://arxiv.org/abs/2405.14497,,2405.14497.pdf,Improving Single Domain-Generalized Object Detection: A Focus on Diversification and Alignment,"In this work, we tackle the problem of domain generalization for object
+detection, specifically focusing on the scenario where only a single source
+domain is available. We propose an effective approach that involves two key
+steps: diversifying the source domain and aligning detections based on class
+prediction confidence and localization. Firstly, we demonstrate that by
+carefully selecting a set of augmentations, a base detector can outperform
+existing methods for single domain generalization by a good margin. This
+highlights the importance of domain diversification in improving the
+performance of object detectors. Secondly, we introduce a method to align
+detections from multiple views, considering both classification and
+localization outputs. This alignment procedure leads to better generalized and
+well-calibrated object detector models, which are crucial for accurate
+decision-making in safety-critical applications. Our approach is
+detector-agnostic and can be seamlessly applied to both single-stage and
+two-stage detectors. To validate the effectiveness of our proposed methods, we
+conduct extensive experiments and ablations on challenging domain-shift
+scenarios. The results consistently demonstrate the superiority of our approach
+compared to existing methods. Our code and models are available at:
+https://github.com/msohaildanish/DivAlign",cs.CV,['cs.CV']
+Universal Segmentation at Arbitrary Granularity with Language Instruction,Yong Liu · Cairong Zhang · Yitong Wang · Jiahao Wang · Yujiu Yang · Yansong Tang, ,https://arxiv.org/abs/2312.01623,,2312.01623.pdf,Universal Segmentation at Arbitrary Granularity with Language Instruction,"This paper aims to achieve universal segmentation of arbitrary semantic
+level. Despite significant progress in recent years, specialist segmentation
+approaches are limited to specific tasks and data distribution. Retraining a
+new model for adaptation to new scenarios or settings takes expensive
+computation and time cost, which raises the demand for versatile and universal
+segmentation model that can cater to various granularity. Although some
+attempts have been made for unifying different segmentation tasks or
+generalization to various scenarios, limitations in the definition of paradigms
+and input-output spaces make it difficult for them to achieve accurate
+understanding of content at arbitrary granularity. To this end, we present
+UniLSeg, a universal segmentation model that can perform segmentation at any
+semantic level with the guidance of language instructions. For training
+UniLSeg, we reorganize a group of tasks from original diverse distributions
+into a unified data format, where images with texts describing segmentation
+targets as input and corresponding masks are output. Combined with a automatic
+annotation engine for utilizing numerous unlabeled data, UniLSeg achieves
+excellent performance on various tasks and settings, surpassing both specialist
+and unified segmentation models.",cs.CV,['cs.CV']
+ScanFormer: Referring Expression Comprehension by Iteratively Scanning,Wei Su · Peihan Miao · Huanzhang Dou · Xi Li, ,http://export.arxiv.org/abs/2306.04451,,2306.04451.pdf,Referring Expression Comprehension Using Language Adaptive Inference,"Different from universal object detection, referring expression comprehension
+(REC) aims to locate specific objects referred to by natural language
+expressions. The expression provides high-level concepts of relevant visual and
+contextual patterns, which vary significantly with different expressions and
+account for only a few of those encoded in the REC model. This leads us to a
+question: do we really need the entire network with a fixed structure for
+various referring expressions? Ideally, given an expression, only
+expression-relevant components of the REC model are required. These components
+should be small in number as each expression only contains very few visual and
+contextual clues. This paper explores the adaptation between expressions and
+REC models for dynamic inference. Concretely, we propose a neat yet efficient
+framework named Language Adaptive Dynamic Subnets (LADS), which can extract
+language-adaptive subnets from the REC model conditioned on the referring
+expressions. By using the compact subnet, the inference can be more economical
+and efficient. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and
+Referit show that the proposed method achieves faster inference speed and
+higher accuracy against state-of-the-art approaches.",cs.CV,['cs.CV']
+SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing,Tomoki Ichikawa · Shohei Nobuhara · Ko Nishino,https://vision.ist.i.kyoto-u.ac.jp/research/spiders/,https://arxiv.org/abs/2312.04553,,2312.04553.pdf,SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing,"Can we capture shape and reflectance in stealth? Such capability would be
+valuable for many application domains in vision, xR, robotics, and HCI. We
+introduce structured polarization for invisible depth and reflectance sensing
+(SPIDeRS), the first depth and reflectance sensing method using patterns of
+polarized light. The key idea is to modulate the angle of linear polarization
+(AoLP) of projected light at each pixel. The use of polarization makes it
+invisible and lets us recover not only depth but also directly surface normals
+and even reflectance. We implement SPIDeRS with a liquid crystal spatial light
+modulator (SLM) and a polarimetric camera. We derive a novel method for
+robustly extracting the projected structured polarization pattern from the
+polarimetric object appearance. We evaluate the effectiveness of SPIDeRS by
+applying it to a number of real-world objects. The results show that our method
+successfully reconstructs object shapes of various materials and is robust to
+diffuse reflection and ambient light. We also demonstrate relighting using
+recovered surface normals and reflectance. We believe SPIDeRS opens a new
+avenue of polarization use in visual sensing.",cs.CV,"['cs.CV', 'eess.IV']"
+MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning,Chaoyi Zhang · Kevin Lin · Zhengyuan Yang · Jianfeng Wang · Linjie Li · Chung-Ching Lin · Zicheng Liu · Lijuan Wang, ,https://arxiv.org/abs/2311.17435,,2311.17435.pdf,MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning,"We present MM-Narrator, a novel system leveraging GPT-4 with multimodal
+in-context learning for the generation of audio descriptions (AD). Unlike
+previous methods that primarily focused on downstream fine-tuning with short
+video clips, MM-Narrator excels in generating precise audio descriptions for
+videos of extensive lengths, even beyond hours, in an autoregressive manner.
+This capability is made possible by the proposed memory-augmented generation
+process, which effectively utilizes both the short-term textual context and
+long-term visual memory through an efficient register-and-recall mechanism.
+These contextual memories compile pertinent past information, including
+storylines and character identities, ensuring an accurate tracking and
+depicting of story-coherent and character-centric audio descriptions.
+Maintaining the training-free design of MM-Narrator, we further propose a
+complexity-based demonstration selection strategy to largely enhance its
+multi-step reasoning capability via few-shot multimodal in-context learning
+(MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator
+consistently outperforms both the existing fine-tuning-based approaches and
+LLM-based approaches in most scenarios, as measured by standard evaluation
+metrics. Additionally, we introduce the first segment-based evaluator for
+recurrent text generation. Empowered by GPT-4, this evaluator comprehensively
+reasons and marks AD generation performance in various extendable dimensions.",cs.CV,"['cs.CV', 'cs.AI']"
+DisCo: Disentangled Control for Realistic Human Dance Generation,Tan Wang · Linjie Li · Kevin Lin · Yuanhao Zhai · Chung-Ching Lin · Zhengyuan Yang · Hanwang Zhang · Zicheng Liu · Lijuan Wang, ,https://arxiv.org/abs/2307.00040,,2307.00040.pdf,DisCo: Disentangled Control for Realistic Human Dance Generation,"Generative AI has made significant strides in computer vision, particularly
+in text-driven image/video synthesis (T2I/T2V). Despite the notable
+advancements, it remains challenging in human-centric content synthesis such as
+realistic dance generation. Current methodologies, primarily tailored for human
+motion transfer, encounter difficulties when confronted with real-world dance
+scenarios (e.g., social media dance), which require to generalize across a wide
+spectrum of poses and intricate human details. In this paper, we depart from
+the traditional paradigm of human motion transfer and emphasize two additional
+critical attributes for the synthesis of human dance content in social media
+contexts: (i) Generalizability: the model should be able to generalize beyond
+generic human viewpoints as well as unseen human subjects, backgrounds, and
+poses; (ii) Compositionality: it should allow for the seamless composition of
+seen/unseen subjects, backgrounds, and poses from different sources. To address
+these challenges, we introduce DISCO, which includes a novel model architecture
+with disentangled control to improve the compositionality of dance synthesis,
+and an effective human attribute pre-training for better generalizability to
+unseen humans. Extensive qualitative and quantitative results demonstrate that
+DisCc can generate high-quality human dance images and videos with diverse
+appearances and flexible motions. Code is available at
+https://disco-dance.github.io/.",cs.CV,"['cs.CV', 'cs.AI']"
+DeMatch: Deep Decomposition of Motion Field for Two-View Correspondence Learning,Shihua Zhang · Zizhuo Li · Yuan Gao · Jiayi Ma, ,,https://ojs.aaai.org/index.php/AAAI/article/view/25456,,,,,nan
+EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning,Hongxia Xie · Chu-Jun Peng · Yu-Wen Tseng · Hung-Jen Chen · Chan-Feng Hsu · Hong-Han Shuai · Wen-Huang Cheng, ,https://arxiv.org/abs/2404.16670,,2404.16670.pdf,EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning,"Visual Instruction Tuning represents a novel learning paradigm involving the
+fine-tuning of pre-trained language models using task-specific instructions.
+This paradigm shows promising zero-shot results in various natural language
+processing tasks but is still unexplored in vision emotion understanding. In
+this work, we focus on enhancing the model's proficiency in understanding and
+adhering to instructions related to emotional contexts. Initially, we identify
+key visual clues critical to visual emotion recognition. Subsequently, we
+introduce a novel GPT-assisted pipeline for generating emotion visual
+instruction data, effectively addressing the scarcity of annotated instruction
+data in this domain. Expanding on the groundwork established by InstructBLIP,
+our proposed EmoVIT architecture incorporates emotion-specific instruction
+data, leveraging the powerful capabilities of Large Language Models to enhance
+performance. Through extensive experiments, our model showcases its proficiency
+in emotion classification, adeptness in affective reasoning, and competence in
+comprehending humor. The comparative analysis provides a robust benchmark for
+Emotion Visual Instruction Tuning in the era of LLMs, providing valuable
+insights and opening avenues for future exploration in this domain. Our code is
+available at \url{https://github.com/aimmemotion/EmoVIT}.",cs.CV,"['cs.CV', 'cs.AI']"
+PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion,Ying-Tian Liu · Yuan-Chen Guo · Guan Luo · Heyi Sun · Wei Yin · Song-Hai Zhang, ,https://arxiv.org/abs/2312.09069,,2312.09069.pdf,PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion,"Diffusion models trained on large-scale text-image datasets have demonstrated
+a strong capability of controllable high-quality image generation from
+arbitrary text prompts. However, the generation quality and generalization
+ability of 3D diffusion models is hindered by the scarcity of high-quality and
+large-scale 3D datasets. In this paper, we present PI3D, a framework that fully
+leverages the pre-trained text-to-image diffusion models' ability to generate
+high-quality 3D shapes from text prompts in minutes. The core idea is to
+connect the 2D and 3D domains by representing a 3D shape as a set of Pseudo RGB
+Images. We fine-tune an existing text-to-image diffusion model to produce such
+pseudo-images using a small number of text-3D pairs. Surprisingly, we find that
+it can already generate meaningful and consistent 3D shapes given complex text
+descriptions. We further take the generated shapes as the starting point for a
+lightweight iterative refinement using score distillation sampling to achieve
+high-quality generation under a low budget. PI3D generates a single 3D shape
+from text in only 3 minutes and the quality is validated to outperform existing
+3D generative models by a large margin.",cs.CV,['cs.CV']
+VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction,Jiaqi Lin · Zhihao Li · Xiao Tang · Jianzhuang Liu · Shiyong Liu · Jiayue Liu · Yangdi Lu · Xiaofei Wu · Songcen Xu · Youliang Yan · Wenming Yang, ,https://arxiv.org/abs/2402.17427,,2402.17427.pdf,VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction,"Existing NeRF-based methods for large scene reconstruction often have
+limitations in visual quality and rendering speed. While the recent 3D Gaussian
+Splatting works well on small-scale and object-centric scenes, scaling it up to
+large scenes poses challenges due to limited video memory, long optimization
+time, and noticeable appearance variations. To address these challenges, we
+present VastGaussian, the first method for high-quality reconstruction and
+real-time rendering on large scenes based on 3D Gaussian Splatting. We propose
+a progressive partitioning strategy to divide a large scene into multiple
+cells, where the training cameras and point cloud are properly distributed with
+an airspace-aware visibility criterion. These cells are merged into a complete
+scene after parallel optimization. We also introduce decoupled appearance
+modeling into the optimization process to reduce appearance variations in the
+rendered images. Our approach outperforms existing NeRF-based methods and
+achieves state-of-the-art results on multiple large scene datasets, enabling
+fast optimization and high-fidelity real-time rendering.",cs.CV,['cs.CV']
+Open-Vocabulary Segmentation with Semantic-Assisted Calibration,Yong Liu · Sule Bai · Guanbin Li · Yitong Wang · Yansong Tang, ,https://arxiv.org/abs/2312.04089,,,Open-Vocabulary Segmentation with Semantic-Assisted Calibration,"This paper studies open-vocabulary segmentation (OVS) through calibrating
+in-vocabulary and domain-biased embedding space with generalized contextual
+prior of CLIP. As the core of open-vocabulary understanding, alignment of
+visual content with the semantics of unbounded text has become the bottleneck
+of this field. To address this challenge, recent works propose to utilize CLIP
+as an additional classifier and aggregate model predictions with CLIP
+classification results. Despite their remarkable progress, performance of OVS
+methods in relevant scenarios is still unsatisfactory compared with supervised
+counterparts. We attribute this to the in-vocabulary embedding and
+domain-biased CLIP prediction. To this end, we present a Semantic-assisted
+CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior
+of CLIP into proposal embedding to avoid collapsing on known categories.
+Besides, a contextual shift strategy is applied to mitigate the lack of global
+context and unnatural background noise. With above designs, SCAN achieves
+state-of-the-art performance on all popular open-vocabulary segmentation
+benchmarks. Furthermore, we also focus on the problem of existing evaluation
+system that ignores semantic duplication across categories, and propose a new
+metric called Semantic-Guided IoU (SG-IoU).",cs.CV,['cs.CV']
+GPT4Point: A Unified Framework for Point-Language Understanding and Generation,Zhangyang Qi · Ye Fang · Zeyi Sun · Xiaoyang Wu · Tong Wu · Jiaqi Wang · Dahua Lin · Hengshuang Zhao, ,https://arxiv.org/abs/2312.02980,,2312.02980.pdf,GPT4Point: A Unified Framework for Point-Language Understanding and Generation,"Multimodal Large Language Models (MLLMs) have excelled in 2D image-text
+comprehension and image generation, but their understanding of the 3D world is
+notably deficient, limiting progress in 3D language understanding and
+generation. To solve this problem, we introduce GPT4Point, an innovative
+groundbreaking point-language multimodal model designed specifically for
+unified 3D object understanding and generation within the MLLM framework.
+GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text
+reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point
+is equipped with advanced capabilities for controllable 3D generation, it can
+get high-quality results through a low-quality point-text feature maintaining
+the geometric shapes and colors. To support the expansive needs of 3D
+object-text pairs, we develop Pyramid-XL, a point-language dataset annotation
+engine. It constructs a large-scale database over 1M objects of varied text
+granularity levels from the Objaverse-XL dataset, essential for training
+GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D
+point-language understanding capabilities. In extensive evaluations, GPT4Point
+has demonstrated superior performance in understanding and generation.",cs.CV,['cs.CV']
+FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation,Pengchong Qiao · Lei Shang · Chang Liu · Baigui Sun · Xiangyang Ji · Jie Chen, ,,https://paperswithcode.com/paper/facechain-sude-building-derived-class-to,,,,,nan
+TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding,Yun Liu · Haolin Yang · Xu Si · Ling Liu · Zipeng Li · Yuxiang Zhang · Yebin Liu · Li Yi, ,https://arxiv.org/abs/2401.08399,,2401.08399.pdf,TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding,"Humans commonly work with multiple objects in daily life and can intuitively
+transfer manipulation skills to novel objects by understanding object
+functional regularities. However, existing technical approaches for analyzing
+and synthesizing hand-object manipulation are mostly limited to handling a
+single hand and object due to the lack of data support. To address this, we
+construct TACO, an extensive bimanual hand-object-interaction dataset spanning
+a large variety of tool-action-object compositions for daily human activities.
+TACO contains 2.5K motion sequences paired with third-person and egocentric
+views, precise hand-object 3D meshes, and action labels. To rapidly expand the
+data scale, we present a fully automatic data acquisition pipeline combining
+multi-view sensing with an optical motion capture system. With the vast
+research fields provided by TACO, we benchmark three generalizable
+hand-object-interaction tasks: compositional action recognition, generalizable
+hand-object motion forecasting, and cooperative grasp synthesis. Extensive
+experiments reveal new insights, challenges, and opportunities for advancing
+the studies of generalizable hand-object motion analysis and synthesis. Our
+data and code are available at https://taco2024.github.io.",cs.CV,['cs.CV']
+Alpha-CLIP: A CLIP Model Focusing on Wherever You Want,Zeyi Sun · Ye Fang · Tong Wu · Pan Zhang · Yuhang Zang · Shu Kong · Yuanjun Xiong · Dahua Lin · Jiaqi Wang,https://aleafy.github.io/alpha-clip/,https://arxiv.org/abs/2312.03818,,2312.03818.pdf,Alpha-CLIP: A CLIP Model Focusing on Wherever You Want,"Contrastive Language-Image Pre-training (CLIP) plays an essential role in
+extracting valuable content information from images across diverse tasks. It
+aligns textual and visual modalities to comprehend the entire image, including
+all the details, even those irrelevant to specific tasks. However, for a finer
+understanding and controlled editing of images, it becomes crucial to focus on
+specific regions of interest, which can be indicated as points, masks, or boxes
+by humans or perception models. To fulfill the requirements, we introduce
+Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to
+suggest attentive regions and fine-tuned with constructed millions of RGBA
+region-text pairs. Alpha-CLIP not only preserves the visual recognition ability
+of CLIP but also enables precise control over the emphasis of image contents.
+It demonstrates effectiveness in various tasks, including but not limited to
+open-world recognition, multimodal large language models, and conditional 2D /
+3D generation. It has a strong potential to serve as a versatile tool for
+image-related tasks.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+VCoder: Versatile Vision Encoders for Multimodal Large Language Models,Jitesh Jain · Jianwei Yang · Humphrey Shi,https://praeclarumjj3.github.io/vcoder/,https://arxiv.org/abs/2312.14233,,2312.14233.pdf,VCoder: Versatile Vision Encoders for Multimodal Large Language Models,"Humans possess the remarkable skill of Visual Perception, the ability to see
+and understand the seen, helping them make sense of the visual world and, in
+turn, reason. Multimodal Large Language Models (MLLM) have recently achieved
+impressive performance on vision-language tasks ranging from visual
+question-answering and image captioning to visual reasoning and image
+generation. However, when prompted to identify or count (perceive) the entities
+in a given image, existing MLLM systems fail. Working towards developing an
+accurate MLLM system for perception and reasoning, we propose using Versatile
+vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the
+VCoder with perception modalities such as segmentation or depth maps, improving
+the MLLM's perception abilities. Secondly, we leverage the images from COCO and
+outputs from off-the-shelf vision perception models to create our COCO
+Segmentation Text (COST) dataset for training and evaluating MLLMs on the
+object perception task. Thirdly, we introduce metrics to assess the object
+perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive
+experimental evidence proving the VCoder's improved object-level perception
+skills over existing Multimodal LLMs, including GPT-4V. We open-source our
+dataset, code, and models to promote research. We open-source our code at
+https://github.com/SHI-Labs/VCoder",cs.CV,['cs.CV']
+Emotional Speech-Driven 3D Body Animation via Disentangled Latent Diffusion,Kiran Chhatre · Radek Danecek · Nikos Athanasiou · Giorgio Becherini · Christopher Peters · Michael J. Black · Timo Bolkart,https://amuse.is.tue.mpg.de/,https://arxiv.org/abs/2312.04466,,2312.04466.pdf,Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion,"Existing methods for synthesizing 3D human gestures from speech have shown
+promising results, but they do not explicitly model the impact of emotions on
+the generated gestures. Instead, these methods directly output animations from
+speech without control over the expressed emotion. To address this limitation,
+we present AMUSE, an emotional speech-driven body animation model based on
+latent diffusion. Our observation is that content (i.e., gestures related to
+speech rhythm and word utterances), emotion, and personal style are separable.
+To account for this, AMUSE maps the driving audio to three disentangled latent
+vectors: one for content, one for emotion, and one for personal style. A latent
+diffusion model, trained to generate gesture motion sequences, is then
+conditioned on these latent vectors. Once trained, AMUSE synthesizes 3D human
+gestures directly from speech with control over the expressed emotions and
+style by combining the content from the driving speech with the emotion and
+style of another speech sequence. Randomly sampling the noise of the diffusion
+model further generates variations of the gesture with the same emotional
+expressivity. Qualitative, quantitative, and perceptual evaluations demonstrate
+that AMUSE outputs realistic gesture sequences. Compared to the state of the
+art, the generated gestures are better synchronized with the speech content,
+and better represent the emotion expressed by the input speech. Our code is
+available at amuse.is.tue.mpg.de.",cs.CV,['cs.CV']
+Accept the Modality Gap: An Exploration in the Hyperbolic Space,Sameera Ramasinghe · Violetta Shevchenko · Gil Avraham · Thalaiyasingam Ajanthan, ,,https://openreview.net/forum?id=KiespDPaRH,,,,,nan
+Transferable Structural Sparse Adversarial Attack Via Exact Group Sparsity Training,Di Ming · Peng Ren · Yunlong Wang · Xin Feng,https://github.com/MisterRpeng/EGS-TSSA,,https://midasdming.github.io/news/announcement_17/,,,,,nan
+DreamComposer: Controllable 3D Object Generation via Multi-View Conditions,Yunhan Yang · Yukun Huang · Xiaoyang Wu · Yuan-Chen Guo · Song-Hai Zhang · Hengshuang Zhao · Tong He · Xihui Liu, ,https://arxiv.org/abs/2312.03611,,2312.03611.pdf,DreamComposer: Controllable 3D Object Generation via Multi-View Conditions,"Utilizing pre-trained 2D large-scale generative models, recent works are
+capable of generating high-quality novel views from a single in-the-wild image.
+However, due to the lack of information from multiple views, these works
+encounter difficulties in generating controllable novel views. In this paper,
+we present DreamComposer, a flexible and scalable framework that can enhance
+existing view-aware diffusion models by injecting multi-view conditions.
+Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
+3D representations of an object from multiple views. Then, it renders the
+latent features of the target view from 3D representations with the multi-view
+feature fusion module. Finally the target view features extracted from
+multi-view inputs are injected into a pre-trained diffusion model. Experiments
+show that DreamComposer is compatible with state-of-the-art diffusion models
+for zero-shot novel view synthesis, further enhancing them to generate
+high-fidelity novel view images with multi-view conditions, ready for
+controllable 3D object reconstruction and various other applications.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Pose Adapted Shape Learning for Large-Pose Face Reenactment,Gee-Sern Hsu · Jie-Ying Zhang · Yu-Hsiang Huang · Wei-Jie Hong, ,,https://ieeexplore.ieee.org/abstract/document/10219601,,,,,nan
+LayoutFormer: Hierarchical Text Detection Towards Scene Text Understanding,Min Liang · Jia-Wei Ma · Xiaobin Zhu · Jingyan Qin · Xu-Cheng Yin, ,https://ar5iv.labs.arxiv.org/html/2207.12955,,2207.12955.pdf,Contextual Text Block Detection towards Scene Text Understanding,"Most existing scene text detectors focus on detecting characters or words
+that only capture partial text messages due to missing contextual information.
+For a better understanding of text in scenes, it is more desired to detect
+contextual text blocks (CTBs) which consist of one or multiple integral text
+units (e.g., characters, words, or phrases) in natural reading order and
+transmit certain complete text messages. This paper presents contextual text
+detection, a new setup that detects CTBs for better understanding of texts in
+scenes. We formulate the new setup by a dual detection task which first detects
+integral text units and then groups them into a CTB. To this end, we design a
+novel scene text clustering technique that treats integral text units as tokens
+and groups them (belonging to the same CTB) into an ordered token sequence. In
+addition, we create two datasets SCUT-CTW-Context and ReCTS-Context to
+facilitate future research, where each CTB is well annotated by an ordered
+sequence of integral text units. Further, we introduce three metrics that
+measure contextual text detection in local accuracy, continuity, and global
+accuracy. Extensive experiments show that our method accurately detects CTBs
+which effectively facilitates downstream tasks such as text classification and
+translation. The project is available at
+https://sg-vilab.github.io/publication/xue2022contextual/.",cs.CV,['cs.CV']
+PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos,Qi Zhao · M. Salman Asif · Zhan Ma, ,https://arxiv.org/abs/2404.08921,,2404.08921.pdf,PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos,"The primary focus of Neural Representation for Videos (NeRV) is to
+effectively model its spatiotemporal consistency. However, current NeRV systems
+often face a significant issue of spatial inconsistency, leading to decreased
+perceptual quality. To address this issue, we introduce the Pyramidal Neural
+Representation for Videos (PNeRV), which is built on a multi-scale information
+connection and comprises a lightweight rescaling operator, Kronecker
+Fully-connected layer (KFc), and a Benign Selective Memory (BSM) mechanism. The
+KFc, inspired by the tensor decomposition of the vanilla Fully-connected layer,
+facilitates low-cost rescaling and global correlation modeling. BSM merges
+high-level features with granular ones adaptively. Furthermore, we provide an
+analysis based on the Universal Approximation Theory of the NeRV system and
+validate the effectiveness of the proposed PNeRV.We conducted comprehensive
+experiments to demonstrate that PNeRV surpasses the performance of contemporary
+NeRV models, achieving the best results in video regression on UVG and DAVIS
+under various metrics (PSNR, SSIM, LPIPS, and FVD). Compared to vanilla NeRV,
+PNeRV achieves a +4.49 dB gain in PSNR and a 231% increase in FVD on UVG, along
+with a +3.28 dB PSNR and 634% FVD increase on DAVIS.",cs.CV,['cs.CV']
+Bézier Everywhere All at Once: Learning Drivable Lanes as Bézier Graphs,Hugh Blayney · Hanlin Tian · Hamish Scott · Nils Goldbeck · Chess Stetson · Panagiotis Angeloudis, ,,https://screenrant.com/everything-everywhere-all-at-once-real-meaning-explained/,,,,,nan
+Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer,Jiwoo Chung · Sangeek Hyun · Jae-Pil Heo,https://jiwoogit.github.io/StyleID_site/,https://arxiv.org/abs/2312.09008,,2312.09008.pdf,Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer,"Despite the impressive generative capabilities of diffusion models, existing
+diffusion model-based style transfer methods require inference-stage
+optimization (e.g. fine-tuning or textual inversion of style) which is
+time-consuming, or fails to leverage the generative ability of large-scale
+diffusion models. To address these issues, we introduce a novel artistic style
+transfer method based on a pre-trained large-scale diffusion model without any
+optimization. Specifically, we manipulate the features of self-attention layers
+as the way the cross-attention mechanism works; in the generation process,
+substituting the key and value of content with those of style image. This
+approach provides several desirable characteristics for style transfer
+including 1) preservation of content by transferring similar styles into
+similar image patches and 2) transfer of style based on similarity of local
+texture (e.g. edge) between content and style images. Furthermore, we introduce
+query preservation and attention temperature scaling to mitigate the issue of
+disruption of original content, and initial latent Adaptive Instance
+Normalization (AdaIN) to deal with the disharmonious color (failure to transfer
+the colors of style). Our experimental results demonstrate that our proposed
+method surpasses state-of-the-art methods in both conventional and
+diffusion-based style transfer baselines.",cs.CV,['cs.CV']
+Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes,Diandian Guo · Deng-Ping Fan · Tongyu Lu · Christos Sakaridis · Luc Van Gool,https://github.com/RascalGdd/VPSeg,https://arxiv.org/abs/2401.15261,,2401.15261.pdf,Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes,"The estimation of implicit cross-frame correspondences and the high
+computational cost have long been major challenges in video semantic
+segmentation (VSS) for driving scenes. Prior works utilize keyframes, feature
+propagation, or cross-frame attention to address these issues. By contrast, we
+are the first to harness vanishing point (VP) priors for more effective
+segmentation. Intuitively, objects near VPs (i.e., away from the vehicle) are
+less discernible. Moreover, they tend to move radially away from the VP over
+time in the usual case of a forward-facing camera, a straight road, and linear
+forward motion of the vehicle. Our novel, efficient network for VSS, named
+VPSeg, incorporates two modules that utilize exactly this pair of static and
+dynamic VP priors: sparse-to-dense feature mining (DenseVP) and VP-guided
+motion fusion (MotionVP). MotionVP employs VP-guided motion estimation to
+establish explicit correspondences across frames and help attend to the most
+relevant features from neighboring frames, while DenseVP enhances weak dynamic
+features in distant regions around VPs. These modules operate within a
+context-detail framework, which separates contextual features from
+high-resolution local features at different input resolutions to reduce
+computational costs. Contextual and local features are integrated through
+contextualized motion attention (CMA) for the final prediction. Extensive
+experiments on two popular driving segmentation benchmarks, Cityscapes and
+ACDC, demonstrate that VPSeg outperforms previous SOTA methods, with only
+modest computational overhead.",cs.CV,['cs.CV']
+TransNeXt: Robust Foveal Visual Perception for Vision Transformers,Dai Shi, ,https://arxiv.org/abs/2311.17132,,2311.17132.pdf,TransNeXt: Robust Foveal Visual Perception for Vision Transformers,"Due to the depth degradation effect in residual connections, many efficient
+Vision Transformers models that rely on stacking layers for information
+exchange often fail to form sufficient information mixing, leading to unnatural
+visual perception. To address this issue, in this paper, we propose Aggregated
+Attention, a biomimetic design-based token mixer that simulates biological
+foveal vision and continuous eye movement while enabling each token on the
+feature map to have a global perception. Furthermore, we incorporate learnable
+tokens that interact with conventional queries and keys, which further
+diversifies the generation of affinity matrices beyond merely relying on the
+similarity between queries and keys. Our approach does not rely on stacking for
+information exchange, thus effectively avoiding depth degradation and achieving
+natural visual perception. Additionally, we propose Convolutional GLU, a
+channel mixer that bridges the gap between GLU and SE mechanism, which empowers
+each token to have channel attention based on its nearest neighbor image
+features, enhancing local modeling capability and model robustness. We combine
+aggregated attention and convolutional GLU to create a new visual backbone
+called TransNeXt. Extensive experiments demonstrate that our TransNeXt achieves
+state-of-the-art performance across multiple model sizes. At a resolution of
+$224^2$, TransNeXt-Tiny attains an ImageNet accuracy of 84.0%, surpassing
+ConvNeXt-B with 69% fewer parameters. Our TransNeXt-Base achieves an ImageNet
+accuracy of 86.2% and an ImageNet-A accuracy of 61.6% at a resolution of
+$384^2$, a COCO object detection mAP of 57.1, and an ADE20K semantic
+segmentation mIoU of 54.7.",cs.CV,"['cs.CV', 'cs.AI']"
+Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects,Yijia Weng · Bowen Wen · Jonathan Tremblay · Valts Blukis · Dieter Fox · Leonidas Guibas · Stan Birchfield,https://nvlabs.github.io/DigitalTwinArt/,https://arxiv.org/abs/2404.01440,,2404.01440.pdf,Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects,"We address the problem of building digital twins of unknown articulated
+objects from two RGBD scans of the object at different articulation states. We
+decompose the problem into two stages, each addressing distinct aspects. Our
+method first reconstructs object-level shape at each state, then recovers the
+underlying articulation model including part segmentation and joint
+articulations that associate the two states. By explicitly modeling point-level
+correspondences and exploiting cues from images, 3D reconstructions, and
+kinematics, our method yields more accurate and stable results compared to
+prior work. It also handles more than one movable part and does not rely on any
+object shape or structure priors. Project page:
+https://github.com/NVlabs/DigitalTwinArt",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.RO']"
+MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,Xiang Yue · Yuansheng Ni · Kai Zhang · Tianyu Zheng · Ruoqi Liu · Ge Zhang · Samuel Stevens · Dongfu Jiang · Weiming Ren · Yuxuan Sun · Cong Wei · Botao Yu · Ruibin Yuan · Renliang Sun · Ming Yin · Boyuan Zheng · Zhenzhu Yang · Yibo Liu · Wenhao Huang · Huan Sun · Yu Su · Wenhu Chen,https://mmmu-benchmark.github.io/,https://arxiv.org/abs/2311.16502,,2311.16502.pdf,MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,"We introduce MMMU: a new benchmark designed to evaluate multimodal models on
+massive multi-discipline tasks demanding college-level subject knowledge and
+deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal
+questions from college exams, quizzes, and textbooks, covering six core
+disciplines: Art & Design, Business, Science, Health & Medicine, Humanities &
+Social Science, and Tech & Engineering. These questions span 30 subjects and
+183 subfields, comprising 30 highly heterogeneous image types, such as charts,
+diagrams, maps, tables, music sheets, and chemical structures. Unlike existing
+benchmarks, MMMU focuses on advanced perception and reasoning with
+domain-specific knowledge, challenging models to perform tasks akin to those
+faced by experts. The evaluation of 14 open-source LMMs as well as the
+proprietary GPT-4V(ision) and Gemini highlights the substantial challenges
+posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve
+accuracies of 56% and 59% respectively, indicating significant room for
+improvement. We believe MMMU will stimulate the community to build
+next-generation multimodal foundation models towards expert artificial general
+intelligence.",cs.CL,"['cs.CL', 'cs.AI', 'cs.CV']"
+Intriguing Properties of Diffusion Models: An Empirical Study of the Natural Attack Capability in Text-to-Image Generative Models,Takami Sato · Justin Yue · Nanze Chen · Ningfei Wang · Alfred Chen, ,https://arxiv.org/abs/2308.15692,,2308.15692.pdf,Intriguing Properties of Diffusion Models: An Empirical Study of the Natural Attack Capability in Text-to-Image Generative Models,"Denoising probabilistic diffusion models have shown breakthrough performance
+to generate more photo-realistic images or human-level illustrations than the
+prior models such as GANs. This high image-generation capability has stimulated
+the creation of many downstream applications in various areas. However, we find
+that this technology is actually a double-edged sword: We identify a new type
+of attack, called the Natural Denoising Diffusion (NDD) attack based on the
+finding that state-of-the-art deep neural network (DNN) models still hold their
+prediction even if we intentionally remove their robust features, which are
+essential to the human visual system (HVS), through text prompts. The NDD
+attack shows a significantly high capability to generate low-cost,
+model-agnostic, and transferable adversarial attacks by exploiting the natural
+attack capability in diffusion models. To systematically evaluate the risk of
+the NDD attack, we perform a large-scale empirical study with our newly created
+dataset, the Natural Denoising Diffusion Attack (NDDA) dataset. We evaluate the
+natural attack capability by answering 6 research questions. Through a user
+study, we find that it can achieve an 88% detection rate while being stealthy
+to 93% of human subjects; we also find that the non-robust features embedded by
+diffusion models contribute to the natural attack capability. To confirm the
+model-agnostic and transferable attack capability, we perform the NDD attack
+against the Tesla Model 3 and find that 73% of the physically printed attacks
+can be detected as stop signs. Our hope is that the study and dataset can help
+our community be aware of the risks in diffusion models and facilitate further
+research toward robust DNN models.",cs.CV,"['cs.CV', 'cs.CR']"
+SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting,Hoon Kim · Minje Jang · Wonjun Yoon · Jisoo Lee · Donghyun Na · Sanghyun Woo, ,https://arxiv.org/abs/2402.18848,,2402.18848.pdf,SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting,"We introduce a co-designed approach for human portrait relighting that
+combines a physics-guided architecture with a pre-training framework. Drawing
+on the Cook-Torrance reflectance model, we have meticulously configured the
+architecture design to precisely simulate light-surface interactions.
+Furthermore, to overcome the limitation of scarce high-quality lightstage data,
+we have developed a self-supervised pre-training strategy. This novel
+combination of accurate physical modeling and expanded training dataset
+establishes a new benchmark in relighting realism.",cs.CV,['cs.CV']
+Context-Aware Integration of Language and Visual References for Natural Language Tracking,Yanyan Shao · Shuting He · Qi Ye · Yuchao Feng · Wenhan Luo · Jiming Chen,https://github.com/twotwo2/QueryNLT,https://arxiv.org/abs/2403.19975,,2403.19975.pdf,Context-Aware Integration of Language and Visual References for Natural Language Tracking,"Tracking by natural language specification (TNL) aims to consistently
+localize a target in a video sequence given a linguistic description in the
+initial frame. Existing methodologies perform language-based and template-based
+matching for target reasoning separately and merge the matching results from
+two sources, which suffer from tracking drift when language and visual
+templates miss-align with the dynamic target state and ambiguity in the later
+merging stage. To tackle the issues, we propose a joint multi-modal tracking
+framework with 1) a prompt modulation module to leverage the complementarity
+between temporal visual templates and language expressions, enabling precise
+and context-aware appearance and linguistic cues, and 2) a unified target
+decoding module to integrate the multi-modal reference cues and executes the
+integrated queries on the search image to predict the target location in an
+end-to-end manner directly. This design ensures spatio-temporal consistency by
+leveraging historical visual information and introduces an integrated solution,
+generating predictions in a single step. Extensive experiments conducted on
+TNL2K, OTB-Lang, LaSOT, and RefCOCOg validate the efficacy of our proposed
+approach. The results demonstrate competitive performance against
+state-of-the-art methods for both tracking and grounding.",cs.CV,['cs.CV']
+Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities,Mingcheng Li · Dingkang Yang · Xiao Zhao · Shuaibing Wang · Yan Wang · Kun Yang · Mingyang Sun · Dongliang Kou · Qian · Lihua Zhang, ,https://arxiv.org/abs/2404.16456,,2404.16456.pdf,Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities,"Multimodal sentiment analysis (MSA) aims to understand human sentiment
+through multimodal data. Most MSA efforts are based on the assumption of
+modality completeness. However, in real-world applications, some practical
+factors cause uncertain modality missingness, which drastically degrades the
+model's performance. To this end, we propose a Correlation-decoupled Knowledge
+Distillation (CorrKD) framework for the MSA task under uncertain missing
+modalities. Specifically, we present a sample-level contrastive distillation
+mechanism that transfers comprehensive knowledge containing cross-sample
+correlations to reconstruct missing semantics. Moreover, a category-guided
+prototype distillation mechanism is introduced to capture cross-category
+correlations using category prototypes to align feature distributions and
+generate favorable joint representations. Eventually, we design a
+response-disentangled consistency distillation strategy to optimize the
+sentiment decision boundaries of the student network through response
+disentanglement and mutual information maximization. Comprehensive experiments
+on three datasets indicate that our framework can achieve favorable
+improvements compared with several baselines.",cs.CV,['cs.CV']
+Pose-Transformed Equivariant Network for  3D Point Trajectory Prediction,Ruixuan Yu · Jian Sun, ,https://arxiv.org/abs/2308.06564,,2308.06564.pdf,EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction,"Accurate trajectory prediction is crucial for the safe and efficient
+operation of autonomous vehicles. The growing popularity of deep learning has
+led to the development of numerous methods for trajectory prediction. While
+deterministic deep learning models have been widely used, deep generative
+models have gained popularity as they learn data distributions from training
+data and account for trajectory uncertainties. In this study, we propose
+EquiDiff, a deep generative model for predicting future vehicle trajectories.
+EquiDiff is based on the conditional diffusion model, which generates future
+trajectories by incorporating historical information and random Gaussian noise.
+The backbone model of EquiDiff is an SO(2)-equivariant transformer that fully
+utilizes the geometric properties of location coordinates. In addition, we
+employ Recurrent Neural Networks and Graph Attention Networks to extract social
+interactions from historical trajectories. To evaluate the performance of
+EquiDiff, we conduct extensive experiments on the NGSIM dataset. Our results
+demonstrate that EquiDiff outperforms other baseline models in short-term
+prediction, but has slightly higher errors for long-term prediction.
+Furthermore, we conduct an ablation study to investigate the contribution of
+each component of EquiDiff to the prediction accuracy. Additionally, we present
+a visualization of the generation process of our diffusion model, providing
+insights into the uncertainty of the prediction.",cs.LG,"['cs.LG', 'cs.RO']"
+SC-Tune: Unleashing Self-Consistent Referential Comprehension  in Large Vision Language Models,Tongtian Yue · Jie Cheng · Longteng Guo · Xingyuan Dai · Zijia Zhao · Xingjian He · Gang Xiong · Yisheng Lv · Jing Liu, ,https://arxiv.org/abs/2403.13263,,2403.13263.pdf,SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models,"Recent trends in Large Vision Language Models (LVLMs) research have been
+increasingly focusing on advancing beyond general image understanding towards
+more nuanced, object-level referential comprehension. In this paper, we present
+and delve into the self-consistency capability of LVLMs, a crucial aspect that
+reflects the models' ability to both generate informative captions for specific
+objects and subsequently utilize these captions to accurately re-identify the
+objects in a closed-loop process. This capability significantly mirrors the
+precision and reliability of fine-grained visual-language understanding. Our
+findings reveal that the self-consistency level of existing LVLMs falls short
+of expectations, posing limitations on their practical applicability and
+potential. To address this gap, we introduce a novel fine-tuning paradigm named
+Self-Consistency Tuning (SC-Tune). It features the synergistic learning of a
+cyclic describer-locator system. This paradigm is not only data-efficient but
+also exhibits generalizability across multiple LVLMs. Through extensive
+experiments, we demonstrate that SC-Tune significantly elevates performance
+across a spectrum of object-level vision-language benchmarks and maintains
+competitive or improved performance on image-level vision-language benchmarks.
+Both our model and code will be publicly available at
+https://github.com/ivattyue/SC-Tune.",cs.CV,['cs.CV']
+Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models,Huimin Huang · Yawen Huang · Lanfen Lin · Ruofeng Tong · Yen-Wei Chen · Hao Zheng · Yuexiang Li · Yefeng Zheng, ,https://arxiv.org/abs/2405.14136,,,Efficient Multitask Dense Predictor via Binarization,"Multi-task learning for dense prediction has emerged as a pivotal area in
+computer vision, enabling simultaneous processing of diverse yet interrelated
+pixel-wise prediction tasks. However, the substantial computational demands of
+state-of-the-art (SoTA) models often limit their widespread deployment. This
+paper addresses this challenge by introducing network binarization to compress
+resource-intensive multi-task dense predictors. Specifically, our goal is to
+significantly accelerate multi-task dense prediction models via Binary Neural
+Networks (BNNs) while maintaining and even improving model performance at the
+same time. To reach this goal, we propose a Binary Multi-task Dense Predictor,
+Bi-MTDP, and several variants of Bi-MTDP, in which a multi-task dense predictor
+is constructed via specified binarized modules. Our systematical analysis of
+this predictor reveals that performance drop from binarization is primarily
+caused by severe information degradation. To address this issue, we introduce a
+deep information bottleneck layer that enforces representations for downstream
+tasks satisfying Gaussian distribution in forward propagation. Moreover, we
+introduce a knowledge distillation mechanism to correct the direction of
+information flow in backward propagation. Intriguingly, one variant of Bi-MTDP
+outperforms full-precision (FP) multi-task dense prediction SoTAs, ARTC
+(CNN-based) and InvPT (ViT-Based). This result indicates that Bi-MTDP is not
+merely a naive trade-off between performance and efficiency, but is rather a
+benefit of the redundant information flow thanks to the multi-task
+architecture. Code is available at https://github.com/42Shawn/BiMTDP.",cs.CV,['cs.CV']
+Clustering Propagation for Universal Medical Image Segmentation,Yuhang Ding · Liulei Li · Wenguan Wang · Yi Yang, ,https://arxiv.org/abs/2403.16646,,2403.16646.pdf,Clustering Propagation for Universal Medical Image Segmentation,"Prominent solutions for medical image segmentation are typically tailored for
+automatic or interactive setups, posing challenges in facilitating progress
+achieved in one task to another.$_{\!}$ This$_{\!}$ also$_{\!}$
+necessitates$_{\!}$ separate$_{\!}$ models for each task, duplicating both
+training time and parameters.$_{\!}$ To$_{\!}$ address$_{\!}$ above$_{\!}$
+issues,$_{\!}$ we$_{\!}$ introduce$_{\!}$ S2VNet,$_{\!}$ a$_{\!}$
+universal$_{\!}$ framework$_{\!}$ that$_{\!}$ leverages$_{\!}$
+Slice-to-Volume$_{\!}$ propagation$_{\!}$ to$_{\!}$ unify automatic/interactive
+segmentation within a single model and one training session. Inspired by
+clustering-based segmentation techniques, S2VNet makes full use of the
+slice-wise structure of volumetric data by initializing cluster centers from
+the cluster$_{\!}$ results$_{\!}$ of$_{\!}$ previous$_{\!}$ slice.$_{\!}$ This
+enables knowledge acquired from prior slices to assist in the segmentation of
+the current slice, further efficiently bridging the communication between
+remote slices using mere 2D networks. Moreover, such a framework readily
+accommodates interactive segmentation with no architectural change, simply by
+initializing centroids from user inputs. S2VNet distinguishes itself by swift
+inference speeds and reduced memory consumption compared to prevailing 3D
+solutions. It can also handle multi-class interactions with each of them
+serving to initialize different centroids. Experiments on three benchmarks
+demonstrate S2VNet surpasses task-specified solutions on both
+automatic/interactive setups.",cs.CV,['cs.CV']
+Tri-Modal Motion Retrieval by Learning a Joint Embedding Space,Kangning Yin · Shihao Zou · Yuxuan Ge · Zheng Tian, ,https://arxiv.org/abs/2403.00691,,2403.00691.pdf,Tri-Modal Motion Retrieval by Learning a Joint Embedding Space,"Information retrieval is an ever-evolving and crucial research domain. The
+substantial demand for high-quality human motion data especially in online
+acquirement has led to a surge in human motion research works. Prior works have
+mainly concentrated on dual-modality learning, such as text and motion tasks,
+but three-modality learning has been rarely explored. Intuitively, an extra
+introduced modality can enrich a model's application scenario, and more
+importantly, an adequate choice of the extra modality can also act as an
+intermediary and enhance the alignment between the other two disparate
+modalities. In this work, we introduce LAVIMO (LAnguage-VIdeo-MOtion
+alignment), a novel framework for three-modality learning integrating
+human-centric videos as an additional modality, thereby effectively bridging
+the gap between text and motion. Moreover, our approach leverages a specially
+designed attention mechanism to foster enhanced alignment and synergistic
+effects among text, video, and motion modalities. Empirically, our results on
+the HumanML3D and KIT-ML datasets show that LAVIMO achieves state-of-the-art
+performance in various motion-related cross-modal retrieval tasks, including
+text-to-motion, motion-to-text, video-to-motion and motion-to-video.",cs.CV,"['cs.CV', 'cs.AI']"
+Rethinking Human Motion Prediction with Symplectic Integral,Haipeng Chen · Kedi L yu · Zhenguang Liu · Yifang Yin · Xun Yang · Yingda Lyu, ,https://arxiv.org/abs/2312.06184,,2312.06184.pdf,Recent Advances in Deterministic Human Motion Prediction: A Review,"In recent years, with the continuous advancement of deep learning and the
+emergence of large-scale human motion datasets, human motion prediction
+technology has gradually gained prominence in various fields such as
+human-computer interaction, autonomous driving, sports analysis, and personnel
+tracking. This article introduces common model architectures in this domain
+along with their respective advantages and disadvantages. It also
+systematically summarizes recent research innovations, focusing on in-depth
+discussions of relevant papers in these areas, thereby highlighting
+forward-looking insights into the field's development. Furthermore, this paper
+provides a comprehensive overview of existing methods, commonly used datasets,
+and evaluation metrics in this field. Finally, it discusses some of the current
+limitations in the field and proposes potential future research directions to
+address these challenges and promote further advancements in human motion
+prediction.",cs.CV,['cs.CV']
+UniPAD: A Universal Pre-training Paradigm for Autonomous Driving,Honghui Yang · Sha Zhang · Di Huang · Xiaoyang Wu · Haoyi Zhu · Tong He · SHIXIANG TANG · Hengshuang Zhao · Qibo Qiu · Binbin Lin · Xiaofei He · Wanli Ouyang,https://github.com/Nightmare-n/UniPAD,https://arxiv.org/abs/2310.08370,,2310.08370.pdf,UniPAD: A Universal Pre-training Paradigm for Autonomous Driving,"In the context of autonomous driving, the significance of effective feature
+learning is widely acknowledged. While conventional 3D self-supervised
+pre-training methods have shown widespread success, most methods follow the
+ideas originally designed for 2D images. In this paper, we present UniPAD, a
+novel self-supervised learning paradigm applying 3D volumetric differentiable
+rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction
+of continuous 3D shape structures and the intricate appearance characteristics
+of their 2D projections. The flexibility of our method enables seamless
+integration into both 2D and 3D frameworks, enabling a more holistic
+comprehension of the scenes. We manifest the feasibility and effectiveness of
+UniPAD by conducting extensive experiments on various downstream 3D tasks. Our
+method significantly improves lidar-, camera-, and lidar-camera-based baseline
+by 9.1, 7.7, and 6.9 NDS, respectively. Notably, our pre-training pipeline
+achieves 73.2 NDS for 3D object detection and 79.4 mIoU for 3D semantic
+segmentation on the nuScenes validation set, achieving state-of-the-art results
+in comparison with previous methods. The code will be available at
+https://github.com/Nightmare-n/UniPAD.",cs.CV,['cs.CV']
+Dual-Enhanced Coreset Selection with Class-wise Collaboration for Online Blurry Class Incremental Learning,Yutian Luo · Shiqi Zhao · Haoran Wu · Zhiwu Lu, ,https://arxiv.org/abs/2308.09303,,2308.09303.pdf,Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning,"Continual learning aims to learn a model from a continuous stream of data,
+but it mainly assumes a fixed number of data and tasks with clear task
+boundaries. However, in real-world scenarios, the number of input data and
+tasks is constantly changing in a statistical way, not a static way. Although
+recently introduced incremental learning scenarios having blurry task
+boundaries somewhat address the above issues, they still do not fully reflect
+the statistical properties of real-world situations because of the fixed ratio
+of disjoint and blurry samples. In this paper, we propose a new Stochastic
+incremental Blurry task boundary scenario, called Si-Blurry, which reflects the
+stochastic properties of the real-world. We find that there are two major
+challenges in the Si-Blurry scenario: (1) inter- and intra-task forgettings and
+(2) class imbalance problem. To alleviate them, we introduce Mask and Visual
+Prompt tuning (MVP). In MVP, to address the inter- and intra-task forgetting
+issues, we propose a novel instance-wise logit masking and contrastive visual
+prompt tuning loss. Both of them help our model discern the classes to be
+learned in the current batch. It results in consolidating the previous
+knowledge. In addition, to alleviate the class imbalance problem, we introduce
+a new gradient similarity-based focal loss and adaptive feature scaling to ease
+overfitting to the major classes and underfitting to the minor classes.
+Extensive experiments show that our proposed MVP significantly outperforms the
+existing state-of-the-art methods in our challenging Si-Blurry scenario.",cs.CV,"['cs.CV', 'cs.LG']"
+Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection,Zhiwei Yang · Jing Liu · Peng Wu, ,https://arxiv.org/abs/2404.08531,,2404.08531.pdf,Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection,"Weakly supervised video anomaly detection (WSVAD) is a challenging task.
+Generating fine-grained pseudo-labels based on weak-label and then
+self-training a classifier is currently a promising solution. However, since
+the existing methods use only RGB visual modality and the utilization of
+category text information is neglected, thus limiting the generation of more
+accurate pseudo-labels and affecting the performance of self-training. Inspired
+by the manual labeling process based on the event description, in this paper,
+we propose a novel pseudo-label generation and self-training framework based on
+Text Prompt with Normality Guidance (TPWNG) for WSVAD. Our idea is to transfer
+the rich language-visual knowledge of the contrastive language-image
+pre-training (CLIP) model for aligning the video event description text and
+corresponding video frames to generate pseudo-labels. Specifically, We first
+fine-tune the CLIP for domain adaptation by designing two ranking losses and a
+distributional inconsistency loss. Further, we propose a learnable text prompt
+mechanism with the assist of a normality visual prompt to further improve the
+matching accuracy of video event description text and video frames. Then, we
+design a pseudo-label generation module based on the normality guidance to
+infer reliable frame-level pseudo-labels. Finally, we introduce a temporal
+context self-adaptive learning module to learn the temporal dependencies of
+different video events more flexibly and accurately. Extensive experiments show
+that our method achieves state-of-the-art performance on two benchmark
+datasets, UCF-Crime and XD-Viole",cs.CV,['cs.CV']
+Partial-to-Partial Shape Matching with Geometric Consistency,Viktoria Ehm · Maolin Gao · Paul Roetzer · Marvin Eisenberger · Daniel Cremers · Florian Bernard,https://vikiehm.github.io/publications/gcppsm/,https://arxiv.org/abs/2404.12209,,2404.12209.pdf,Partial-to-Partial Shape Matching with Geometric Consistency,"Finding correspondences between 3D shapes is an important and long-standing
+problem in computer vision, graphics and beyond. A prominent challenge are
+partial-to-partial shape matching settings, which occur when the shapes to
+match are only observed incompletely (e.g. from 3D scanning). Although
+partial-to-partial matching is a highly relevant setting in practice, it is
+rarely explored. Our work bridges the gap between existing (rather artificial)
+3D full shape matching and partial-to-partial real-world settings by exploiting
+geometric consistency as a strong constraint. We demonstrate that it is indeed
+possible to solve this challenging problem in a variety of settings. For the
+first time, we achieve geometric consistency for partial-to-partial matching,
+which is realized by a novel integer non-linear program formalism building on
+triangle product spaces, along with a new pruning algorithm based on linear
+integer programming. Further, we generate a new inter-class dataset for
+partial-to-partial shape-matching. We show that our method outperforms current
+SOTA methods on both an established intra-class dataset and our novel
+inter-class dataset.",cs.CV,['cs.CV']
+Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss,Jaeha Kim · Junghun Oh · Kyoung Mu Lee, ,https://arxiv.org/abs/2404.01692,,2404.01692.pdf,Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss,"In real-world scenarios, image recognition tasks, such as semantic
+segmentation and object detection, often pose greater challenges due to the
+lack of information available within low-resolution (LR) content. Image
+super-resolution (SR) is one of the promising solutions for addressing the
+challenges. However, due to the ill-posed property of SR, it is challenging for
+typical SR methods to restore task-relevant high-frequency contents, which may
+dilute the advantage of utilizing the SR method. Therefore, in this paper, we
+propose Super-Resolution for Image Recognition (SR4IR) that effectively guides
+the generation of SR images beneficial to achieving satisfactory image
+recognition performance when processing LR images. The critical component of
+our SR4IR is the task-driven perceptual (TDP) loss that enables the SR network
+to acquire task-specific knowledge from a network tailored for a specific task.
+Moreover, we propose a cross-quality patch mix and an alternate training
+framework that significantly enhances the efficacy of the TDP loss by
+addressing potential problems when employing the TDP loss. Through extensive
+experiments, we demonstrate that our SR4IR achieves outstanding task
+performance by generating SR images useful for a specific image recognition
+task, including semantic segmentation, object detection, and image
+classification. The implementation code is available at
+https://github.com/JaehaKim97/SR4IR.",cs.CV,['cs.CV']
+FaceCom: Towards High-fidelity 3D Facial Shape Completion via Optimization and Inpainting Guidance,Yinglong Li · Hongyu Wu · Wang · Qingzhao Qin · yijiao zhao · Yong Wang · Aimin Hao, ,https://arxiv.org/abs/2308.16758,,2308.16758.pdf,Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images,"Generating 3D faces from textual descriptions has a multitude of
+applications, such as gaming, movie, and robotics. Recent progresses have
+demonstrated the success of unconditional 3D face generation and text-to-3D
+shape generation. However, due to the limited text-3D face data pairs,
+text-driven 3D face generation remains an open problem. In this paper, we
+propose a text-guided 3D faces generation method, refer as TG-3DFace, for
+generating realistic 3D faces using text guidance. Specifically, we adopt an
+unconditional 3D face generation framework and equip it with text conditions,
+which learns the text-guided 3D face generation with only text-2D face data. On
+top of that, we propose two text-to-face cross-modal alignment techniques,
+including the global contrastive learning and the fine-grained alignment
+module, to facilitate high semantic consistency between generated 3D faces and
+input texts. Besides, we present directional classifier guidance during the
+inference process, which encourages creativity for out-of-domain generations.
+Compared to the existing methods, TG-3DFace creates more realistic and
+aesthetically pleasing 3D faces, boosting 9% multi-view consistency (MVIC) over
+Latent3D. The rendered face images generated by TG-3DFace achieve higher FID
+and CLIP score than text-to-2D face/image generation models, demonstrating our
+superiority in generating realistic and semantic-consistent textures.",cs.CV,['cs.CV']
+4K4D: Real-Time 4D View Synthesis at 4K Resolution,Zhen Xu · Sida Peng · Haotong Lin · Guangzhao He · Jiaming Sun · Yujun Shen · Hujun Bao · Xiaowei Zhou,https://zju3dv.github.io/4k4d,https://arxiv.org/abs/2310.11448,,2310.11448.pdf,4K4D: Real-Time 4D View Synthesis at 4K Resolution,"This paper targets high-fidelity and real-time view synthesis of dynamic 3D
+scenes at 4K resolution. Recently, some methods on dynamic view synthesis have
+shown impressive rendering quality. However, their speed is still limited when
+rendering high-resolution images. To overcome this problem, we propose 4K4D, a
+4D point cloud representation that supports hardware rasterization and enables
+unprecedented rendering speed. Our representation is built on a 4D feature grid
+so that the points are naturally regularized and can be robustly optimized. In
+addition, we design a novel hybrid appearance model that significantly boosts
+the rendering quality while preserving efficiency. Moreover, we develop a
+differentiable depth peeling algorithm to effectively learn the proposed model
+from RGB videos. Experiments show that our representation can be rendered at
+over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the
+ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x
+faster than previous methods and achieves the state-of-the-art rendering
+quality. Our project page is available at https://zju3dv.github.io/4k4d/.",cs.CV,['cs.CV']
+VILA: On Pre-training for Visual Language Models,Ji Lin · Danny Yin · Wei Ping · Pavlo Molchanov · Mohammad Shoeybi · Song Han,https://github.com/NVlabs/VILA,https://arxiv.org/abs/2312.07533,,,VILA: On Pre-training for Visual Language Models,"Visual language models (VLMs) rapidly progressed with the recent success of
+large language models. There have been growing efforts on visual instruction
+tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
+visual language pre-training process, where the model learns to perform joint
+modeling on both modalities. In this work, we examine the design options for
+VLM pre-training by augmenting LLM towards VLM through step-by-step
+controllable comparisons. We introduce three main findings: (1) freezing LLMs
+during pre-training can achieve decent zero-shot performance, but lack
+in-context learning capability, which requires unfreezing the LLM; (2)
+interleaved pre-training data is beneficial whereas image-text pairs alone are
+not optimal; (3) re-blending text-only instruction data to image-text data
+during instruction fine-tuning not only remedies the degradation of text-only
+tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
+we build VILA, a Visual Language model family that consistently outperforms the
+state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
+and whistles. Multi-modal pre-training also helps unveil appealing properties
+of VILA, including multi-image reasoning, enhanced in-context learning, and
+better world knowledge.",cs.CV,['cs.CV']
+GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting,Chi Yan · Delin Qu · Dong Wang · Dan Xu · Zhigang Wang · Bin Zhao · Xuelong Li,https://gs-slam.github.io/,https://arxiv.org/abs/2311.11700,,2311.11700.pdf,GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting,"In this paper, we introduce \textbf{GS-SLAM} that first utilizes 3D Gaussian
+representation in the Simultaneous Localization and Mapping (SLAM) system. It
+facilitates a better balance between efficiency and accuracy. Compared to
+recent SLAM methods employing neural implicit representations, our method
+utilizes a real-time differentiable splatting rendering pipeline that offers
+significant speedup to map optimization and RGB-D rendering. Specifically, we
+propose an adaptive expansion strategy that adds new or deletes noisy 3D
+Gaussians in order to efficiently reconstruct new observed scene geometry and
+improve the mapping of previously observed areas. This strategy is essential to
+extend 3D Gaussian representation to reconstruct the whole scene rather than
+synthesize a static object in existing methods. Moreover, in the pose tracking
+process, an effective coarse-to-fine technique is designed to select reliable
+3D Gaussian representations to optimize camera pose, resulting in runtime
+reduction and robust estimation. Our method achieves competitive performance
+compared with existing state-of-the-art real-time methods on the Replica,
+TUM-RGBD datasets. Project page: https://gs-slam.github.io/.",cs.CV,['cs.CV']
+Generating Content for HDR Deghosting from Frequency View,Tao Hu · Qingsen Yan · Yuankai Qi · Yanning Zhang, ,https://arxiv.org/abs/2404.00849,,2404.00849.pdf,Generating Content for HDR Deghosting from Frequency View,"Recovering ghost-free High Dynamic Range (HDR) images from multiple Low
+Dynamic Range (LDR) images becomes challenging when the LDR images exhibit
+saturation and significant motion. Recent Diffusion Models (DMs) have been
+introduced in HDR imaging field, demonstrating promising performance,
+particularly in achieving visually perceptible results compared to previous
+DNN-based methods. However, DMs require extensive iterations with large models
+to estimate entire images, resulting in inefficiency that hinders their
+practical application. To address this challenge, we propose the Low-Frequency
+aware Diffusion (LF-Diff) model for ghost-free HDR imaging. The key idea of
+LF-Diff is implementing the DMs in a highly compacted latent space and
+integrating it into a regression-based model to enhance the details of
+reconstructed images. Specifically, as low-frequency information is closely
+related to human visual perception we propose to utilize DMs to create compact
+low-frequency priors for the reconstruction process. In addition, to take full
+advantage of the above low-frequency priors, the Dynamic HDR Reconstruction
+Network (DHRNet) is carried out in a regression-based manner to obtain final
+HDR images. Extensive experiments conducted on synthetic and real-world
+benchmark datasets demonstrate that our LF-Diff performs favorably against
+several state-of-the-art methods and is 10$\times$ faster than previous
+DM-based methods.",cs.CV,['cs.CV']
+Neural Sign Actors: A diffusion model for 3D sign language production from text,Vasileios Baltatzis · Rolandos Alexandros Potamias · Evangelos Ververas · Guanxiong Sun · Jiankang Deng · Stefanos Zafeiriou, ,https://arxiv.org/abs/2312.02702,,2312.02702.pdf,Neural Sign Actors: A diffusion model for 3D sign language production from text,"Sign Languages (SL) serve as the primary mode of communication for the Deaf
+and Hard of Hearing communities. Deep learning methods for SL recognition and
+translation have achieved promising results. However, Sign Language Production
+(SLP) poses a challenge as the generated motions must be realistic and have
+precise semantic meaning. Most SLP methods rely on 2D data, which hinders their
+realism. In this work, a diffusion-based SLP model is trained on a curated
+large-scale dataset of 4D signing avatars and their corresponding text
+transcripts. The proposed method can generate dynamic sequences of 3D avatars
+from an unconstrained domain of discourse using a diffusion process formed on a
+novel and anatomically informed graph neural network defined on the SMPL-X body
+skeleton. Through quantitative and qualitative experiments, we show that the
+proposed method considerably outperforms previous methods of SLP. This work
+makes an important step towards realistic neural sign avatars, bridging the
+communication gap between Deaf and hearing communities.",cs.CV,['cs.CV']
+Steerers: A framework for rotation equivariant keypoint descriptors,Georg Bökman · Johan Edstedt · Michael Felsberg · Fredrik Kahl, ,https://arxiv.org/abs/2312.02152,,2312.02152.pdf,Steerers: A framework for rotation equivariant keypoint descriptors,"Image keypoint descriptions that are discriminative and matchable over large
+changes in viewpoint are vital for 3D reconstruction. However, descriptions
+output by learned descriptors are typically not robust to camera rotation.
+While they can be made more robust by, e.g., data augmentation, this degrades
+performance on upright images. Another approach is test-time augmentation,
+which incurs a significant increase in runtime. Instead, we learn a linear
+transform in description space that encodes rotations of the input image. We
+call this linear transform a steerer since it allows us to transform the
+descriptions as if the image was rotated. From representation theory, we know
+all possible steerers for the rotation group. Steerers can be optimized (A)
+given a fixed descriptor, (B) jointly with a descriptor or (C) we can optimize
+a descriptor given a fixed steerer. We perform experiments in these three
+settings and obtain state-of-the-art results on the rotation invariant image
+matching benchmarks AIMS and Roto-360. We publish code and model weights at
+https://github.com/georg-bn/rotation-steerers.",cs.CV,['cs.CV']
+LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning,Siyuan Cheng · Guanhong Tao · Yingqi Liu · Guangyu Shen · Shengwei An · Shiwei Feng · Xiangzhe Xu · Kaiyuan Zhang · Shiqing Ma · Xiangyu Zhang,https://github.com/Megum1/LOTUS,https://arxiv.org/abs/2403.17188,,2403.17188.pdf,LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning,"Backdoor attack poses a significant security threat to Deep Learning
+applications. Existing attacks are often not evasive to established backdoor
+detection techniques. This susceptibility primarily stems from the fact that
+these attacks typically leverage a universal trigger pattern or transformation
+function, such that the trigger can cause misclassification for any input. In
+response to this, recent papers have introduced attacks using sample-specific
+invisible triggers crafted through special transformation functions. While
+these approaches manage to evade detection to some extent, they reveal
+vulnerability to existing backdoor mitigation techniques. To address and
+enhance both evasiveness and resilience, we introduce a novel backdoor attack
+LOTUS. Specifically, it leverages a secret function to separate samples in the
+victim class into a set of partitions and applies unique triggers to different
+partitions. Furthermore, LOTUS incorporates an effective trigger focusing
+mechanism, ensuring only the trigger corresponding to the partition can induce
+the backdoor behavior. Extensive experimental results show that LOTUS can
+achieve high attack success rate across 4 datasets and 7 model structures, and
+effectively evading 13 backdoor detection and mitigation techniques. The code
+is available at https://github.com/Megum1/LOTUS.",cs.CV,"['cs.CV', 'cs.CR']"
+Language-only Training of Zero-shot Composed Image Retrieval,Geonmo Gu · Sanghyuk Chun · Wonjae Kim · Yoohoon Kang · Sangdoo Yun,https://github.com/navervision/lincir,https://arxiv.org/abs/2312.01998,,2312.01998.pdf,Language-only Efficient Training of Zero-shot Composed Image Retrieval,"Composed image retrieval (CIR) task takes a composed query of image and text,
+aiming to search relative images for both conditions. Conventional CIR
+approaches need a training dataset composed of triplets of query image, query
+text, and target image, which is very expensive to collect. Several recent
+works have worked on the zero-shot (ZS) CIR paradigm to tackle the issue
+without using pre-collected triplets. However, the existing ZS-CIR methods show
+limited backbone scalability and generalizability due to the lack of diversity
+of the input texts during training. We propose a novel CIR framework, only
+using language for its training. Our LinCIR (Language-only training for CIR)
+can be trained only with text datasets by a novel self-supervision named
+self-masking projection (SMP). We project the text latent embedding to the
+token embedding space and construct a new text by replacing the keyword tokens
+of the original text. Then, we let the new and original texts have the same
+latent embedding vector. With this simple strategy, LinCIR is surprisingly
+efficient and highly effective; LinCIR with CLIP ViT-G backbone is trained in
+48 minutes and shows the best ZS-CIR performances on four different CIR
+benchmarks, CIRCO, GeneCIS, FashionIQ, and CIRR, even outperforming supervised
+method on FashionIQ. Code is available at https://github.com/navervision/lincir",cs.CV,"['cs.CV', 'cs.IR']"
+"""Previously on ..."" From Recaps to Story Summarization",Aditya Kumar Singh · Dhruv Srivastava · Makarand Tapaswi, ,https://arxiv.org/abs/2405.11487,,2405.11487.pdf,"""Previously on ..."" From Recaps to Story Summarization","We introduce multimodal story summarization by leveraging TV episode recaps -
+short video sequences interweaving key story moments from previous episodes to
+bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime
+thriller TV shows with rich recaps and long episodes of 40 minutes. Story
+summarization labels are unlocked by matching recap shots to corresponding
+sub-stories in the episode. We propose a hierarchical model TaleSumm that
+processes entire episodes by creating compact shot and dialog representations,
+and predicts importance scores for each video shot and dialog utterance by
+enabling interactions between local story groups. Unlike traditional
+summarization, our method extracts multiple plot points from long videos. We
+present a thorough evaluation on story summarization, including promising
+cross-series generalization. TaleSumm also shows good results on classic video
+summarization benchmarks.",cs.CV,['cs.CV']
+Learning Equi-angular Representations for Online Continual Learning,Minhyuk Seo · Hyunseo Koh · Wonje Jeung · Minjae Lee · San Kim · Hankook Lee · Sungjun Cho · Sungik Choi · Hyunwoo Kim · Jonghyun Choi, ,https://arxiv.org/abs/2404.01628,,2404.01628.pdf,Learning Equi-angular Representations for Online Continual Learning,"Online continual learning suffers from an underfitted solution due to
+insufficient training for prompt model update (e.g., single-epoch training). To
+address the challenge, we propose an efficient online continual learning method
+using the neural collapse phenomenon. In particular, we induce neural collapse
+to form a simplex equiangular tight frame (ETF) structure in the representation
+space so that the continuously learned model with a single epoch can better fit
+to the streamed data by proposing preparatory data training and residual
+correction in the representation space. With an extensive set of empirical
+validations using CIFAR-10/100, TinyImageNet, ImageNet-200, and ImageNet-1K, we
+show that our proposed method outperforms state-of-the-art methods by a
+noticeable margin in various online continual learning scenarios such as
+disjoint and Gaussian scheduled continuous (i.e., boundary-free) data setups.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Holodeck: Language Guided Generation of 3D Embodied AI Environments,Yue Yang · Fan-Yun Sun · Luca Weihs · Eli VanderBilt · Alvaro Herrasti · Winson Han · Jiajun Wu · Nick Haber · Ranjay Krishna · Lingjie Liu · Chris Callison-Burch · Mark Yatskar · Aniruddha Kembhavi · Christopher Clark,https://yueyang1996.github.io/holodeck/,https://arxiv.org/abs/2312.09067,,2312.09067.pdf,Holodeck: Language Guided Generation of 3D Embodied AI Environments,"3D simulated environments play a critical role in Embodied AI, but their
+creation requires expertise and extensive manual effort, restricting their
+diversity and scope. To mitigate this limitation, we present Holodeck, a system
+that generates 3D environments to match a user-supplied prompt fully
+automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
+museums, adjust the designs for styles, and can capture the semantics of
+complex queries such as ""apartment for a researcher with a cat"" and ""office of
+a professor who is a fan of Star Wars"". Holodeck leverages a large language
+model (i.e., GPT-4) for common sense knowledge about what the scene might look
+like and uses a large collection of 3D assets from Objaverse to populate the
+scene with diverse objects. To address the challenge of positioning objects
+correctly, we prompt GPT-4 to generate spatial relational constraints between
+objects and then optimize the layout to satisfy those constraints. Our
+large-scale human evaluation shows that annotators prefer Holodeck over
+manually designed procedural baselines in residential scenes and that Holodeck
+can produce high-quality outputs for diverse scene types. We also demonstrate
+an exciting application of Holodeck in Embodied AI, training agents to navigate
+in novel scenes like music rooms and daycares without human-constructed data,
+which is a significant step forward in developing general-purpose embodied
+agents.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.RO']"
+NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation,Jiahao Chen · Yipeng Qin · Lingjie Liu · Jiangbo Lu · Guanbin Li, ,https://arxiv.org/abs/2403.17537,,2403.17537.pdf,NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation,"Neural Radiance Field (NeRF) has been widely recognized for its excellence in
+novel view synthesis and 3D scene reconstruction. However, their effectiveness
+is inherently tied to the assumption of static scenes, rendering them
+susceptible to undesirable artifacts when confronted with transient distractors
+such as moving objects or shadows. In this work, we propose a novel paradigm,
+namely ""Heuristics-Guided Segmentation"" (HuGS), which significantly enhances
+the separation of static scenes from transient distractors by harmoniously
+combining the strengths of hand-crafted heuristics and state-of-the-art
+segmentation models, thus significantly transcending the limitations of
+previous solutions. Furthermore, we delve into the meticulous design of
+heuristics, introducing a seamless fusion of Structure-from-Motion (SfM)-based
+heuristics and color residual heuristics, catering to a diverse range of
+texture profiles. Extensive experiments demonstrate the superiority and
+robustness of our method in mitigating transient distractors for NeRFs trained
+in non-static scenes. Project page: https://cnhaox.github.io/NeRF-HuGS/.",cs.CV,['cs.CV']
+FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning,Junyuan Zhang · Shuang Zeng · Miao Zhang · Runxi Wang · Feifei Wang · Yuyin Zhou · Paul Pu Liang · Liangqiong Qu,https://carkham.github.io/FL_Het_Bench/,https://arxiv.org/abs/2306.05172,,2306.05172.pdf,FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systems,"Federated Machine Learning (FL) has received considerable attention in recent
+years. FL benchmarks are predominantly explored in either simulated systems or
+data center environments, neglecting the setups of real-world systems, which
+are often closely linked to edge computing. We close this research gap by
+introducing FLEdge, a benchmark targeting FL workloads in edge computing
+systems. We systematically study hardware heterogeneity, energy efficiency
+during training, and the effect of various differential privacy levels on
+training in FL systems. To make this benchmark applicable to real-world
+scenarios, we evaluate the impact of client dropouts on state-of-the-art FL
+strategies with failure rates as high as 50%. FLEdge provides new insights,
+such as that training state-of-the-art FL workloads on older GPU-accelerated
+embedded devices is up to 3x more energy efficient than on modern server-grade
+GPUs.",cs.LG,"['cs.LG', 'cs.DC', 'I.2.11; C.2.4; C.4; D.2.8']"
+Hyper-MD: Mesh Denoising with Customized Parameters Aware of Noise Intensity and Geometric Characteristics,Xingtao Wang · Hongliang Wei · Xiaopeng Fan · Debin Zhao, ,https://arxiv.org/abs/2405.06536,,2405.06536.pdf,Mesh Denoising Transformer,"Mesh denoising, aimed at removing noise from input meshes while preserving
+their feature structures, is a practical yet challenging task. Despite the
+remarkable progress in learning-based mesh denoising methodologies in recent
+years, their network designs often encounter two principal drawbacks: a
+dependence on single-modal geometric representations, which fall short in
+capturing the multifaceted attributes of meshes, and a lack of effective global
+feature aggregation, hindering their ability to fully understand the mesh's
+comprehensive structure. To tackle these issues, we propose SurfaceFormer, a
+pioneering Transformer-based mesh denoising framework. Our first contribution
+is the development of a new representation known as Local Surface Descriptor,
+which is crafted by establishing polar systems on each mesh face, followed by
+sampling points from adjacent surfaces using geodesics. The normals of these
+points are organized into 2D patches, mimicking images to capture local
+geometric intricacies, whereas the poles and vertex coordinates are
+consolidated into a point cloud to embody spatial information. This advancement
+surmounts the hurdles posed by the irregular and non-Euclidean characteristics
+of mesh data, facilitating a smooth integration with Transformer architecture.
+Next, we propose a dual-stream structure consisting of a Geometric Encoder
+branch and a Spatial Encoder branch, which jointly encode local geometry
+details and spatial information to fully explore multimodal information for
+mesh denoising. A subsequent Denoising Transformer module receives the
+multimodal information and achieves efficient global feature aggregation
+through self-attention operators. Our experimental evaluations demonstrate that
+this novel approach outperforms existing state-of-the-art methods in both
+objective and subjective assessments, marking a significant leap forward in
+mesh denoising.",cs.CV,['cs.CV']
+Boosting Diffusion Models with Moving Average Sampling in Frequency Domain,Yurui Qian · Qi Cai · Yingwei Pan · Yehao Li · Ting Yao · Qibin Sun · Tao Mei, ,https://arxiv.org/abs/2403.17870,,2403.17870.pdf,Boosting Diffusion Models with Moving Average Sampling in Frequency Domain,"Diffusion models have recently brought a powerful revolution in image
+generation. Despite showing impressive generative capabilities, most of these
+models rely on the current sample to denoise the next one, possibly resulting
+in denoising instability. In this paper, we reinterpret the iterative denoising
+process as model optimization and leverage a moving average mechanism to
+ensemble all the prior samples. Instead of simply applying moving average to
+the denoised samples at different timesteps, we first map the denoised samples
+to data space and then perform moving average to avoid distribution shift
+across timesteps. In view that diffusion models evolve the recovery from
+low-frequency components to high-frequency details, we further decompose the
+samples into different frequency components and execute moving average
+separately on each component. We name the complete approach ""Moving Average
+Sampling in Frequency domain (MASF)"". MASF could be seamlessly integrated into
+mainstream pre-trained diffusion models and sampling schedules. Extensive
+experiments on both unconditional and conditional diffusion models demonstrate
+that our MASF leads to superior performances compared to the baselines, with
+almost negligible additional complexity cost.",cs.CV,"['cs.CV', 'cs.MM']"
+Task-Aware Encoder Control for Deep Video Compression,Xingtong Ge · Jixiang Luo · XINJIE ZHANG · Tongda Xu · Guo Lu · Dailan He · Jing Geng · Yan Wang · Jun Zhang · Hongwei Qin, ,https://arxiv.org/abs/2404.04848,,2404.04848.pdf,Task-Aware Encoder Control for Deep Video Compression,"Prior research on deep video compression (DVC) for machine tasks typically
+necessitates training a unique codec for each specific task, mandating a
+dedicated decoder per task. In contrast, traditional video codecs employ a
+flexible encoder controller, enabling the adaptation of a single codec to
+different tasks through mechanisms like mode prediction. Drawing inspiration
+from this, we introduce an innovative encoder controller for deep video
+compression for machines. This controller features a mode prediction and a
+Group of Pictures (GoP) selection module. Our approach centralizes control at
+the encoding stage, allowing for adaptable encoder adjustments across different
+tasks, such as detection and tracking, while maintaining compatibility with a
+standard pre-trained DVC decoder. Empirical evidence demonstrates that our
+method is applicable across multiple tasks with various existing pre-trained
+DVCs. Moreover, extensive experiments demonstrate that our method outperforms
+previous DVC by about 25% bitrate for different tasks, with only one
+pre-trained decoder.",eess.IV,"['eess.IV', 'cs.AI', 'cs.CV']"
+NEAT: Distilling 3D Wireframes from Neural Attraction Fields,Nan Xue · Bin Tan · Yuxi Xiao · Liang Dong · Gui-Song Xia · Tianfu Wu · Yujun Shen,https://github.com/cherubicXN/neat,https://arxiv.org/abs/2307.10206,,2307.10206.pdf,NEAT: Distilling 3D Wireframes from Neural Attraction Fields,"This paper studies the problem of structured 3D reconstruction using
+wireframes that consist of line segments and junctions, focusing on the
+computation of structured boundary geometries of scenes. Instead of leveraging
+matching-based solutions from 2D wireframes (or line segments) for 3D wireframe
+reconstruction as done in prior arts, we present NEAT, a rendering-distilling
+formulation using neural fields to represent 3D line segments with 2D
+observations, and bipartite matching for perceiving and distilling of a sparse
+set of 3D global junctions. The proposed {NEAT} enjoys the joint optimization
+of the neural fields and the global junctions from scratch, using
+view-dependent 2D observations without precomputed cross-view feature matching.
+Comprehensive experiments on the DTU and BlendedMVS datasets demonstrate our
+NEAT's superiority over state-of-the-art alternatives for 3D wireframe
+reconstruction. Moreover, the distilled 3D global junctions by NEAT, are a
+better initialization than SfM points, for the recently-emerged 3D Gaussian
+Splatting for high-fidelity novel view synthesis using about 20 times fewer
+initial 3D points. Project page: \url{https://xuenan.net/neat}.",cs.CV,"['cs.CV', 'cs.GR']"
+Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval,Jiamian Wang · Guohao Sun · Pichao Wang · Dongfang Liu · Sohail Dianat · MAJID RABBANI · Raghuveer Rao · ZHIQIANG TAO, ,https://arxiv.org/abs/2403.17998,,2403.17998.pdf,Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval,"The increasing prevalence of video clips has sparked growing interest in
+text-video retrieval. Recent advances focus on establishing a joint embedding
+space for text and video, relying on consistent embedding representations to
+compute similarity. However, the text content in existing datasets is generally
+short and concise, making it hard to fully describe the redundant semantics of
+a video. Correspondingly, a single text embedding may be less expressive to
+capture the video embedding and empower the retrieval. In this study, we
+propose a new stochastic text modeling method T-MASS, i.e., text is modeled as
+a stochastic embedding, to enrich text embedding with a flexible and resilient
+semantic range, yielding a text mass. To be specific, we introduce a
+similarity-aware radius module to adapt the scale of the text mass upon the
+given text-video pairs. Plus, we design and develop a support text
+regularization to further control the text mass during the training. The
+inference pipeline is also tailored to fully exploit the text mass for accurate
+retrieval. Empirical evidence suggests that T-MASS not only effectively
+attracts relevant text-video pairs while distancing irrelevant ones, but also
+enables the determination of precise text embeddings for relevant pairs. Our
+experimental results show a substantial improvement of T-MASS over baseline (3%
+to 6.3% by R@1). Also, T-MASS achieves state-of-the-art performance on five
+benchmark datasets, including MSRVTT, LSMDC, DiDeMo, VATEX, and Charades.",cs.CV,['cs.CV']
+Optimizing Diffusion Noise Can Serve As Universal Motion Priors,Korrawe Karunratanakul · Konpat Preechakul · Emre Aksan · Thabo Beeler · Supasorn Suwajanakorn · Siyu Tang,https://korrawe.github.io/dno-project/,https://arxiv.org/abs/2312.11994v1,,2312.11994v1.pdf,Optimizing Diffusion Noise Can Serve As Universal Motion Priors,"We propose Diffusion Noise Optimization (DNO), a new method that effectively
+leverages existing motion diffusion models as motion priors for a wide range of
+motion-related tasks. Instead of training a task-specific diffusion model for
+each new task, DNO operates by optimizing the diffusion latent noise of an
+existing pre-trained text-to-motion model. Given the corresponding latent noise
+of a human motion, it propagates the gradient from the target criteria defined
+on the motion space through the whole denoising process to update the diffusion
+latent noise. As a result, DNO supports any use cases where criteria can be
+defined as a function of motion. In particular, we show that, for motion
+editing and control, DNO outperforms existing methods in both achieving the
+objective and preserving the motion content. DNO accommodates a diverse range
+of editing modes, including changing trajectory, pose, joint locations, or
+avoiding newly added obstacles. In addition, DNO is effective in motion
+denoising and completion, producing smooth and realistic motion from noisy and
+partial inputs. DNO achieves these results at inference time without the need
+for model retraining, offering great versatility for any defined reward or loss
+function on the motion representation.",cs.CV,['cs.CV']
+Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation,Ziyang Chen · Yongsheng Pan · Yiwen Ye · Mengkang Lu · Yong Xia,https://github.com/Chen-Ziyang/VPTTA,https://arxiv.org/abs/2311.18363,,2311.18363.pdf,Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation,"Distribution shift widely exists in medical images acquired from different
+medical centres and poses a significant obstacle to deploying the pre-trained
+semantic segmentation model in real-world applications. Test-time adaptation
+has proven its effectiveness in tackling the cross-domain distribution shift
+during inference. However, most existing methods achieve adaptation by updating
+the pre-trained models, rendering them susceptible to error accumulation and
+catastrophic forgetting when encountering a series of distribution shifts
+(i.e., under the continual test-time adaptation setup). To overcome these
+challenges caused by updating the models, in this paper, we freeze the
+pre-trained model and propose the Visual Prompt-based Test-Time Adaptation
+(VPTTA) method to train a specific prompt for each test image to align the
+statistics in the batch normalization layers. Specifically, we present the
+low-frequency prompt, which is lightweight with only a few parameters and can
+be effectively trained in a single iteration. To enhance prompt initialization,
+we equip VPTTA with a memory bank to benefit the current prompt from previous
+ones. Additionally, we design a warm-up mechanism, which mixes source and
+target statistics to construct warm-up statistics, thereby facilitating the
+training process. Extensive experiments demonstrate the superiority of our
+VPTTA over other state-of-the-art methods on two medical image segmentation
+benchmark tasks. The code and weights of pre-trained source models are
+available at https://github.com/Chen-Ziyang/VPTTA.",cs.CV,['cs.CV']
+A Stealthy Wrongdoer: Feature-Oriented Reconstruction Attack against Split Learning,Xiaoyang Xu · Mengda Yang · Wenzhe Yi · Ziang Li · Juan Wang · Hongxin Hu · Yong ZHUANG · Yaxin Liu, ,https://arxiv.org/abs/2405.04115,,2405.04115.pdf,A Stealthy Wrongdoer: Feature-Oriented Reconstruction Attack against Split Learning,"Split Learning (SL) is a distributed learning framework renowned for its
+privacy-preserving features and minimal computational requirements. Previous
+research consistently highlights the potential privacy breaches in SL systems
+by server adversaries reconstructing training data. However, these studies
+often rely on strong assumptions or compromise system utility to enhance attack
+performance. This paper introduces a new semi-honest Data Reconstruction Attack
+on SL, named Feature-Oriented Reconstruction Attack (FORA). In contrast to
+prior works, FORA relies on limited prior knowledge, specifically that the
+server utilizes auxiliary samples from the public without knowing any client's
+private information. This allows FORA to conduct the attack stealthily and
+achieve robust performance. The key vulnerability exploited by FORA is the
+revelation of the model representation preference in the smashed data output by
+victim client. FORA constructs a substitute client through feature-level
+transfer learning, aiming to closely mimic the victim client's representation
+preference. Leveraging this substitute client, the server trains the attack
+model to effectively reconstruct private data. Extensive experiments showcase
+FORA's superior performance compared to state-of-the-art methods. Furthermore,
+the paper systematically evaluates the proposed method's applicability across
+diverse settings and advanced defense strategies.",cs.CR,['cs.CR']
+Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models,David Stotko · Nils Wandel · Reinhard Klein,https://cg.cs.uni-bonn.de/publication/stotko2024-Physics-guided-SfT,https://arxiv.org/abs/2311.12796,,2311.12796.pdf,Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models,"3D reconstruction of dynamic scenes is a long-standing problem in computer
+graphics and increasingly difficult the less information is available.
+Shape-from-Template (SfT) methods aim to reconstruct a template-based geometry
+from RGB images or video sequences, often leveraging just a single monocular
+camera without depth information, such as regular smartphone recordings.
+Unfortunately, existing reconstruction methods are either unphysical and noisy
+or slow in optimization. To solve this problem, we propose a novel SfT
+reconstruction algorithm for cloth using a pre-trained neural surrogate model
+that is fast to evaluate, stable, and produces smooth reconstructions due to a
+regularizing physics simulation. Differentiable rendering of the simulated mesh
+enables pixel-wise comparisons between the reconstruction and a target video
+sequence that can be used for a gradient-based optimization procedure to
+extract not only shape information but also physical parameters such as
+stretching, shearing, or bending stiffness of the cloth. This allows to retain
+a precise, stable, and smooth reconstructed geometry while reducing the runtime
+by a factor of 400-500 compared to $\phi$-SfT, a state-of-the-art physics-based
+SfT approach.",cs.CV,"['cs.CV', 'cs.LG']"
+Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition,Anqi Zhu · Qiuhong Ke · Mingming Gong · James Bailey, ,https://arxiv.org/abs/2404.07487,,2404.07487.pdf,Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition,"Skeleton-based zero-shot action recognition aims to recognize unknown human
+actions based on the learned priors of the known skeleton-based actions and a
+semantic descriptor space shared by both known and unknown categories. However,
+previous works focus on establishing the bridges between the known skeleton
+representation space and semantic descriptions space at the coarse-grained
+level for recognizing unknown action categories, ignoring the fine-grained
+alignment of these two spaces, resulting in suboptimal performance in
+distinguishing high-similarity action categories. To address these challenges,
+we propose a novel method via Side information and dual-prompts learning for
+skeleton-based zero-shot action recognition (STAR) at the fine-grained level.
+Specifically, 1) we decompose the skeleton into several parts based on its
+topology structure and introduce the side information concerning multi-part
+descriptions of human body movements for alignment between the skeleton and the
+semantic space at the fine-grained level; 2) we design the visual-attribute and
+semantic-part prompts to improve the intra-class compactness within the
+skeleton space and inter-class separability within the semantic space,
+respectively, to distinguish the high-similarity actions. Extensive experiments
+show that our method achieves state-of-the-art performance in ZSL and GZSL
+settings on NTU RGB+D, NTU RGB+D 120, and PKU-MMD datasets.",cs.CV,['cs.CV']
+MICap: A Unified Model for Identity-aware Movie Descriptions,Haran Raajesh · Naveen Reddy Desanur · Zeeshan Khan · Makarand Tapaswi, ,https://arxiv.org/abs/2405.11483,,2405.11483.pdf,MICap: A Unified Model for Identity-aware Movie Descriptions,"Characters are an important aspect of any storyline and identifying and
+including them in descriptions is necessary for story understanding. While
+previous work has largely ignored identity and generated captions with someone
+(anonymized names), recent work formulates id-aware captioning as a
+fill-in-the-blanks (FITB) task, where, given a caption with blanks, the goal is
+to predict person id labels. However, to predict captions with ids, a two-stage
+approach is required: first predict captions with someone, then fill in
+identities. In this work, we present a new single stage approach that can
+seamlessly switch between id-aware caption generation or FITB when given a
+caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared
+auto-regressive decoder that benefits from training with FITB and full-caption
+generation objectives, while the encoder can benefit from or disregard captions
+with blanks as input. Another challenge with id-aware captioning is the lack of
+a metric to capture subtle differences between person ids. To this end, we
+introduce iSPICE, a caption evaluation metric that focuses on identity tuples
+created through intermediate scene graphs. We evaluate MICap on Large-Scale
+Movie Description Challenge (LSMDC), where we show a 4.2% improvement in FITB
+accuracy, and a 1-2% bump in classic captioning metrics.",cs.CV,['cs.CV']
+DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection,Lewei Yao · Renjie Pi · Jianhua Han · Xiaodan Liang · Hang Xu · Wei Zhang · Zhenguo Li · Dan Xu, ,https://arxiv.org/abs/2404.09216,,,DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection,"Existing open-vocabulary object detectors typically require a predefined set
+of categories from users, significantly confining their application scenarios.
+In this paper, we introduce DetCLIPv3, a high-performing detector that excels
+not only at both open-vocabulary object detection, but also generating
+hierarchical labels for detected objects. DetCLIPv3 is characterized by three
+core designs: 1. Versatile model architecture: we derive a robust open-set
+detection framework which is further empowered with generation ability via the
+integration of a caption head. 2. High information density data: we develop an
+auto-annotation pipeline leveraging visual large language model to refine
+captions for large-scale image-text pairs, providing rich, multi-granular
+object labels to enhance the training. 3. Efficient training strategy: we
+employ a pre-training stage with low-resolution inputs that enables the object
+captioner to efficiently learn a broad spectrum of visual concepts from
+extensive image-text paired data. This is followed by a fine-tuning stage that
+leverages a small number of high-resolution samples to further enhance
+detection performance. With these effective designs, DetCLIPv3 demonstrates
+superior open-vocabulary detection performance, \eg, our Swin-T backbone model
+achieves a notable 47.0 zero-shot fixed AP on the LVIS minival benchmark,
+outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP,
+respectively. DetCLIPv3 also achieves a state-of-the-art 19.7 AP in dense
+captioning task on VG dataset, showcasing its strong generative capability.",cs.CV,['cs.CV']
+Label Propagation for Zero-shot Classification with Vision-Language Models,Vladan Stojnić · Yannis Kalantidis · Giorgos Tolias,https://github.com/vladan-stojnic/ZLaP,https://arxiv.org/abs/2404.04072,,2404.04072.pdf,Label Propagation for Zero-shot Classification with Vision-Language Models,"Vision-Language Models (VLMs) have demonstrated impressive performance on
+zero-shot classification, i.e. classification when provided merely with a list
+of class names. In this paper, we tackle the case of zero-shot classification
+in the presence of unlabeled data. We leverage the graph structure of the
+unlabeled data and introduce ZLaP, a method based on label propagation (LP)
+that utilizes geodesic distances for classification. We tailor LP to graphs
+containing both text and image features and further propose an efficient method
+for performing inductive inference based on a dual solution and a
+sparsification step. We perform extensive experiments to evaluate the
+effectiveness of our method on 14 common datasets and show that ZLaP
+outperforms the latest related works. Code:
+https://github.com/vladan-stojnic/ZLaP",cs.CV,"['cs.CV', 'cs.LG']"
+KVQ: Kwai Video Quality Assessment for Short-form Videos,Yiting Lu · Xin Li · Yajing Pei · Kun Yuan · Qizhi Xie · Yunpeng Qu · Ming Sun · Chao Zhou · Zhibo Chen,https://github.com/lixinustc/KVQ-Challenge-CVPR-NTIRE2024,https://arxiv.org/abs/2402.07220,,2402.07220.pdf,KVQ: Kwai Video Quality Assessment for Short-form Videos,"Short-form UGC video platforms, like Kwai and TikTok, have been an emerging
+and irreplaceable mainstream media form, thriving on user-friendly engagement,
+and kaleidoscope creation, etc. However, the advancing content-generation
+modes, e.g., special effects, and sophisticated processing workflows, e.g.,
+de-artifacts, have introduced significant challenges to recent UGC video
+quality assessment: (i) the ambiguous contents hinder the identification of
+quality-determined regions. (ii) the diverse and complicated hybrid distortions
+are hard to distinguish. To tackle the above challenges and assist in the
+development of short-form videos, we establish the first large-scale
+Kaleidoscope short Video database for Quality assessment, termed KVQ, which
+comprises 600 user-uploaded short videos and 3600 processed videos through the
+diverse practical processing workflows, including pre-processing, transcoding,
+and enhancement. Among them, the absolute quality score of each video and
+partial ranking score among indistinguishable samples are provided by a team of
+professional researchers specializing in image processing. Based on this
+database, we propose the first short-form video quality evaluator, i.e., KSVQE,
+which enables the quality evaluator to identify the quality-determined
+semantics with the content understanding of large vision language models (i.e.,
+CLIP) and distinguish the distortions with the distortion understanding module.
+Experimental results have shown the effectiveness of KSVQE on our KVQ database
+and popular VQA databases.",eess.IV,"['eess.IV', 'cs.CV']"
+StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equation,Yining Shi · Kun JIANG · Ke Wang · Jiusi Li · Yunlong Wang · Mengmeng Yang · Diange Yang, ,,https://github.com/keithAND2020/awesome-Occupancy-research,,,,,nan
+HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion,Jingbo Zhang · Xiaoyu Li · Qi Zhang · Yan-Pei Cao · Ying Shan · Jing Liao, ,https://arxiv.org/abs/2311.16961v1,,2311.16961v1.pdf,HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion,"Generating a 3D human model from a single reference image is challenging
+because it requires inferring textures and geometries in invisible views while
+maintaining consistency with the reference image. Previous methods utilizing 3D
+generative models are limited by the availability of 3D training data.
+Optimization-based methods that lift text-to-image diffusion models to 3D
+generation often fail to preserve the texture details of the reference image,
+resulting in inconsistent appearances in different views. In this paper, we
+propose HumanRef, a 3D human generation framework from a single-view input. To
+ensure the generated 3D model is photorealistic and consistent with the input
+image, HumanRef introduces a novel method called reference-guided score
+distillation sampling (Ref-SDS), which effectively incorporates image guidance
+into the generation process. Furthermore, we introduce region-aware attention
+to Ref-SDS, ensuring accurate correspondence between different body regions.
+Experimental results demonstrate that HumanRef outperforms state-of-the-art
+methods in generating 3D clothed humans with fine geometry, photorealistic
+textures, and view-consistent appearances.",cs.CV,['cs.CV']
+SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery,Xin Guo · Jiangwei Lao · Bo Dang · Yingying Zhang · Lei Yu · Lixiang Ru · Liheng Zhong · Ziyuan Huang · Kang Wu · Dingxiang Hu · HUIMEI HE · Jian Wang · Jingdong Chen · Ming Yang · Yongjun Zhang · Yansheng Li, ,https://arxiv.org/abs/2312.10115,,2312.10115.pdf,SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery,"Prior studies on Remote Sensing Foundation Model (RSFM) reveal immense
+potential towards a generic model for Earth Observation. Nevertheless, these
+works primarily focus on a single modality without temporal and geo-context
+modeling, hampering their capabilities for diverse tasks. In this study, we
+present SkySense, a generic billion-scale model, pre-trained on a curated
+multi-modal Remote Sensing Imagery (RSI) dataset with 21.5 million temporal
+sequences. SkySense incorporates a factorized multi-modal spatiotemporal
+encoder taking temporal sequences of optical and Synthetic Aperture Radar (SAR)
+data as input. This encoder is pre-trained by our proposed Multi-Granularity
+Contrastive Learning to learn representations across different modal and
+spatial granularities. To further enhance the RSI representations by the
+geo-context clue, we introduce Geo-Context Prototype Learning to learn
+region-aware prototypes upon RSI's multi-modal spatiotemporal features. To our
+best knowledge, SkySense is the largest Multi-Modal RSFM to date, whose modules
+can be flexibly combined or used individually to accommodate various tasks. It
+demonstrates remarkable generalization capabilities on a thorough evaluation
+encompassing 16 datasets over 7 tasks, from single- to multi-modal, static to
+temporal, and classification to localization. SkySense surpasses 18 recent
+RSFMs in all test scenarios. Specifically, it outperforms the latest models
+such as GFM, SatLas and Scale-MAE by a large margin, i.e., 2.76%, 3.67% and
+3.61% on average respectively. We will release the pre-trained weights to
+facilitate future research and Earth Observation applications.",cs.CV,['cs.CV']
+BioCLIP: A Vision Foundation Model for the Tree of Life,Samuel Stevens · Jiaman Wu · Matthew Thompson · Elizabeth Campolongo · Chan Hee Song · David Carlyn · Li Dong · Wasila Dahdul · Charles Stewart · Tanya Berger-Wolf · Wei-Lun Chao · Yu Su, ,https://arxiv.org/abs/2311.18803,,2311.18803.pdf,BioCLIP: A Vision Foundation Model for the Tree of Life,"Images of the natural world, collected by a variety of cameras, from drones
+to individual phones, are increasingly abundant sources of biological
+information. There is an explosion of computational methods and tools,
+particularly computer vision, for extracting biologically relevant information
+from images for science and conservation. Yet most of these are bespoke
+approaches designed for a specific task and are not easily adaptable or
+extendable to new questions, contexts, and datasets. A vision model for general
+organismal biology questions on images is of timely need. To approach this, we
+curate and release TreeOfLife-10M, the largest and most diverse ML-ready
+dataset of biology images. We then develop BioCLIP, a foundation model for the
+tree of life, leveraging the unique properties of biology captured by
+TreeOfLife-10M, namely the abundance and variety of images of plants, animals,
+and fungi, together with the availability of rich structured biological
+knowledge. We rigorously benchmark our approach on diverse fine-grained biology
+classification tasks and find that BioCLIP consistently and substantially
+outperforms existing baselines (by 16% to 17% absolute). Intrinsic evaluation
+reveals that BioCLIP has learned a hierarchical representation conforming to
+the tree of life, shedding light on its strong generalizability.
+https://imageomics.github.io/bioclip has models, data and code.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+MAFA: Managing False Negatives for Vision-Language Pre-training,Jaeseok Byun · Dohoon Kim · Taesup Moon, ,https://arxiv.org/abs/2312.06112,,2312.06112.pdf,Converting and Smoothing False Negatives for Vision-Language Pre-training,"We consider the critical issue of false negatives in Vision-Language
+Pre-training (VLP), a challenge that arises from the inherent many-to-many
+correspondence of image-text pairs in large-scale web-crawled datasets. The
+presence of false negatives can impede achieving optimal performance and even
+lead to learning failures. To address this challenge, we propose a method
+called COSMO (COnverting and SMOoothing false negatives) that manages the false
+negative issues, especially powerful in hard negative sampling. Building upon
+the recently developed GRouped mIni-baTch sampling (GRIT) strategy, our
+approach consists of two pivotal components: 1) an efficient connection mining
+process that identifies and converts false negatives into positives, and 2)
+label smoothing for the image-text contrastive loss (ITC). Our comprehensive
+experiments verify the effectiveness of COSMO across multiple downstream tasks,
+emphasizing the crucial role of addressing false negatives in VLP, potentially
+even surpassing the importance of addressing false positives. In addition, the
+compatibility of COSMO with the recent BLIP-family model is also demonstrated.",cs.CV,"['cs.CV', 'cs.AI']"
+General Object Foundation Model for Images and Videos at Scale,Junfeng Wu · Yi Jiang · Qihao Liu · Zehuan Yuan · Xiang Bai · Song Bai,https://glee-vision.github.io/,https://arxiv.org/abs/2312.09158,,2312.09158.pdf,General Object Foundation Model for Images and Videos at Scale,"We present GLEE in this work, an object-level foundation model for locating
+and identifying objects in images and videos. Through a unified framework, GLEE
+accomplishes detection, segmentation, tracking, grounding, and identification
+of arbitrary objects in the open world scenario for various object perception
+tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from
+diverse data sources with varying supervision levels to formulate general
+object representations, excelling in zero-shot transfer to new data and tasks.
+Specifically, we employ an image encoder, text encoder, and visual prompter to
+handle multi-modal inputs, enabling to simultaneously solve various
+object-centric downstream tasks while maintaining state-of-the-art performance.
+Demonstrated through extensive training on over five million images from
+diverse benchmarks, GLEE exhibits remarkable versatility and improved
+generalization performance, efficiently tackling downstream tasks without the
+need for task-specific adaptation. By integrating large volumes of
+automatically labeled data, we further enhance its zero-shot generalization
+capabilities. Additionally, GLEE is capable of being integrated into Large
+Language Models, serving as a foundational model to provide universal
+object-level information for multi-modal tasks. We hope that the versatility
+and universality of our method will mark a significant step in the development
+of efficient visual foundation models for AGI systems. The model and code will
+be released at https://glee-vision.github.io .",cs.CV,['cs.CV']
+Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning,Siteng Huang · Biao Gong · Yutong Feng · Zhang Min · Yiliang Lv · Donglin Wang, ,https://arxiv.org/abs/2311.14749,,2311.14749.pdf,Compositional Zero-shot Learning via Progressive Language-based Observations,"Compositional zero-shot learning aims to recognize unseen state-object
+compositions by leveraging known primitives (state and object) during training.
+However, effectively modeling interactions between primitives and generalizing
+knowledge to novel compositions remains a perennial challenge. There are two
+key factors: object-conditioned and state-conditioned variance, i.e., the
+appearance of states (or objects) can vary significantly when combined with
+different objects (or states). For instance, the state ""old"" can signify a
+vintage design for a ""car"" or an advanced age for a ""cat"". In this paper, we
+argue that these variances can be mitigated by predicting composition
+categories based on pre-observed primitive. To this end, we propose Progressive
+Language-based Observations (PLO), which can dynamically determine a better
+observation order of primitives. These observations comprise a series of
+concepts or languages that allow the model to understand image content in a
+step-by-step manner. Specifically, PLO adopts pre-trained vision-language
+models (VLMs) to empower the model with observation capabilities. We further
+devise two variants: 1) PLO-VLM: a two-step method, where a pre-observing
+classifier dynamically determines the observation order of two primitives. 2)
+PLO-LLM: a multi-step scheme, which utilizes large language models (LLMs) to
+craft composition-specific prompts for step-by-step observing. Extensive
+ablations on three challenging datasets demonstrate the superiority of PLO
+compared with state-of-the-art methods, affirming its abilities in
+compositional recognition.",cs.CV,['cs.CV']
+EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models,Sijie Cheng · Zhicheng Guo · Jingwen Wu · Kechen Fang · Peng Li · Huaping Liu · Yang Liu,https://adacheng.github.io/EgoThink/,https://arxiv.org/abs/2311.15596,,2311.15596.pdf,EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models,"Vision-language models (VLMs) have recently shown promising results in
+traditional downstream tasks. Evaluation studies have emerged to assess their
+abilities, with the majority focusing on the third-person perspective, and only
+a few addressing specific tasks from the first-person perspective. However, the
+capability of VLMs to ""think"" from a first-person perspective, a crucial
+attribute for advancing autonomous agents and robotics, remains largely
+unexplored. To bridge this research gap, we introduce EgoThink, a novel visual
+question-answering benchmark that encompasses six core capabilities with twelve
+detailed dimensions. The benchmark is constructed using selected clips from
+egocentric videos, with manually annotated question-answer pairs containing
+first-person information. To comprehensively assess VLMs, we evaluate eighteen
+popular VLMs on EgoThink. Moreover, given the open-ended format of the answers,
+we use GPT-4 as the automatic judge to compute single-answer grading.
+Experimental results indicate that although GPT-4V leads in numerous
+dimensions, all evaluated VLMs still possess considerable potential for
+improvement in first-person perspective tasks. Meanwhile, enlarging the number
+of trainable parameters has the most significant impact on model performance on
+EgoThink. In conclusion, EgoThink serves as a valuable addition to existing
+evaluation benchmarks for VLMs, providing an indispensable resource for future
+research in the realm of embodied artificial intelligence and robotics.",cs.CV,"['cs.CV', 'cs.CL']"
+Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields,Haoyuan Wang · Wenbo Hu · Lei Zhu · Rynson W.H. Lau,https://www.whyy.site/paper/nep,https://arxiv.org/abs/2403.16224,,2403.16224.pdf,Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields,"Inverse rendering aims at recovering both geometry and materials of objects.
+It provides a more compatible reconstruction for conventional rendering
+engines, compared with the neural radiance fields (NeRFs). On the other hand,
+existing NeRF-based inverse rendering methods cannot handle glossy objects with
+local light interactions well, as they typically oversimplify the illumination
+as a 2D environmental map, which assumes infinite lights only. Observing the
+superiority of NeRFs in recovering radiance fields, we propose a novel 5D
+Neural Plenoptic Function (NeP) based on NeRFs and ray tracing, such that more
+accurate lighting-object interactions can be formulated via the rendering
+equation. We also design a material-aware cone sampling strategy to efficiently
+integrate lights inside the BRDF lobes with the help of pre-filtered radiance
+fields. Our method has two stages: the geometry of the target object and the
+pre-filtered environmental radiance fields are reconstructed in the first
+stage, and materials of the target object are estimated in the second stage
+with the proposed NeP and material-aware cone sampling strategy. Extensive
+experiments on the proposed real-world and synthetic datasets demonstrate that
+our method can reconstruct high-fidelity geometry/materials of challenging
+glossy objects with complex lighting interactions from nearby objects. Project
+webpage: https://whyy.site/paper/nep",cs.CV,['cs.CV']
+Collaborating Foundation models for Domain Generalized Semantic Segmentation,Yasser Benigmim · Subhankar Roy · Slim Essid · Vicky Kalogeiton · Stéphane Lathuilière,https://yasserben.github.io/CLOUDS/,https://arxiv.org/abs/2312.09788,,2312.09788.pdf,Collaborating Foundation Models for Domain Generalized Semantic Segmentation,"Domain Generalized Semantic Segmentation (DGSS) deals with training a model
+on a labeled source domain with the aim of generalizing to unseen domains
+during inference. Existing DGSS methods typically effectuate robust features by
+means of Domain Randomization (DR). Such an approach is often limited as it can
+only account for style diversification and not content. In this work, we take
+an orthogonal approach to DGSS and propose to use an assembly of CoLlaborative
+FOUndation models for Domain Generalized Semantic Segmentation (CLOUDS). In
+detail, CLOUDS is a framework that integrates FMs of various kinds: (i) CLIP
+backbone for its robust feature representation, (ii) generative models to
+diversify the content, thereby covering various modes of the possible target
+distribution, and (iii) Segment Anything Model (SAM) for iteratively refining
+the predictions of the segmentation model. Extensive experiments show that our
+CLOUDS excels in adapting from synthetic to real DGSS benchmarks and under
+varying weather conditions, notably outperforming prior methods by 5.6% and
+6.7% on averaged miou, respectively. The code is available at :
+https://github.com/yasserben/CLOUDS",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+CausalPC: Improving the Robustness of Point Cloud Classification by Causal Effect Identification,Yuanmin Huang · Mi Zhang · Daizong Ding · Erling Jiang · Zhaoxiang Wang · Min Yang, ,,https://www.semanticscholar.org/paper/Deep-learning-for-large-scale-point-cloud-in-causal-Zhang-Ji/e1c76c0ba122201e813e3349dc0ebc8bde90eb34,,,,,nan
+Multi-Scale Video Anomaly Detection by Multi-Grained Spatio-Temporal Representation Learning,Menghao Zhang · Jingyu Wang · Qi Qi · Haifeng Sun · Zirui Zhuang · Pengfei Ren · Ruilong Ma · Jianxin Liao, ,https://arxiv.org/abs/2306.10239,,2306.10239.pdf,Multi-scale Spatial-temporal Interaction Network for Video Anomaly Detection,"Video Anomaly Detection (VAD) is an essential yet challenging task in signal
+processing. Since certain anomalies cannot be detected by isolated analysis of
+either temporal or spatial information, the interaction between these two types
+of data is considered crucial for VAD. However, current dual-stream
+architectures either confine this integral interaction to the bottleneck of the
+autoencoder or introduce anomaly-irrelevant background pixels into the
+interactive process, hindering the accuracy of VAD. To address these
+deficiencies, we propose a Multi-scale Spatial-Temporal Interaction Network
+(MSTI-Net) for VAD. First, to prioritize the detection of moving objects in the
+scene and harmonize the substantial semantic discrepancies between the two
+types of data, we propose an Attention-based Spatial-Temporal Fusion Module
+(ASTFM) as a substitute for the conventional direct fusion. Furthermore, we
+inject multi-ASTFM-based connections that bridge the appearance and motion
+streams of the dual-stream network, thus fostering multi-scale spatial-temporal
+interaction. Finally, to bolster the delineation between normal and abnormal
+activities, our system records the regular information in a memory module.
+Experimental results on three benchmark datasets validate the effectiveness of
+our approach, which achieves AUCs of 96.8%, 87.6%, and 73.9% on the UCSD Ped2,
+CUHK Avenue, and ShanghaiTech datasets, respectively.",cs.CV,['cs.CV']
+Data-Free Quantization via Pseudo-label Filtering,Chunxiao Fan · Ziqi Wang · Dan Guo · Meng Wang, ,http://export.arxiv.org/abs/2403.11256,,2403.11256.pdf,Uncertainty-Aware Pseudo-Label Filtering for Source-Free Unsupervised Domain Adaptation,"Source-free unsupervised domain adaptation (SFUDA) aims to enable the
+utilization of a pre-trained source model in an unlabeled target domain without
+access to source data. Self-training is a way to solve SFUDA, where confident
+target samples are iteratively selected as pseudo-labeled samples to guide
+target model learning. However, prior heuristic noisy pseudo-label filtering
+methods all involve introducing extra models, which are sensitive to model
+assumptions and may introduce additional errors or mislabeling. In this work,
+we propose a method called Uncertainty-aware Pseudo-label-filtering Adaptation
+(UPA) to efficiently address this issue in a coarse-to-fine manner. Specially,
+we first introduce a sample selection module named Adaptive Pseudo-label
+Selection (APS), which is responsible for filtering noisy pseudo labels. The
+APS utilizes a simple sample uncertainty estimation method by aggregating
+knowledge from neighboring samples and confident samples are selected as clean
+pseudo-labeled. Additionally, we incorporate Class-Aware Contrastive Learning
+(CACL) to mitigate the memorization of pseudo-label noise by learning robust
+pair-wise representation supervised by pseudo labels. Through extensive
+experiments conducted on three widely used benchmarks, we demonstrate that our
+proposed method achieves competitive performance on par with state-of-the-art
+SFUDA methods. Code is available at https://github.com/chenxi52/UPA.",cs.CV,['cs.CV']
+Adaptive Softassign via Hadamard-Equipped Sinkhorn,Binrui Shen · Qiang Niu · Shengxin Zhu, ,https://arxiv.org/abs/2309.13855,,2309.13855.pdf,Adaptive Softassign via Hadamard-Equipped Sinkhorn,"Softassign is a pivotal method in graph matching and other learning tasks.
+Many softassign-based algorithms exhibit performance sensitivity to a parameter
+in the softassign. However, tuning the parameter is challenging and almost done
+empirically. This paper proposes an adaptive softassign method for graph
+matching by analyzing the relationship between the objective score and the
+parameter. This method can automatically tune the parameter based on a given
+error bound to guarantee accuracy. The Hadamard-Equipped Sinkhorn formulas
+introduced in this study significantly enhance the efficiency and stability of
+the adaptive softassign. Moreover, these formulas can also be used in optimal
+transport problems. The resulting adaptive softassign graph matching algorithm
+enjoys significantly higher accuracy than previous state-of-the-art large graph
+matching algorithms while maintaining comparable efficiency.",math.OC,"['math.OC', 'math.CO']"
+SIGNeRF: Scene Integrated Generation for Neural Radiance Fields,Jan-Niklas Dihlmann · Andreas Engelhardt · Hendrik Lensch,https://signerf.jdihlmann.com/,https://arxiv.org/abs/2401.01647,,2401.01647.pdf,SIGNeRF: Scene Integrated Generation for Neural Radiance Fields,"Advances in image diffusion models have recently led to notable improvements
+in the generation of high-quality images. In combination with Neural Radiance
+Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
+generative 3D approaches are object-centric and applying them to editing
+existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
+approach for fast and controllable NeRF scene editing and scene-integrated
+object generation. A new generative update strategy ensures 3D consistency
+across the edited images, without requiring iterative optimization. We find
+that depth-conditioned diffusion models inherently possess the capability to
+generate 3D consistent views by requesting a grid of images instead of single
+views. Based on these insights, we introduce a multi-view reference sheet of
+modified images. Our method updates an image collection consistently based on
+the reference sheet and refines the original NeRF with the newly generated
+image set in one go. By exploiting the depth conditioning mechanism of the
+image diffusion model, we gain fine control over the spatial location of the
+edit and enforce shape guidance by a selected region or an external mesh.",cs.CV,"['cs.CV', 'cs.GR']"
+Putting the Object Back into Video Object Segmentation,Ho Kei Cheng · Seoung Wug Oh · Brian Price · Joon-Young Lee · Alexander G. Schwing,https://hkchengrex.com/Cutie/,https://arxiv.org/abs/2310.12982,,2310.12982.pdf,Putting the Object Back into Video Object Segmentation,"We present Cutie, a video object segmentation (VOS) network with object-level
+memory reading, which puts the object representation from memory back into the
+video object segmentation result. Recent works on VOS employ bottom-up
+pixel-level memory reading which struggles due to matching noise, especially in
+the presence of distractors, resulting in lower performance in more challenging
+data. In contrast, Cutie performs top-down object-level memory reading by
+adapting a small set of object queries. Via those, it interacts with the
+bottom-up pixel features iteratively with a query-based object transformer (qt,
+hence Cutie). The object queries act as a high-level summary of the target
+object, while high-resolution feature maps are retained for accurate
+segmentation. Together with foreground-background masked attention, Cutie
+cleanly separates the semantics of the foreground object from the background.
+On the challenging MOSE dataset, Cutie improves by 8.7 J&F over XMem with a
+similar running time and improves by 4.2 J&F over DeAOT while being three times
+faster. Code is available at: https://hkchengrex.github.io/Cutie",cs.CV,['cs.CV']
+Generalized Predictive Model for Autonomous Driving,Jiazhi Yang · Shenyuan Gao · Yihang Qiu · Li Chen · Tianyu Li · Bo Dai · Kashyap Chitta · Penghao Wu · Jia Zeng · Ping Luo · Jun Zhang · Andreas Geiger · Yu Qiao · Hongyang Li,https://github.com/OpenDriveLab/DriveAGI,https://arxiv.org/abs/2403.09630,,2403.09630.pdf,Generalized Predictive Model for Autonomous Driving,"In this paper, we introduce the first large-scale video prediction model in
+the autonomous driving discipline. To eliminate the restriction of high-cost
+data collection and empower the generalization ability of our model, we acquire
+massive data from the web and pair it with diverse and high-quality text
+descriptions. The resultant dataset accumulates over 2000 hours of driving
+videos, spanning areas all over the world with diverse weather conditions and
+traffic scenarios. Inheriting the merits from recent latent diffusion models,
+our model, dubbed GenAD, handles the challenging dynamics in driving scenes
+with novel temporal reasoning blocks. We showcase that it can generalize to
+various unseen driving datasets in a zero-shot manner, surpassing general or
+driving-specific video prediction counterparts. Furthermore, GenAD can be
+adapted into an action-conditioned prediction model or a motion planner,
+holding great potential for real-world driving applications.",cs.CV,['cs.CV']
+BlockGCN: Redefine Topology Awareness for Skeleton-Based Action Recognition,Yuxuan Zhou · Xudong Yan · Zhi-Qi Cheng · Yan Yan · Qi Dai · Xian-Sheng Hua,https://github.com/ZhouYuxuanYX/BlockGCN,https://arxiv.org/html/2305.11468v3,,2305.11468v3.pdf,Overcoming Topology Agnosticism: Enhancing Skeleton-Based Action Recognition through Redefined Skeletal Topology Awareness,"Graph Convolutional Networks (GCNs) have long defined the state-of-the-art in
+skeleton-based action recognition, leveraging their ability to unravel the
+complex dynamics of human joint topology through the graph's adjacency matrix.
+However, an inherent flaw has come to light in these cutting-edge models: they
+tend to optimize the adjacency matrix jointly with the model weights. This
+process, while seemingly efficient, causes a gradual decay of bone connectivity
+data, culminating in a model indifferent to the very topology it sought to map.
+As a remedy, we propose a threefold strategy: (1) We forge an innovative
+pathway that encodes bone connectivity by harnessing the power of graph
+distances. This approach preserves the vital topological nuances often lost in
+conventional GCNs. (2) We highlight an oft-overlooked feature - the temporal
+mean of a skeletal sequence, which, despite its modest guise, carries highly
+action-specific information. (3) Our investigation revealed strong variations
+in joint-to-joint relationships across different actions. This finding exposes
+the limitations of a single adjacency matrix in capturing the variations of
+relational configurations emblematic of human movement, which we remedy by
+proposing an efficient refinement to Graph Convolutions (GC) - the BlockGC.
+This evolution slashes parameters by a substantial margin (above 40%), while
+elevating performance beyond original GCNs. Our full model, the BlockGCN,
+establishes new standards in skeleton-based action recognition for small model
+sizes. Its high accuracy, notably on the large-scale NTU RGB+D 120 dataset,
+stand as compelling proof of the efficacy of BlockGCN.",cs.CV,['cs.CV']
+MotionEditor: Editing Video Motion via Content-Aware Diffusion,Shuyuan Tu · Qi Dai · Zhi-Qi Cheng · Han Hu · Xintong Han · Zuxuan Wu · Yu-Gang Jiang, ,https://arxiv.org/abs/2311.18830,,2311.18830.pdf,MotionEditor: Editing Video Motion via Content-Aware Diffusion,"Existing diffusion-based video editing models have made gorgeous advances for
+editing attributes of a source video over time but struggle to manipulate the
+motion information while preserving the original protagonist's appearance and
+background. To address this, we propose MotionEditor, a diffusion model for
+video motion editing. MotionEditor incorporates a novel content-aware motion
+adapter into ControlNet to capture temporal motion correspondence. While
+ControlNet enables direct generation based on skeleton poses, it encounters
+challenges when modifying the source motion in the inverted noise due to
+contradictory signals between the noise (source) and the condition (reference).
+Our adapter complements ControlNet by involving source content to transfer
+adapted control signals seamlessly. Further, we build up a two-branch
+architecture (a reconstruction branch and an editing branch) with a
+high-fidelity attention injection mechanism facilitating branch interaction.
+This mechanism enables the editing branch to query the key and value from the
+reconstruction branch in a decoupled manner, making the editing branch retain
+the original background and protagonist appearance. We also propose a skeleton
+alignment algorithm to address the discrepancies in pose size and position.
+Experiments demonstrate the promising motion editing ability of MotionEditor,
+both qualitatively and quantitatively.",cs.CV,['cs.CV']
+ReconFusion: 3D Reconstruction with Diffusion Priors,Rundi Wu · Ben Mildenhall · Philipp Henzler · Ruiqi Gao · Keunhong Park · Daniel Watson · Pratul P. Srinivasan · Dor Verbin · Jonathan T. Barron · Ben Poole · Aleksander Holynski,https://reconfusion.github.io,https://arxiv.org/abs/2312.02981v1,,2312.02981v1.pdf,ReconFusion: 3D Reconstruction with Diffusion Priors,"3D reconstruction methods such as Neural Radiance Fields (NeRFs) excel at
+rendering photorealistic novel views of complex scenes. However, recovering a
+high-quality NeRF typically requires tens to hundreds of input images,
+resulting in a time-consuming capture process. We present ReconFusion to
+reconstruct real-world scenes using only a few photos. Our approach leverages a
+diffusion prior for novel view synthesis, trained on synthetic and multiview
+datasets, which regularizes a NeRF-based 3D reconstruction pipeline at novel
+camera poses beyond those captured by the set of input images. Our method
+synthesizes realistic geometry and texture in underconstrained regions while
+preserving the appearance of observed regions. We perform an extensive
+evaluation across various real-world datasets, including forward-facing and
+360-degree scenes, demonstrating significant performance improvements over
+previous few-view NeRF reconstruction approaches.",cs.CV,['cs.CV']
+Learning Vision from Models Rivals Learning Vision from Data,Yonglong Tian · Lijie Fan · Kaifeng Chen · Dina Katabi · Dilip Krishnan · Phillip Isola,https://github.com/google-research/syn-rep-learn/tree/main/SynCLR,https://arxiv.org/abs/2312.17742,,2312.17742.pdf,Learning Vision from Models Rivals Learning Vision from Data,"We introduce SynCLR, a novel approach for learning visual representations
+exclusively from synthetic images and synthetic captions, without any real
+data. We synthesize a large dataset of image captions using LLMs, then use an
+off-the-shelf text-to-image model to generate multiple images corresponding to
+each synthetic caption. We perform visual representation learning on these
+synthetic images via contrastive learning, treating images sharing the same
+caption as positive pairs. The resulting representations transfer well to many
+downstream tasks, competing favorably with other general-purpose visual
+representation learners such as CLIP and DINO v2 in image classification tasks.
+Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
+outperforms previous self-supervised methods by a significant margin, e.g.,
+improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.",cs.CV,['cs.CV']
+"Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization",Mainak Singha · Ankit Jha · Shirsha Bose · Ashwin Nair · Moloud Abdar · Biplab Banerjee, ,https://arxiv.org/abs/2404.00710,,2404.00710.pdf,"Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization","We delve into Open Domain Generalization (ODG), marked by domain and category
+shifts between training's labeled source and testing's unlabeled target
+domains. Existing solutions to ODG face limitations due to constrained
+generalizations of traditional CNN backbones and errors in detecting target
+open samples in the absence of prior knowledge. Addressing these pitfalls, we
+introduce ODG-CLIP, harnessing the semantic prowess of the vision-language
+model, CLIP. Our framework brings forth three primary innovations: Firstly,
+distinct from prevailing paradigms, we conceptualize ODG as a multi-class
+classification challenge encompassing both known and novel categories. Central
+to our approach is modeling a unique prompt tailored for detecting unknown
+class samples, and to train this, we employ a readily accessible stable
+diffusion model, elegantly generating proxy images for the open class.
+Secondly, aiming for domain-tailored classification (prompt) weights while
+ensuring a balance of precision and simplicity, we devise a novel visual
+stylecentric prompt learning mechanism. Finally, we infuse images with
+class-discriminative knowledge derived from the prompt space to augment the
+fidelity of CLIP's visual embeddings. We introduce a novel objective to
+safeguard the continuity of this infused semantic intel across domains,
+especially for the shared classes. Through rigorous testing on diverse
+datasets, covering closed and open-set DG contexts, ODG-CLIP demonstrates clear
+supremacy, consistently outpacing peers with performance boosts between 8%-16%.
+Code will be available at https://github.com/mainaksingha01/ODG-CLIP.",cs.CV,['cs.CV']
+Density-guided Translator Boosts Synthetic-to-Real Unsupervised Domain Adaptive Segmentation of 3D Point Clouds,Zhimin Yuan · Wankang Zeng · Yanfei Su · Weiquan Liu · Ming Cheng · Yulan Guo · Cheng Wang,https://github.com/yuan-zm/DGT-ST,https://arxiv.org/abs/2403.18469,,2403.18469.pdf,Density-guided Translator Boosts Synthetic-to-Real Unsupervised Domain Adaptive Segmentation of 3D Point Clouds,"3D synthetic-to-real unsupervised domain adaptive segmentation is crucial to
+annotating new domains. Self-training is a competitive approach for this task,
+but its performance is limited by different sensor sampling patterns (i.e.,
+variations in point density) and incomplete training strategies. In this work,
+we propose a density-guided translator (DGT), which translates point density
+between domains, and integrates it into a two-stage self-training pipeline
+named DGT-ST. First, in contrast to existing works that simultaneously conduct
+data generation and feature/output alignment within unstable adversarial
+training, we employ the non-learnable DGT to bridge the domain gap at the input
+level. Second, to provide a well-initialized model for self-training, we
+propose a category-level adversarial network in stage one that utilizes the
+prototype to prevent negative transfer. Finally, by leveraging the designs
+above, a domain-mixed self-training method with source-aware consistency loss
+is proposed in stage two to narrow the domain gap further. Experiments on two
+synthetic-to-real segmentation tasks (SynLiDAR $\rightarrow$ semanticKITTI and
+SynLiDAR $\rightarrow$ semanticPOSS) demonstrate that DGT-ST outperforms
+state-of-the-art methods, achieving 9.4$\%$ and 4.3$\%$ mIoU improvements,
+respectively. Code is available at \url{https://github.com/yuan-zm/DGT-ST}.",cs.CV,"['cs.CV', 'cs.AI']"
+Absolute Pose from One or Two Scaled and Oriented Features,Jonathan Ventura · Zuzana Kukelova · Torsten Sattler · Daniel Barath,https://github.com/danini/absolute-pose-from-oriented-and-scaled-features,https://arxiv.org/abs/2404.16552,,,Efficient Solution of Point-Line Absolute Pose,"We revisit certain problems of pose estimation based on 3D--2D
+correspondences between features which may be points or lines. Specifically, we
+address the two previously-studied minimal problems of estimating camera
+extrinsics from $p \in \{ 1, 2 \}$ point--point correspondences and $l=3-p$
+line--line correspondences. To the best of our knowledge, all of the
+previously-known practical solutions to these problems required computing the
+roots of degree $\ge 4$ (univariate) polynomials when $p=2$, or degree $\ge 8$
+polynomials when $p=1.$ We describe and implement two elementary solutions
+which reduce the degrees of the needed polynomials from $4$ to $2$ and from $8$
+to $4$, respectively. We show experimentally that the resulting solvers are
+numerically stable and fast: when compared to the previous state-of-the art, we
+may obtain nearly an order of magnitude speedup. The code is available at
+\url{https://github.com/petrhruby97/efficient\_absolute}",cs.CV,"['cs.CV', '68T45', 'I.4.5']"
+IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images,Yushuang Wu · Luyue Shi · Junhao Cai · Weihao Yuan · Lingteng Qiu · Zilong Dong · Liefeng Bo · Shuguang Cui · Xiaoguang Han,https://yushuang-wu.github.io/IPoD/,https://arxiv.org/abs/2404.00269,,2404.00269.pdf,IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images,"Generalizable 3D object reconstruction from single-view RGB-D images remains
+a challenging task, particularly with real-world data. Current state-of-the-art
+methods develop Transformer-based implicit field learning, necessitating an
+intensive learning paradigm that requires dense query-supervision uniformly
+sampled throughout the entire space. We propose a novel approach, IPoD, which
+harmonizes implicit field learning with point diffusion. This approach treats
+the query points for implicit field learning as a noisy point cloud for
+iterative denoising, allowing for their dynamic adaptation to the target object
+shape. Such adaptive query points harness diffusion learning's capability for
+coarse shape recovery and also enhances the implicit representation's ability
+to delineate finer details. Besides, an additional self-conditioning mechanism
+is designed to use implicit predictions as the guidance of diffusion learning,
+leading to a cooperative system. Experiments conducted on the CO3D-v2 dataset
+affirm the superiority of IPoD, achieving 7.8% improvement in F-score and 28.6%
+in Chamfer distance over existing methods. The generalizability of IPoD is also
+demonstrated on the MVImgNet dataset. Our project page is at
+https://yushuang-wu.github.io/IPoD.",cs.CV,['cs.CV']
+MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures,Zhangyang Xiong · Chenghong Li · Kenkun Liu · Hongjie Liao · Jianqiao HU · Junyi Zhu · Shuliang Ning · Lingteng Qiu · Chongjie Wang · Shijie Wang · Shuguang Cui · Xiaoguang Han, ,https://arxiv.org/abs/2312.02963,,2312.02963.pdf,MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures,"In this era, the success of large language models and text-to-image models
+can be attributed to the driving force of large-scale datasets. However, in the
+realm of 3D vision, while remarkable progress has been made with models trained
+on large-scale synthetic and real-captured object data like Objaverse and
+MVImgNet, a similar level of progress has not been observed in the domain of
+human-centric tasks partially due to the lack of a large-scale human dataset.
+Existing datasets of high-fidelity 3D human capture continue to be mid-sized
+due to the significant challenges in acquiring large-scale high-quality 3D
+human data. To bridge this gap, we present MVHumanNet, a dataset that comprises
+multi-view human action sequences of 4,500 human identities. The primary focus
+of our work is on collecting human data that features a large number of diverse
+identities and everyday clothing using a multi-view human capture system, which
+facilitates easily scalable data collection. Our dataset contains 9,000 daily
+outfits, 60,000 motion sequences and 645 million frames with extensive
+annotations, including human masks, camera parameters, 2D and 3D keypoints,
+SMPL/SMPLX parameters, and corresponding textual descriptions. To explore the
+potential of MVHumanNet in various 2D and 3D visual tasks, we conducted pilot
+studies on view-consistent action recognition, human NeRF reconstruction,
+text-driven view-unconstrained human image generation, as well as 2D
+view-unconstrained human image and 3D avatar generation. Extensive experiments
+demonstrate the performance improvements and effective applications enabled by
+the scale provided by MVHumanNet. As the current largest-scale 3D human
+dataset, we hope that the release of MVHumanNet data with annotations will
+foster further innovations in the domain of 3D human-centric tasks at scale.",cs.CV,['cs.CV']
+SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing,Zeyinzi Jiang · Chaojie Mao · Yulin Pan · Zhen Han · Jingfeng Zhang,https://scedit.github.io/,https://arxiv.org/abs/2312.11392,,2312.11392.pdf,SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing,"Image diffusion models have been utilized in various tasks, such as
+text-to-image generation and controllable image synthesis. Recent research has
+introduced tuning methods that make subtle adjustments to the original models,
+yielding promising results in specific adaptations of foundational generative
+diffusion models. Rather than modifying the main backbone of the diffusion
+model, we delve into the role of skip connection in U-Net and reveal that
+hierarchical features aggregating long-distance information across encoder and
+decoder make a significant impact on the content and quality of image
+generation. Based on the observation, we propose an efficient generative tuning
+framework, dubbed SCEdit, which integrates and edits Skip Connection using a
+lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
+allows for straightforward extension to controllable image synthesis by
+injecting different conditions with Controllable SC-Tuner, simplifying and
+unifying the network design for multi-condition inputs. Our SCEdit
+substantially reduces training parameters, memory usage, and computational
+expense due to its lightweight tuners, with backward propagation only passing
+to the decoder blocks. Extensive experiments conducted on text-to-image
+generation and controllable image synthesis tasks demonstrate the superiority
+of our method in terms of efficiency and performance. Project page:
+\url{https://scedit.github.io/}",cs.CV,['cs.CV']
+Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis,Zanlin Ni · Yulin Wang · Renping Zhou · Jiayi Guo · Jinyi Hu · Zhiyuan Liu · Shiji Song · Yuan Yao · Gao Huang, ,https://arxiv.org/html/2312.14988v1,,2312.14988v1.pdf,Emage: Non-Autoregressive Text-to-Image Generation,"Autoregressive and diffusion models drive the recent breakthroughs on
+text-to-image generation. Despite their huge success of generating
+high-realistic images, a common shortcoming of these models is their high
+inference latency - autoregressive models run more than a thousand times
+successively to produce image tokens and diffusion models convert Gaussian
+noise into images with many hundreds of denoising steps. In this work, we
+explore non-autoregressive text-to-image models that efficiently generate
+hundreds of image tokens in parallel. We develop many model variations with
+different learning and inference strategies, initialized text encoders, etc.
+Compared with autoregressive baselines that needs to run one thousand times,
+our model only runs 16 times to generate images of competitive quality with an
+order of magnitude lower inference latency. Our non-autoregressive model with
+346M parameters generates an image of 256$\times$256 with about one second on
+one V100 GPU.",cs.CV,['cs.CV']
+360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model,Qian Wang · Weiqi Li · Chong Mou · Xinhua Cheng · Jian Zhang, ,https://arxiv.org/abs/2401.06578,,2401.06578.pdf,360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model,"Panorama video recently attracts more interest in both study and application,
+courtesy of its immersive experience. Due to the expensive cost of capturing
+360-degree panoramic videos, generating desirable panorama videos by prompts is
+urgently required. Lately, the emerging text-to-video (T2V) diffusion methods
+demonstrate notable effectiveness in standard video generation. However, due to
+the significant gap in content and motion patterns between panoramic and
+standard videos, these methods encounter challenges in yielding satisfactory
+360-degree panoramic videos. In this paper, we propose a pipeline named
+360-Degree Video Diffusion model (360DVD) for generating 360-degree panoramic
+videos based on the given prompts and motion conditions. Specifically, we
+introduce a lightweight 360-Adapter accompanied by 360 Enhancement Techniques
+to transform pre-trained T2V models for panorama video generation. We further
+propose a new panorama dataset named WEB360 consisting of panoramic video-text
+pairs for training 360DVD, addressing the absence of captioned panoramic video
+datasets. Extensive experiments demonstrate the superiority and effectiveness
+of 360DVD for panorama video generation. Our project page is at
+https://akaneqwq.github.io/360DVD/.",cs.CV,['cs.CV']
+All in One Framework for Multimodal Re-identification in the Wild,He Li · Mang Ye · Ming Zhang · Bo Du, ,https://arxiv.org/abs/2405.04741,,2405.04741.pdf,All in One Framework for Multimodal Re-identification in the Wild,"In Re-identification (ReID), recent advancements yield noteworthy progress in
+both unimodal and cross-modal retrieval tasks. However, the challenge persists
+in developing a unified framework that could effectively handle varying
+multimodal data, including RGB, infrared, sketches, and textual information.
+Additionally, the emergence of large-scale models shows promising performance
+in various vision tasks but the foundation model in ReID is still blank. In
+response to these challenges, a novel multimodal learning paradigm for ReID is
+introduced, referred to as All-in-One (AIO), which harnesses a frozen
+pre-trained big model as an encoder, enabling effective multimodal retrieval
+without additional fine-tuning. The diverse multimodal data in AIO are
+seamlessly tokenized into a unified space, allowing the modality-shared frozen
+encoder to extract identity-consistent features comprehensively across all
+modalities. Furthermore, a meticulously crafted ensemble of cross-modality
+heads is designed to guide the learning trajectory. AIO is the \textbf{first}
+framework to perform all-in-one ReID, encompassing four commonly used
+modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not
+only adeptly handles various modal data but also excels in challenging
+contexts, showcasing exceptional performance in zero-shot and domain
+generalization scenarios.",cs.CV,['cs.CV']
+TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models,Zhongwei Zhang · Fuchen Long · Yingwei Pan · Zhaofan Qiu · Ting Yao · Yang Cao · Tao Mei,https://trip-i2v.github.io/TRIP/,https://arxiv.org/abs/2403.17005v1,,2403.17005v1.pdf,TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models,"Recent advances in text-to-video generation have demonstrated the utility of
+powerful diffusion models. Nevertheless, the problem is not trivial when
+shaping diffusion models to animate static image (i.e., image-to-video
+generation). The difficulty originates from the aspect that the diffusion
+process of subsequent animated frames should not only preserve the faithful
+alignment with the given image but also pursue temporal coherence among
+adjacent frames. To alleviate this, we present TRIP, a new recipe of
+image-to-video diffusion paradigm that pivots on image noise prior derived from
+static image to jointly trigger inter-frame relational reasoning and ease the
+coherent temporal modeling via temporal residual learning. Technically, the
+image noise prior is first attained through one-step backward diffusion process
+based on both static image and noised video latent codes. Next, TRIP executes a
+residual-like dual-path scheme for noise prediction: 1) a shortcut path that
+directly takes image noise prior as the reference noise of each frame to
+amplify the alignment between the first frame and subsequent frames; 2) a
+residual path that employs 3D-UNet over noised video and static image latent
+codes to enable inter-frame relational reasoning, thereby easing the learning
+of the residual noise for each frame. Furthermore, both reference and residual
+noise of each frame are dynamically merged via attention mechanism for final
+video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
+datasets demonstrate the effectiveness of our TRIP for image-to-video
+generation. Please see our project page at https://trip-i2v.github.io/TRIP/.",cs.CV,"['cs.CV', 'cs.MM']"
+Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities,Yiyuan Zhang · Xiaohan Ding · Kaixiong Gong · Yixiao Ge · Ying Shan · Xiangyu Yue, ,https://arxiv.org/abs/2401.14405,,2401.14405.pdf,Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities,"We propose to improve transformers of a specific modality with irrelevant
+data from other modalities, e.g., improve an ImageNet model with audio or point
+cloud datasets. We would like to highlight that the data samples of the target
+modality are irrelevant to the other modalities, which distinguishes our method
+from other works utilizing paired (e.g., CLIP) or interleaved data of different
+modalities. We propose a methodology named Multimodal Pathway - given a target
+modality and a transformer designed for it, we use an auxiliary transformer
+trained with data of another modality and construct pathways to connect
+components of the two models so that data of the target modality can be
+processed by both models. In this way, we utilize the universal
+sequence-to-sequence modeling abilities of transformers obtained from two
+modalities. As a concrete implementation, we use a modality-specific tokenizer
+and task-specific head as usual but utilize the transformer blocks of the
+auxiliary model via a proposed method named Cross-Modal Re-parameterization,
+which exploits the auxiliary weights without any inference costs. On the image,
+point cloud, video, and audio recognition tasks, we observe significant and
+consistent performance improvements with irrelevant data from other modalities.
+The code and models are available at https://github.com/AILab-CVC/M2PT.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Geometry Transfer for Stylizing Radiance Fields,Hyunyoung Jung · Seonghyeon Nam · Nikolaos Sarafianos · Sungjoo Yoo · Alexander Sorkine-Hornung · Rakesh Ranjan,https://hyblue.github.io/geo-srf/,https://arxiv.org/abs/2402.00863,,2402.00863.pdf,Geometry Transfer for Stylizing Radiance Fields,"Shape and geometric patterns are essential in defining stylistic identity.
+However, current 3D style transfer methods predominantly focus on transferring
+colors and textures, often overlooking geometric aspects. In this paper, we
+introduce Geometry Transfer, a novel method that leverages geometric
+deformation for 3D style transfer. This technique employs depth maps to extract
+a style guide, subsequently applied to stylize the geometry of radiance fields.
+Moreover, we propose new techniques that utilize geometric cues from the 3D
+scene, thereby enhancing aesthetic expressiveness and more accurately
+reflecting intended styles. Our extensive experiments show that Geometry
+Transfer enables a broader and more expressive range of stylizations, thereby
+significantly expanding the scope of 3D style transfer.",cs.CV,['cs.CV']
+HRVDA: High-Resolution Visual Document Assistant,Chaohu Liu · Kun Yin · Haoyu Cao · Xinghua Jiang · Xin Li · Yinsong Liu · Deqiang Jiang · Xing Sun · Linli Xu, ,https://arxiv.org/abs/2404.06918,,2404.06918.pdf,HRVDA: High-Resolution Visual Document Assistant,"Leveraging vast training data, multimodal large language models (MLLMs) have
+demonstrated formidable general visual comprehension capabilities and achieved
+remarkable performance across various tasks. However, their performance in
+visual document understanding still leaves much room for improvement. This
+discrepancy is primarily attributed to the fact that visual document
+understanding is a fine-grained prediction task. In natural scenes, MLLMs
+typically use low-resolution images, leading to a substantial loss of visual
+information. Furthermore, general-purpose MLLMs do not excel in handling
+document-oriented instructions. In this paper, we propose a High-Resolution
+Visual Document Assistant (HRVDA), which bridges the gap between MLLMs and
+visual document understanding. This model employs a content filtering mechanism
+and an instruction filtering module to separately filter out the
+content-agnostic visual tokens and instruction-agnostic visual tokens, thereby
+achieving efficient model training and inference for high-resolution images. In
+addition, we construct a document-oriented visual instruction tuning dataset
+and apply a multi-stage training strategy to enhance the model's document
+modeling capabilities. Extensive experiments demonstrate that our model
+achieves state-of-the-art performance across multiple document understanding
+datasets, while maintaining training efficiency and inference speed comparable
+to low-resolution models.",cs.CV,['cs.CV']
+TeMO: Towards Text-Driven 3D Stylization for Multi-Object Meshes,Xuying Zhang · Bo-Wen Yin · yuming chen · Zheng Lin · Yunheng Li · Qibin Hou · Ming-Ming Cheng, ,https://arxiv.org/abs/2312.04248,,2312.04248.pdf,TeMO: Towards Text-Driven 3D Stylization for Multi-Object Meshes,"Recent progress in the text-driven 3D stylization of a single object has been
+considerably promoted by CLIP-based methods. However, the stylization of
+multi-object 3D scenes is still impeded in that the image-text pairs used for
+pre-training CLIP mostly consist of an object. Meanwhile, the local details of
+multiple objects may be susceptible to omission due to the existing supervision
+manner primarily relying on coarse-grained contrast of image-text pairs. To
+overcome these challenges, we present a novel framework, dubbed TeMO, to parse
+multi-object 3D scenes and edit their styles under the contrast supervision at
+multiple levels. We first propose a Decoupled Graph Attention (DGA) module to
+distinguishably reinforce the features of 3D surface points. Particularly, a
+cross-modal graph is constructed to align the object points accurately and noun
+phrases decoupled from the 3D mesh and textual description. Then, we develop a
+Cross-Grained Contrast (CGC) supervision system, where a fine-grained loss
+between the words in the textual description and the randomly rendered images
+are constructed to complement the coarse-grained loss. Extensive experiments
+show that our method can synthesize high-quality stylized content and
+outperform the existing methods over a wide range of multi-object 3D meshes.
+Our code and results will be made publicly available",cs.CV,['cs.CV']
+Revisiting Single Image Reflection Removal In the Wild,Yurui Zhu · Bo Li · Xueyang Fu · Peng-Tao Jiang · Hao Zhang · Qibin Sun · Zheng-Jun Zha · Jinwei Chen, ,https://arxiv.org/abs/2311.17320,,2311.17320.pdf,Revisiting Single Image Reflection Removal In the Wild,"This research focuses on the issue of single-image reflection removal (SIRR)
+in real-world conditions, examining it from two angles: the collection pipeline
+of real reflection pairs and the perception of real reflection locations. We
+devise an advanced reflection collection pipeline that is highly adaptable to a
+wide range of real-world reflection scenarios and incurs reduced costs in
+collecting large-scale aligned reflection pairs. In the process, we develop a
+large-scale, high-quality reflection dataset named Reflection Removal in the
+Wild (RRW). RRW contains over 14,950 high-resolution real-world reflection
+pairs, a dataset forty-five times larger than its predecessors. Regarding
+perception of reflection locations, we identify that numerous virtual
+reflection objects visible in reflection images are not present in the
+corresponding ground-truth images. This observation, drawn from the aligned
+pairs, leads us to conceive the Maximum Reflection Filter (MaxRF). The MaxRF
+could accurately and explicitly characterize reflection locations from pairs of
+images. Building upon this, we design a reflection location-aware cascaded
+framework, specifically tailored for SIRR. Powered by these innovative
+techniques, our solution achieves superior performance than current leading
+methods across multiple real-world benchmarks. Codes and datasets will be
+publicly available.",cs.CV,['cs.CV']
+Inlier Confidence Calibration for Point Cloud Registration,Yongzhe Yuan · Yue Wu · Xiaolong Fan · Maoguo Gong · Qiguang Miao · Wenping Ma, ,https://arxiv.org/abs/2307.14019,,2307.14019.pdf,One-Nearest Neighborhood Guides Inlier Estimation for Unsupervised Point Cloud Registration,"The precision of unsupervised point cloud registration methods is typically
+limited by the lack of reliable inlier estimation and self-supervised signal,
+especially in partially overlapping scenarios. In this paper, we propose an
+effective inlier estimation method for unsupervised point cloud registration by
+capturing geometric structure consistency between the source point cloud and
+its corresponding reference point cloud copy. Specifically, to obtain a high
+quality reference point cloud copy, an One-Nearest Neighborhood (1-NN) point
+cloud is generated by input point cloud. This facilitates matching map
+construction and allows for integrating dual neighborhood matching scores of
+1-NN point cloud and input point cloud to improve matching confidence.
+Benefiting from the high quality reference copy, we argue that the neighborhood
+graph formed by inlier and its neighborhood should have consistency between
+source point cloud and its corresponding reference copy. Based on this
+observation, we construct transformation-invariant geometric structure
+representations and capture geometric structure consistency to score the inlier
+confidence for estimated correspondences between source point cloud and its
+reference copy. This strategy can simultaneously provide the reliable
+self-supervised signal for model optimization. Finally, we further calculate
+transformation estimation by the weighted SVD algorithm with the estimated
+correspondences and corresponding inlier confidence. We train the proposed
+model in an unsupervised manner, and extensive experiments on synthetic and
+real-world datasets illustrate the effectiveness of the proposed method.",cs.CV,"['cs.CV', 'cs.AI']"
+Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation,Yeonguk Yu · Sungho Shin · Seunghyeok Back · Minhwan Ko · Sangjun Noh · Kyoobin Lee,https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA,https://arxiv.org/abs/2404.10966v2,,2404.10966v2.pdf,Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation,"Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test
+domain without access to source data after deployment. Existing approaches
+typically rely on self-training with pseudo-labels since ground-truth cannot be
+obtained from test data. Although the quality of pseudo labels is important for
+stable and accurate long-term adaptation, it has not been previously addressed.
+In this work, we propose DPLOT, a simple yet effective TTA framework that
+consists of two components: (1) domain-specific block selection and (2)
+pseudo-label generation using paired-view images. Specifically, we select
+blocks that involve domain-specific feature extraction and train these blocks
+by entropy minimization. After blocks are adjusted for current test domain, we
+generate pseudo-labels by averaging given test images and corresponding flipped
+counterparts. By simply using flip augmentation, we prevent a decrease in the
+quality of the pseudo-labels, which can be caused by the domain gap resulting
+from strong augmentation. Our experimental results demonstrate that DPLOT
+outperforms previous TTA methods in CIFAR10-C, CIFAR100-C, and ImageNet-C
+benchmarks, reducing error by up to 5.4%, 9.1%, and 2.9%, respectively. Also,
+we provide an extensive analysis to demonstrate effectiveness of our framework.
+Code is available at
+https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA.",cs.CV,['cs.CV']
+BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image,Minje Kim · Tae-Kyun Kim,https://yunminjin2.github.io/projects/bitt/,https://arxiv.org/abs/2403.08262,,2403.08262.pdf,BiTT: Bi-directional Texture Reconstruction of Interacting Two Hands from a Single Image,"Creating personalized hand avatars is important to offer a realistic
+experience to users on AR / VR platforms. While most prior studies focused on
+reconstructing 3D hand shapes, some recent work has tackled the reconstruction
+of hand textures on top of shapes. However, these methods are often limited to
+capturing pixels on the visible side of a hand, requiring diverse views of the
+hand in a video or multiple images as input. In this paper, we propose a novel
+method, BiTT(Bi-directional Texture reconstruction of Two hands), which is the
+first end-to-end trainable method for relightable, pose-free texture
+reconstruction of two interacting hands taking only a single RGB image, by
+three novel components: 1) bi-directional (left $\leftrightarrow$ right)
+texture reconstruction using the texture symmetry of left / right hands, 2)
+utilizing a texture parametric model for hand texture recovery, and 3) the
+overall coarse-to-fine stage pipeline for reconstructing personalized texture
+of two interacting hands. BiTT first estimates the scene light condition and
+albedo image from an input image, then reconstructs the texture of both hands
+through the texture parametric model and bi-directional texture reconstructor.
+In experiments using InterHand2.6M and RGB2Hands datasets, our method
+significantly outperforms state-of-the-art hand texture reconstruction methods
+quantitatively and qualitatively. The code is available at
+https://github.com/yunminjin2/BiTT",cs.CV,['cs.CV']
+Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes,Chi-Hsi Kung · 書緯 呂 · Yi-Hsuan Tsai · Yi-Ting Chen, ,https://arxiv.org/abs/2311.17948,,2311.17948.pdf,Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes,"In this paper, we study multi-label atomic activity recognition. Despite the
+notable progress in action recognition, it is still challenging to recognize
+atomic activities due to a deficiency in a holistic understanding of both
+multiple road users' motions and their contextual information. In this paper,
+we introduce Action-slot, a slot attention-based approach that learns visual
+action-centric representations, capturing both motion and contextual
+information. Our key idea is to design action slots that are capable of paying
+attention to regions where atomic activities occur, without the need for
+explicit perception guidance. To further enhance slot attention, we introduce a
+background slot that competes with action slots, aiding the training process in
+avoiding unnecessary focus on background regions devoid of activities. Yet, the
+imbalanced class distribution in the existing dataset hampers the assessment of
+rare activities. To address the limitation, we collect a synthetic dataset
+called TACO, which is four times larger than OATS and features a balanced
+distribution of atomic activities. To validate the effectiveness of our method,
+we conduct comprehensive experiments and ablation studies against various
+action recognition baselines. We also show that the performance of multi-label
+atomic activity recognition on real-world datasets can be improved by
+pretraining representations on TACO. We will release our source code and
+dataset. See the videos of visualization on the project page:
+https://hcis-lab.github.io/Action-slot/",cs.CV,"['cs.CV', 'cs.LG']"
+"Plug-and-Play, Dense-Label-Free Extraction of Open-Vocabulary Semantic Segmentation from Vision-Language Models",Luo Jiayun · Siddhesh Khandelwal · Leonid Sigal · Boyang Li, ,https://arxiv.org/abs/2311.17095v1,,2311.17095v1.pdf,"Plug-and-Play, Dense-Label-Free Extraction of Open-Vocabulary Semantic Segmentation from Vision-Language Models","From an enormous amount of image-text pairs, large-scale vision-language
+models (VLMs) learn to implicitly associate image regions with words, which is
+vital for tasks such as image captioning and visual question answering.
+However, leveraging such pre-trained models for open-vocabulary semantic
+segmentation remains a challenge. In this paper, we propose a simple, yet
+extremely effective, training-free technique, Plug-and-Play Open-Vocabulary
+Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with
+direct text-to-image cross-attention and an image-text matching loss to produce
+semantic segmentation. However, cross-attention alone tends to over-segment,
+whereas cross-attention plus GradCAM tend to under-segment. To alleviate this
+issue, we introduce Salience Dropout; by iteratively dropping patches that the
+model is most attentive to, we are able to better resolve the entire extent of
+the segmentation mask. Compared to existing techniques, the proposed method
+does not require any neural network training and performs hyperparameter tuning
+without the need for any segmentation annotations, even for a validation set.
+PnP-OVSS demonstrates substantial improvements over a comparable baseline
+(+29.4% mIoU on Pascal VOC, +13.2% mIoU on Pascal Context, +14.0% mIoU on MS
+COCO, +2.4% mIoU on COCO Stuff) and even outperforms most baselines that
+conduct additional network training on top of pretrained VLMs.",cs.CV,"['cs.CV', 'cs.AI']"
+Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange,Yanhao Wu · Tong Zhang · Wei Ke · Congpei Qiu · Sabine Süsstrunk · Mathieu Salzmann, ,,https://www.semanticscholar.org/paper/Mitigating-Object-Dependencies:-Improving-Point-Wu-Zhang/1cafd8d79a0e2242cb1f8a2ce26db175785ebf88,,,,,nan
+MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes,Bor Shiun Wang · Chien-Yi Wang · Wei-Chen Chiu,https://eddie221.github.io/MCPNet/,https://arxiv.org/abs/2404.08968,,2404.08968.pdf,MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes,"Recent advancements in post-hoc and inherently interpretable methods have
+markedly enhanced the explanations of black box classifier models. These
+methods operate either through post-analysis or by integrating concept learning
+during model training. Although being effective in bridging the semantic gap
+between a model's latent space and human interpretation, these explanation
+methods only partially reveal the model's decision-making process. The outcome
+is typically limited to high-level semantics derived from the last feature map.
+We argue that the explanations lacking insights into the decision processes at
+low and mid-level features are neither fully faithful nor useful. Addressing
+this gap, we introduce the Multi-Level Concept Prototypes Classifier (MCPNet),
+an inherently interpretable model. MCPNet autonomously learns meaningful
+concept prototypes across multiple feature map levels using Centered Kernel
+Alignment (CKA) loss and an energy-based weighted PCA mechanism, and it does so
+without reliance on predefined concept labels. Further, we propose a novel
+classifier paradigm that learns and aligns multi-level concept prototype
+distributions for classification purposes via Class-aware Concept Distribution
+(CCD) loss. Our experiments reveal that our proposed MCPNet while being
+adaptable to various model architectures, offers comprehensive multi-level
+explanations while maintaining classification accuracy. Additionally, its
+concept distribution-based classification approach shows improved
+generalization capabilities in few-shot classification scenarios.",cs.CV,"['cs.CV', 'cs.LG']"
+OmniMotionGPT: Animal Motion Generation with Limited Data,Zhangsihao Yang · Mingyuan Zhou · Mengyi Shan · Bingbing Wen · Ziwei Xuan · Mitch Hill · Junjie Bai · Guo-Jun Qi · Yalin Wang, ,https://arxiv.org/abs/2311.18303,,2311.18303.pdf,OmniMotionGPT: Animal Motion Generation with Limited Data,"Our paper aims to generate diverse and realistic animal motion sequences from
+textual descriptions, without a large-scale animal text-motion dataset. While
+the task of text-driven human motion synthesis is already extensively studied
+and benchmarked, it remains challenging to transfer this success to other
+skeleton structures with limited data. In this work, we design a model
+architecture that imitates Generative Pretraining Transformer (GPT), utilizing
+prior knowledge learned from human data to the animal domain. We jointly train
+motion autoencoders for both animal and human motions and at the same time
+optimize through the similarity scores among human motion encoding, animal
+motion encoding, and text CLIP embedding. Presenting the first solution to this
+problem, we are able to generate animal motions with high diversity and
+fidelity, quantitatively and qualitatively outperforming the results of
+training human motion generation baselines on animal data. Additionally, we
+introduce AnimalML3D, the first text-animal motion dataset with 1240 animation
+sequences spanning 36 different animal identities. We hope this dataset would
+mediate the data scarcity problem in text-driven animal motion generation,
+providing a new playground for the research community.",cs.CV,['cs.CV']
+Noisy-Correspondence Learning for Text-to-Image Person Re-identification,Yang Qin · Yingke Chen · Dezhong Peng · Xi Peng · Joey Tianyi Zhou · Peng Hu,https://github.com/QinYang79/RDE,https://arxiv.org/abs/2308.09911,,2308.09911.pdf,Noisy-Correspondence Learning for Text-to-Image Person Re-identification,"Text-to-image person re-identification (TIReID) is a compelling topic in the
+cross-modal community, which aims to retrieve the target person based on a
+textual query. Although numerous TIReID methods have been proposed and achieved
+promising performance, they implicitly assume the training image-text pairs are
+correctly aligned, which is not always the case in real-world scenarios. In
+practice, the image-text pairs inevitably exist under-correlated or even
+false-correlated, a.k.a noisy correspondence (NC), due to the low quality of
+the images and annotation errors. To address this problem, we propose a novel
+Robust Dual Embedding method (RDE) that can learn robust visual-semantic
+associations even with NC. Specifically, RDE consists of two main components:
+1) A Confident Consensus Division (CCD) module that leverages the dual-grained
+decisions of dual embedding modules to obtain a consensus set of clean training
+data, which enables the model to learn correct and reliable visual-semantic
+associations. 2) A Triplet Alignment Loss (TAL) relaxes the conventional
+Triplet Ranking loss with the hardest negative samples to a log-exponential
+upper bound over all negative ones, thus preventing the model collapse under NC
+and can also focus on hard-negative samples for promising performance. We
+conduct extensive experiments on three public benchmarks, namely CUHK-PEDES,
+ICFG-PEDES, and RSTPReID, to evaluate the performance and robustness of our
+RDE. Our method achieves state-of-the-art results both with and without
+synthetic noisy correspondences on all three datasets. Code is available at
+https://github.com/QinYang79/RDE.",cs.CV,"['cs.CV', 'cs.MM']"
+SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer,Rui Zhu · Yingwei Pan · Yehao Li · Ting Yao · Zhenglong Sun · Tao Mei · Chang-Wen Chen, ,https://arxiv.org/abs/2403.17004,,2403.17004.pdf,SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer,"Diffusion Transformer (DiT) has emerged as the new trend of generative
+diffusion models on image generation. In view of extremely slow convergence in
+typical DiT, recent breakthroughs have been driven by mask strategy that
+significantly improves the training efficiency of DiT with additional
+intra-image contextual learning. Despite this progress, mask strategy still
+suffers from two inherent limitations: (a) training-inference discrepancy and
+(b) fuzzy relations between mask reconstruction & generative diffusion process,
+resulting in sub-optimal training of DiT. In this work, we address these
+limitations by novelly unleashing the self-supervised discrimination knowledge
+to boost DiT training. Technically, we frame our DiT in a teacher-student
+manner. The teacher-student discriminative pairs are built on the diffusion
+noises along the same Probability Flow Ordinary Differential Equation (PF-ODE).
+Instead of applying mask reconstruction loss over both DiT encoder and decoder,
+we decouple DiT encoder and decoder to separately tackle discriminative and
+generative objectives. In particular, by encoding discriminative pairs with
+student and teacher DiT encoders, a new discriminative loss is designed to
+encourage the inter-image alignment in the self-supervised embedding space.
+After that, student samples are fed into student DiT decoder to perform the
+typical generative diffusion task. Extensive experiments are conducted on
+ImageNet dataset, and our method achieves a competitive balance between
+training cost and generative capacity.",cs.CV,"['cs.CV', 'cs.MM']"
+CoDeF: Content Deformation Fields for Temporally Consistent Video Processing,Hao Ouyang · Qiuyu Wang · Yuxi Xiao · Qingyan Bai · Juntao Zhang · Kecheng Zheng · Xiaowei Zhou · Qifeng Chen · Yujun Shen,https://qiuyu96.github.io/CoDeF/,https://arxiv.org/abs/2308.07926,,2308.07926.pdf,CoDeF: Content Deformation Fields for Temporally Consistent Video Processing,"We present the content deformation field CoDeF as a new type of video
+representation, which consists of a canonical content field aggregating the
+static contents in the entire video and a temporal deformation field recording
+the transformations from the canonical image (i.e., rendered from the canonical
+content field) to each individual frame along the time axis.Given a target
+video, these two fields are jointly optimized to reconstruct it through a
+carefully tailored rendering pipeline.We advisedly introduce some
+regularizations into the optimization process, urging the canonical content
+field to inherit semantics (e.g., the object shape) from the video.With such a
+design, CoDeF naturally supports lifting image algorithms for video processing,
+in the sense that one can apply an image algorithm to the canonical image and
+effortlessly propagate the outcomes to the entire video with the aid of the
+temporal deformation field.We experimentally show that CoDeF is able to lift
+image-to-image translation to video-to-video translation and lift keypoint
+detection to keypoint tracking without any training.More importantly, thanks to
+our lifting strategy that deploys the algorithms on only one image, we achieve
+superior cross-frame consistency in processed videos compared to existing
+video-to-video translation approaches, and even manage to track non-rigid
+objects like water and smog.Project page can be found at
+https://qiuyu96.github.io/CoDeF/.",cs.CV,['cs.CV']
+Action Detection via an Image Diffusion Process,Lin Geng Foo · Tianjiao Li · Hossein Rahmani · Jun Liu, ,https://arxiv.org/abs/2404.01051,,2404.01051.pdf,Action Detection via an Image Diffusion Process,"Action detection aims to localize the starting and ending points of action
+instances in untrimmed videos, and predict the classes of those instances. In
+this paper, we make the observation that the outputs of the action detection
+task can be formulated as images. Thus, from a novel perspective, we tackle
+action detection via a three-image generation process to generate starting
+point, ending point and action-class predictions as images via our proposed
+Action Detection Image Diffusion (ADI-Diff) framework. Furthermore, since our
+images differ from natural images and exhibit special properties, we further
+explore a Discrete Action-Detection Diffusion Process and a Row-Column
+Transformer design to better handle their processing. Our ADI-Diff framework
+achieves state-of-the-art results on two widely-used datasets.",cs.CV,['cs.CV']
+T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory,Daehee Park · Jaeseok Jeong · Sung-Hoon Yoon · Jaewoo Jeong · Kuk-Jin Yoon, ,https://arxiv.org/abs/2403.10052,,2403.10052.pdf,T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory,"Trajectory prediction is a challenging problem that requires considering
+interactions among multiple actors and the surrounding environment. While
+data-driven approaches have been used to address this complex problem, they
+suffer from unreliable predictions under distribution shifts during test time.
+Accordingly, several online learning methods have been proposed using
+regression loss from the ground truth of observed data leveraging the
+auto-labeling nature of trajectory prediction task. We mainly tackle the
+following two issues. First, previous works underfit and overfit as they only
+optimize the last layer of the motion decoder. To this end, we employ the
+masked autoencoder (MAE) for representation learning to encourage complex
+interaction modeling in shifted test distribution for updating deeper layers.
+Second, utilizing the sequential nature of driving data, we propose an
+actor-specific token memory that enables the test-time learning of actor-wise
+motion characteristics. Our proposed method has been validated across various
+challenging cross-dataset distribution shift scenarios including nuScenes,
+Lyft, Waymo, and Interaction. Our method surpasses the performance of existing
+state-of-the-art online learning methods in terms of both prediction accuracy
+and computational efficiency. The code is available at
+https://github.com/daeheepark/T4P.",cs.CV,['cs.CV']
+Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation,Song Wang · Jiawei Yu · Wentong Li · Wenyu Liu · Xiaolu Liu · Junbo Chen · Jianke Zhu,https://github.com/songw-zju/HASSC,https://arxiv.org/abs/2404.11958,,2404.11958.pdf,Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation,"Semantic scene completion, also known as semantic occupancy prediction, can
+provide dense geometric and semantic information for autonomous vehicles, which
+attracts the increasing attention of both academia and industry. Unfortunately,
+existing methods usually formulate this task as a voxel-wise classification
+problem and treat each voxel equally in 3D space during training. As the hard
+voxels have not been paid enough attention, the performance in some challenging
+regions is limited. The 3D dense space typically contains a large number of
+empty voxels, which are easy to learn but require amounts of computation due to
+handling all the voxels uniformly for the existing models. Furthermore, the
+voxels in the boundary region are more challenging to differentiate than those
+in the interior. In this paper, we propose HASSC approach to train the semantic
+scene completion model with hardness-aware design. The global hardness from the
+network optimization process is defined for dynamical hard voxel selection.
+Then, the local hardness with geometric anisotropy is adopted for voxel-wise
+refinement. Besides, self-distillation strategy is introduced to make training
+process stable and consistent. Extensive experiments show that our HASSC scheme
+can effectively promote the accuracy of the baseline model without incurring
+the extra inference cost. Source code is available at:
+https://github.com/songw-zju/HASSC.",cs.CV,"['cs.CV', 'cs.RO']"
+CogAgent: A Visual Language Model for GUI Agents,Wenyi Hong · Weihan Wang · Qingsong Lv · Jiazheng Xu · Wenmeng Yu · Junhui Ji · Yan Wang · Zihan Wang · Yuxiao Dong · Ming Ding · Jie Tang, ,https://arxiv.org/abs/2312.08914,,2312.08914.pdf,CogAgent: A Visual Language Model for GUI Agents,"People are spending an enormous amount of time on digital devices through
+graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
+language models (LLMs) such as ChatGPT can assist people in tasks like writing
+emails, but struggle to understand and interact with GUIs, thus limiting their
+potential to increase automation levels. In this paper, we introduce CogAgent,
+an 18-billion-parameter visual language model (VLM) specializing in GUI
+understanding and navigation. By utilizing both low-resolution and
+high-resolution image encoders, CogAgent supports input at a resolution of
+1120*1120, enabling it to recognize tiny page elements and text. As a
+generalist visual language model, CogAgent achieves the state of the art on
+five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
+Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
+only screenshots as input, outperforms LLM-based methods that consume extracted
+HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
+advancing the state of the art. The model and codes are available at
+https://github.com/THUDM/CogVLM .",cs.CV,['cs.CV']
+Representing Signs as Language: A New Method for Sign Language Translation from Videos,Jia Gong · Lin Geng Foo · Yixuan He · Hossein Rahmani · Jun Liu, ,https://arxiv.org/abs/2404.00925,,2404.00925.pdf,LLMs are Good Sign Language Translators,"Sign Language Translation (SLT) is a challenging task that aims to translate
+sign videos into spoken language. Inspired by the strong translation
+capabilities of large language models (LLMs) that are trained on extensive
+multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT.
+In this paper, we regularize the sign videos to embody linguistic
+characteristics of spoken language, and propose a novel SignLLM framework to
+transform sign videos into a language-like representation for improved
+readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The
+Vector-Quantized Visual Sign module converts sign videos into a sequence of
+discrete character-level sign tokens, and (2) the Codebook Reconstruction and
+Alignment module converts these character-level tokens into word-level sign
+representations using an optimal transport formulation. A sign-text alignment
+loss further bridges the gap between sign and text tokens, enhancing semantic
+compatibility. We achieve state-of-the-art gloss-free results on two
+widely-used SLT benchmarks.",cs.CV,"['cs.CV', 'cs.CL']"
+Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution,Zhikai Chen · Fuchen Long · Zhaofan Qiu · Ting Yao · Wengang Zhou · Jiebo Luo · Tao Mei, ,https://arxiv.org/abs/2403.17000,,2403.17000.pdf,Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution,"Diffusion models are just at a tipping point for image super-resolution task.
+Nevertheless, it is not trivial to capitalize on diffusion models for video
+super-resolution which necessitates not only the preservation of visual
+appearance from low-resolution to high-resolution videos, but also the temporal
+consistency across video frames. In this paper, we propose a novel approach,
+pursuing Spatial Adaptation and Temporal Coherence (SATeCo), for video
+super-resolution. SATeCo pivots on learning spatial-temporal guidance from
+low-resolution videos to calibrate both latent-space high-resolution video
+denoising and pixel-space video reconstruction. Technically, SATeCo freezes all
+the parameters of the pre-trained UNet and VAE, and only optimizes two
+deliberately-designed spatial feature adaptation (SFA) and temporal feature
+alignment (TFA) modules, in the decoder of UNet and VAE. SFA modulates frame
+features via adaptively estimating affine parameters for each pixel,
+guaranteeing pixel-wise guidance for high-resolution frame synthesis. TFA
+delves into feature interaction within a 3D local window (tubelet) through
+self-attention, and executes cross-attention between tubelet and its
+low-resolution counterpart to guide temporal feature alignment. Extensive
+experiments conducted on the REDS4 and Vid4 datasets demonstrate the
+effectiveness of our approach.",cs.CV,"['cs.CV', 'cs.MM']"
+Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving,JINLONG LI · Baolu Li · Zhengzhong Tu · XINYU LIU · Qing Guo · Felix Juefei Xu · Runsheng Xu · Hongkai Yu, ,https://arxiv.org/abs/2404.04804,,2404.04804.pdf,Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving,"Vision-centric perception systems for autonomous driving have gained
+considerable attention recently due to their cost-effectiveness and
+scalability, especially compared to LiDAR-based systems. However, these systems
+often struggle in low-light conditions, potentially compromising their
+performance and safety. To address this, our paper introduces LightDiff, a
+domain-tailored framework designed to enhance the low-light image quality for
+autonomous driving applications. Specifically, we employ a multi-condition
+controlled diffusion model. LightDiff works without any human-collected paired
+data, leveraging a dynamic data degradation process instead. It incorporates a
+novel multi-condition adapter that adaptively controls the input weights from
+different modalities, including depth maps, RGB images, and text captions, to
+effectively illuminate dark scenes while maintaining context consistency.
+Furthermore, to align the enhanced images with the detection model's knowledge,
+LightDiff employs perception-specific scores as rewards to guide the diffusion
+training process through reinforcement learning. Extensive experiments on the
+nuScenes datasets demonstrate that LightDiff can significantly improve the
+performance of several state-of-the-art 3D detectors in night-time conditions
+while achieving high visual quality scores, highlighting its potential to
+safeguard autonomous driving.",cs.CV,['cs.CV']
+Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration,Chen Zhao · Weiling Cai · Chenyu Dong · Chengwei Hu,https://github.com/zhihefang,https://arxiv.org/abs/2311.16845,,2311.16845.pdf,Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration,"Underwater images are subject to intricate and diverse degradation,
+inevitably affecting the effectiveness of underwater visual tasks. However,
+most approaches primarily operate in the raw pixel space of images, which
+limits the exploration of the frequency characteristics of underwater images,
+leading to an inadequate utilization of deep models' representational
+capabilities in producing high-quality images. In this paper, we introduce a
+novel Underwater Image Enhancement (UIE) framework, named WF-Diff, designed to
+fully leverage the characteristics of frequency domain information and
+diffusion models. WF-Diff consists of two detachable networks: Wavelet-based
+Fourier information interaction network (WFI2-net) and Frequency Residual
+Diffusion Adjustment Module (FRDAM). With our full exploration of the frequency
+domain information, WFI2-net aims to achieve preliminary enhancement of
+frequency information in the wavelet space. Our proposed FRDAM can further
+refine the high- and low-frequency information of the initial enhanced images,
+which can be viewed as a plug-and-play universal module to adjust the detail of
+the underwater images. With the above techniques, our algorithm can show SOTA
+performance on real-world underwater image datasets, and achieves competitive
+performance in visual quality.",cs.CV,['cs.CV']
+Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning,Jaewoo Jeong · Daehee Park · Kuk-Jin Yoon, ,https://arxiv.org/abs/2404.05218v1,,2404.05218v1.pdf,Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning,"Human pose forecasting garners attention for its diverse applications.
+However, challenges in modeling the multi-modal nature of human motion and
+intricate interactions among agents persist, particularly with longer
+timescales and more agents. In this paper, we propose an interaction-aware
+trajectory-conditioned long-term multi-agent human pose forecasting model,
+utilizing a coarse-to-fine prediction approach: multi-modal global trajectories
+are initially forecasted, followed by respective local pose forecasts
+conditioned on each mode. In doing so, our Trajectory2Pose model introduces a
+graph-based agent-wise interaction module for a reciprocal forecast of local
+motion-conditioned global trajectory and trajectory-conditioned local pose. Our
+model effectively handles the multi-modality of human motion and the complexity
+of long-term multi-agent interactions, improving performance in complex
+environments. Furthermore, we address the lack of long-term (6s+) multi-agent
+(5+) datasets by constructing a new dataset from real-world images and 2D
+annotations, enabling a comprehensive evaluation of our proposed model.
+State-of-the-art prediction performance on both complex and simpler datasets
+confirms the generalized effectiveness of our method. The code is available at
+https://github.com/Jaewoo97/T2P.",cs.CV,"['cs.CV', 'cs.AI']"
+Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains,Eunsu Baek · Keondo Park · Ji-yoon Kim · Hyung-Sin Kim,https://github.com/Edw2n/ImageNet-ES,https://arxiv.org/abs/2404.15882,,2404.15882.pdf,Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains,"Computer vision applications predict on digital images acquired by a camera
+from physical scenes through light. However, conventional robustness benchmarks
+rely on perturbations in digitized images, diverging from distribution shifts
+occurring in the image acquisition process. To bridge this gap, we introduce a
+new distribution shift dataset, ImageNet-ES, comprising variations in
+environmental and camera sensor factors by directly capturing 202k images with
+a real camera in a controllable testbed. With the new dataset, we evaluate
+out-of-distribution (OOD) detection and model robustness. We find that existing
+OOD detection methods do not cope with the covariate shifts in ImageNet-ES,
+implying that the definition and detection of OOD should be revisited to
+embrace real-world distribution shifts. We also observe that the model becomes
+more robust in both ImageNet-C and -ES by learning environment and sensor
+variations in addition to existing digital augmentations. Lastly, our results
+suggest that effective shift mitigation via camera sensor control can
+significantly improve performance without increasing model size. With these
+findings, our benchmark may aid future research on robustness, OOD, and camera
+sensor control for computer vision. Our code and dataset are available at
+https://github.com/Edw2n/ImageNet-ES.",cs.CV,"['cs.CV', 'cs.AI']"
+Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training,Xiaoyang Wu · Zhuotao Tian · Xin Wen · Bohao Peng · Xihui Liu · Kaicheng Yu · Hengshuang Zhao,https://github.com/Pointcept/Pointcept,https://arxiv.org/abs/2308.09718,,2308.09718.pdf,Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training,"The rapid advancement of deep learning models often attributes to their
+ability to leverage massive training data. In contrast, such privilege has not
+yet fully benefited 3D deep learning, mainly due to the limited availability of
+large-scale 3D datasets. Merging multiple available data sources and letting
+them collaboratively train a single model is a potential solution. However, due
+to the large domain gap between 3D point cloud datasets, such mixed supervision
+could adversely affect the model's performance and lead to degenerated
+performance (i.e., negative transfer) compared to single-dataset training. In
+view of this challenge, we introduce Point Prompt Training (PPT), a novel
+framework for multi-dataset synergistic learning in the context of 3D
+representation learning that supports multiple pre-training paradigms. Based on
+this framework, we propose Prompt-driven Normalization, which adapts the model
+to different datasets with domain-specific prompts and Language-guided
+Categorical Alignment that decently unifies the multiple-dataset label spaces
+by leveraging the relationship between label text. Extensive experiments verify
+that PPT can overcome the negative transfer associated with synergistic
+learning and produce generalizable representations. Notably, it achieves
+state-of-the-art performance on each dataset using a single weight-shared model
+with supervised multi-dataset training. Moreover, when served as a pre-training
+framework, it outperforms other pre-training approaches regarding
+representation quality and attains remarkable state-of-the-art performance
+across over ten diverse downstream tasks spanning both indoor and outdoor 3D
+scenarios.",cs.CV,['cs.CV']
+How to Train Neural Field Representations: A Comprehensive Study and Benchmark,Samuele Papa · Riccardo Valperga · David Knigge · Miltiadis Kofinas · Phillip Lippe · Jan-Jakob Sonke · Efstratios Gavves,https://fit-a-nef.github.io/,https://arxiv.org/abs/2312.10531,,2312.10531.pdf,How to Train Neural Field Representations: A Comprehensive Study and Benchmark,"Neural fields (NeFs) have recently emerged as a versatile method for modeling
+signals of various modalities, including images, shapes, and scenes.
+Subsequently, a number of works have explored the use of NeFs as
+representations for downstream tasks, e.g. classifying an image based on the
+parameters of a NeF that has been fit to it. However, the impact of the NeF
+hyperparameters on their quality as downstream representation is scarcely
+understood and remains largely unexplored. This is in part caused by the large
+amount of time required to fit datasets of neural fields.
+  In this work, we propose $\verb|fit-a-nef|$, a JAX-based library that
+leverages parallelization to enable fast optimization of large-scale NeF
+datasets, resulting in a significant speed-up. With this library, we perform a
+comprehensive study that investigates the effects of different hyperparameters
+-- including initialization, network architecture, and optimization strategies
+-- on fitting NeFs for downstream tasks. Our study provides valuable insights
+on how to train NeFs and offers guidance for optimizing their effectiveness in
+downstream applications. Finally, based on the proposed library and our
+analysis, we propose Neural Field Arena, a benchmark consisting of neural field
+variants of popular vision datasets, including MNIST, CIFAR, variants of
+ImageNet, and ShapeNetv2. Our library and the Neural Field Arena will be
+open-sourced to introduce standardized benchmarking and promote further
+research on neural fields.",cs.CV,['cs.CV']
+Towards Memorization-Free Diffusion Models,Chen Chen · Daochang Liu · Chang Xu,https://chenchen-usyd.github.io/AMG-Project-Page/,https://arxiv.org/abs/2404.00922,,2404.00922.pdf,Towards Memorization-Free Diffusion Models,"Pretrained diffusion models and their outputs are widely accessible due to
+their exceptional capacity for synthesizing high-quality images and their
+open-source nature. The users, however, may face litigation risks owing to the
+models' tendency to memorize and regurgitate training data during inference. To
+address this, we introduce Anti-Memorization Guidance (AMG), a novel framework
+employing three targeted guidance strategies for the main causes of
+memorization: image and caption duplication, and highly specific user prompts.
+Consequently, AMG ensures memorization-free outputs while maintaining high
+image quality and text alignment, leveraging the synergy of its guidance
+methods, each indispensable in its own right. AMG also features an innovative
+automatic detection system for potential memorization during each step of
+inference process, allows selective application of guidance strategies,
+minimally interfering with the original sampling process to preserve output
+utility. We applied AMG to pretrained Denoising Diffusion Probabilistic Models
+(DDPM) and Stable Diffusion across various generation tasks. The results
+demonstrate that AMG is the first approach to successfully eradicates all
+instances of memorization with no or marginal impacts on image quality and
+text-alignment, as evidenced by FID and CLIP scores.",cs.CV,['cs.CV']
+Gradient Alignment for Cross-domain Face Anti-Spoofing,MINH BINH LE · Simon Woo,https://github.com/Leminhbinh0209/CVPR24-FAS,https://arxiv.org/abs/2402.18817,,2402.18817.pdf,Gradient Alignment for Cross-Domain Face Anti-Spoofing,"Recent advancements in domain generalization (DG) for face anti-spoofing
+(FAS) have garnered considerable attention. Traditional methods have focused on
+designing learning objectives and additional modules to isolate domain-specific
+features while retaining domain-invariant characteristics in their
+representations. However, such approaches often lack guarantees of consistent
+maintenance of domain-invariant features or the complete removal of
+domain-specific features. Furthermore, most prior works of DG for FAS do not
+ensure convergence to a local flat minimum, which has been shown to be
+advantageous for DG. In this paper, we introduce GAC-FAS, a novel learning
+objective that encourages the model to converge towards an optimal flat minimum
+without necessitating additional learning modules. Unlike conventional
+sharpness-aware minimizers, GAC-FAS identifies ascending points for each domain
+and regulates the generalization gradient updates at these points to align
+coherently with empirical risk minimization (ERM) gradient updates. This unique
+approach specifically guides the model to be robust against domain shifts. We
+demonstrate the efficacy of GAC-FAS through rigorous testing on challenging
+cross-domain FAS datasets, where it establishes state-of-the-art performance.
+The code is available at https://github.com/leminhbinh0209/CVPR24-FAS.",cs.CV,['cs.CV']
+MANUS: Markerless Grasp Capture using Articulated 3D Gaussians,Chandradeep Pokhariya · Ishaan Shah · Angela Xing · Zekun Li · Kefan Chen · Avinash Sharma · Srinath Sridhar,https://ivl.cs.brown.edu/research/manus.html,https://arxiv.org/abs/2312.02137,,2312.02137.pdf,MANUS: Markerless Grasp Capture using Articulated 3D Gaussians,"Understanding how we grasp objects with our hands has important applications
+in areas like robotics and mixed reality. However, this challenging problem
+requires accurate modeling of the contact between hands and objects. To capture
+grasps, existing methods use skeletons, meshes, or parametric models that does
+not represent hand shape accurately resulting in inaccurate contacts. We
+present MANUS, a method for Markerless Hand-Object Grasp Capture using
+Articulated 3D Gaussians. We build a novel articulated 3D Gaussians
+representation that extends 3D Gaussian splatting for high-fidelity
+representation of articulating hands. Since our representation uses Gaussian
+primitives, it enables us to efficiently and accurately estimate contacts
+between the hand and the object. For the most accurate results, our method
+requires tens of camera views that current datasets do not provide. We
+therefore build MANUS-Grasps, a new dataset that contains hand-object grasps
+viewed from 50+ cameras across 30+ scenes, 3 subjects, and comprising over 7M
+frames. In addition to extensive qualitative results, we also show that our
+method outperforms others on a quantitative contact evaluation method that uses
+paint transfer from the object to the hand.",cs.CV,['cs.CV']
+Language-guided Image Reflection Separation,Haofeng Zhong · Yuchen Hong · Shuchen Weng · Jinxiu Liang · Boxin Shi, ,https://arxiv.org/abs/2402.11874,,2402.11874.pdf,Language-guided Image Reflection Separation,"This paper studies the problem of language-guided reflection separation,
+which aims at addressing the ill-posed reflection separation problem by
+introducing language descriptions to provide layer content. We propose a
+unified framework to solve this problem, which leverages the cross-attention
+mechanism with contrastive learning strategies to construct the correspondence
+between language descriptions and image layers. A gated network design and a
+randomized training strategy are employed to tackle the recognizable layer
+ambiguity. The effectiveness of the proposed method is validated by the
+significant performance advantage over existing reflection separation methods
+on both quantitative and qualitative comparisons.",cs.CV,['cs.CV']
+A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion,Feng Yu · Teng Zhang · Gilad Lerman, ,https://arxiv.org/abs/2404.11590,,2404.11590.pdf,A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion,"We present the subspace-constrained Tyler's estimator (STE) designed for
+recovering a low-dimensional subspace within a dataset that may be highly
+corrupted with outliers. STE is a fusion of the Tyler's M-estimator (TME) and a
+variant of the fast median subspace. Our theoretical analysis suggests that,
+under a common inlier-outlier model, STE can effectively recover the underlying
+subspace, even when it contains a smaller fraction of inliers relative to other
+methods in the field of robust subspace recovery. We apply STE in the context
+of Structure from Motion (SfM) in two ways: for robust estimation of the
+fundamental matrix and for the removal of outlying cameras, enhancing the
+robustness of the SfM pipeline. Numerical experiments confirm the
+state-of-the-art performance of our method in these applications. This research
+makes significant contributions to the field of robust subspace recovery,
+particularly in the context of computer vision and 3D reconstruction.",cs.CV,['cs.CV']
+CNC-Net: Self-Supervised Learning for CNC Machining Operations,Mohsen Yavartanoo · Sangmin Hong · Reyhaneh Neshatavar · Kyoung Mu Lee,https://github.com/myavartanoo/CNC-Net_PyTorch,https://arxiv.org/abs/2312.09925,,2312.09925.pdf,CNC-Net: Self-Supervised Learning for CNC Machining Operations,"CNC manufacturing is a process that employs computer numerical control (CNC)
+machines to govern the movements of various industrial tools and machinery,
+encompassing equipment ranging from grinders and lathes to mills and CNC
+routers. However, the reliance on manual CNC programming has become a
+bottleneck, and the requirement for expert knowledge can result in significant
+costs. Therefore, we introduce a pioneering approach named CNC-Net,
+representing the use of deep neural networks (DNNs) to simulate CNC machines
+and grasp intricate operations when supplied with raw materials. CNC-Net
+constitutes a self-supervised framework that exclusively takes an input 3D
+model and subsequently generates the essential operation parameters required by
+the CNC machine to construct the object. Our method has the potential to
+transformative automation in manufacturing by offering a cost-effective
+alternative to the high costs of manual CNC programming while maintaining
+exceptional precision in 3D object production. Our experiments underscore the
+effectiveness of our CNC-Net in constructing the desired 3D objects through the
+utilization of CNC operations. Notably, it excels in preserving finer local
+details, exhibiting a marked enhancement in precision compared to the
+state-of-the-art 3D CAD reconstruction approaches.",cs.CV,['cs.CV']
+Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation,Xiyi Chen · Marko Mihajlovic · Shaofei Wang · Sergey Prokudin · Siyu Tang, ,https://arxiv.org/abs/2401.04728,,2401.04728.pdf,Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation,"Recent advances in generative diffusion models have enabled the previously
+unfeasible capability of generating 3D assets from a single input image or a
+text prompt. In this work, we aim to enhance the quality and functionality of
+these models for the task of creating controllable, photorealistic human
+avatars. We achieve this by integrating a 3D morphable model into the
+state-of-the-art multi-view-consistent diffusion approach. We demonstrate that
+accurate conditioning of a generative pipeline on the articulated 3D model
+enhances the baseline model performance on the task of novel view synthesis
+from a single image. More importantly, this integration facilitates a seamless
+and accurate incorporation of facial expression and body pose control into the
+generation process. To the best of our knowledge, our proposed framework is the
+first diffusion model to enable the creation of fully 3D-consistent,
+animatable, and photorealistic human avatars from a single image of an unseen
+subject; extensive quantitative and qualitative evaluations demonstrate the
+advantages of our approach over existing state-of-the-art avatar creation
+models on both novel view and novel expression synthesis tasks. The code for
+our project is publicly available.",cs.CV,"['cs.CV', 'cs.AI']"
+3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting,Zhiyin Qian · Shaofei Wang · Marko Mihajlovic · Andreas Geiger · Siyu Tang, ,https://arxiv.org/abs/2312.09228,,2312.09228.pdf,3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting,"We introduce an approach that creates animatable human avatars from monocular
+videos using 3D Gaussian Splatting (3DGS). Existing methods based on neural
+radiance fields (NeRFs) achieve high-quality novel-view/novel-pose image
+synthesis but often require days of training, and are extremely slow at
+inference time. Recently, the community has explored fast grid structures for
+efficient training of clothed avatars. Albeit being extremely fast at training,
+these methods can barely achieve an interactive rendering frame rate with
+around 15 FPS. In this paper, we use 3D Gaussian Splatting and learn a
+non-rigid deformation network to reconstruct animatable clothed human avatars
+that can be trained within 30 minutes and rendered at real-time frame rates
+(50+ FPS). Given the explicit nature of our representation, we further
+introduce as-isometric-as-possible regularizations on both the Gaussian mean
+vectors and the covariance matrices, enhancing the generalization of our model
+on highly articulated unseen poses. Experimental results show that our method
+achieves comparable and even better performance compared to state-of-the-art
+approaches on animatable avatar creation from a monocular input, while being
+400x and 250x faster in training and inference, respectively.",cs.CV,['cs.CV']
+TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models,Yushi Huang · Ruihao Gong · Jing Liu · Tianlong Chen · Xianglong Liu,https://github.com/ModelTC/TFMQ-DM,https://arxiv.org/abs/2311.16503,,2311.16503.pdf,TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models,"The Diffusion model, a prevalent framework for image generation, encounters
+significant challenges in terms of broad applicability due to its extended
+inference times and substantial memory requirements. Efficient Post-training
+Quantization (PTQ) is pivotal for addressing these issues in traditional
+models. Different from traditional models, diffusion models heavily depend on
+the time-step $t$ to achieve satisfactory multi-round denoising. Usually, $t$
+from the finite set $\{1, \ldots, T\}$ is encoded to a temporal feature by a
+few modules totally irrespective of the sampling data. However, existing PTQ
+methods do not optimize these modules separately. They adopt inappropriate
+reconstruction targets and complex calibration methods, resulting in a severe
+disturbance of the temporal feature and denoising trajectory, as well as a low
+compression efficiency. To solve these, we propose a Temporal Feature
+Maintenance Quantization (TFMQ) framework building upon a Temporal Information
+Block which is just related to the time-step $t$ and unrelated to the sampling
+data. Powered by the pioneering block design, we devise temporal information
+aware reconstruction (TIAR) and finite set calibration (FSC) to align the
+full-precision temporal features in a limited time. Equipped with the
+framework, we can maintain the most temporal information and ensure the
+end-to-end generation quality. Extensive experiments on various datasets and
+diffusion models prove our state-of-the-art results. Remarkably, our
+quantization approach, for the first time, achieves model performance nearly on
+par with the full-precision model under 4-bit weight quantization.
+Additionally, our method incurs almost no extra computational cost and
+accelerates quantization time by $2.0 \times$ on LSUN-Bedrooms $256 \times 256$
+compared to previous works. Our code is publicly available at
+https://github.com/ModelTC/TFMQ-DM.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+"Point Transformer V3: Simpler, Faster, Stronger",Xiaoyang Wu · Li Jiang · Peng-Shuai Wang · Zhijian Liu · Xihui Liu · Yu Qiao · Wanli Ouyang · Tong He · Hengshuang Zhao,https://github.com/Pointcept/PointTransformerV3,https://arxiv.org/abs/2312.10035,,2312.10035.pdf,"Point Transformer V3: Simpler, Faster, Stronger","This paper is not motivated to seek innovation within the attention
+mechanism. Instead, it focuses on overcoming the existing trade-offs between
+accuracy and efficiency within the context of point cloud processing,
+leveraging the power of scale. Drawing inspiration from recent advances in 3D
+large-scale representation learning, we recognize that model performance is
+more influenced by scale than by intricate design. Therefore, we present Point
+Transformer V3 (PTv3), which prioritizes simplicity and efficiency over the
+accuracy of certain mechanisms that are minor to the overall performance after
+scaling, such as replacing the precise neighbor search by KNN with an efficient
+serialized neighbor mapping of point clouds organized with specific patterns.
+This principle enables significant scaling, expanding the receptive field from
+16 to 1024 points while remaining efficient (a 3x increase in processing speed
+and a 10x improvement in memory efficiency compared with its predecessor,
+PTv2). PTv3 attains state-of-the-art results on over 20 downstream tasks that
+span both indoor and outdoor scenarios. Further enhanced with multi-dataset
+joint training, PTv3 pushes these results to a higher level.",cs.CV,['cs.CV']
+Efficient Stitchable Task Adaptation,Haoyu He · Zizheng Pan · Jing Liu · Jianfei Cai · Bohan Zhuang, ,https://arxiv.org/abs/2311.17352,,2311.17352.pdf,Efficient Stitchable Task Adaptation,"The paradigm of pre-training and fine-tuning has laid the foundation for
+deploying deep learning models. However, most fine-tuning methods are designed
+to meet a specific resource budget. Recently, considering diverse deployment
+scenarios with various resource budgets, stitchable neural network (SN-Net) is
+introduced to quickly obtain numerous new networks (stitches) from the
+pre-trained models (anchors) in a model family via model stitching. Although
+promising, SN-Net confronts new challenges when adapting it to new target
+domains, including huge memory and storage requirements and a long and
+sub-optimal multistage adaptation process. In this work, we present a novel
+framework, Efficient Stitchable Task Adaptation (ESTA), to efficiently produce
+a palette of fine-tuned models that adhere to diverse resource constraints.
+Specifically, we first tailor parameter-efficient fine-tuning to share low-rank
+updates among the stitches while maintaining independent bias terms. In this
+way, we largely reduce fine-tuning memory burdens and mitigate the interference
+among stitches that arises in task adaptation. Furthermore, we streamline a
+simple yet effective one-stage deployment pipeline, which estimates the
+important stitches to deploy with training-time gradient statistics. By
+assigning higher sampling probabilities to important stitches, we also get a
+boosted Pareto frontier. Extensive experiments on 25 downstream visual
+recognition tasks demonstrate that our ESTA is capable of generating stitches
+with smooth accuracy-efficiency trade-offs and surpasses the direct SN-Net
+adaptation by remarkable margins with significantly lower training time and
+fewer trainable parameters. Furthermore, we demonstrate the flexibility and
+scalability of our ESTA framework by stitching LLMs from LLaMA family,
+obtaining chatbot stitches of assorted sizes.",cs.LG,"['cs.LG', 'cs.CL', 'cs.CV']"
+CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement,Qiang Zhu · Jinhua Hao · Yukang Ding · Yu Liu · Qiao Mo · Ming Sun · Chao Zhou · Shuyuan Zhu, ,https://arxiv.org/abs/2403.10362,,2403.10362.pdf,CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement,"Recently, numerous approaches have achieved notable success in compressed
+video quality enhancement (VQE). However, these methods usually ignore the
+utilization of valuable coding priors inherently embedded in compressed videos,
+such as motion vectors and residual frames, which carry abundant temporal and
+spatial information. To remedy this problem, we propose the Coding
+Priors-Guided Aggregation (CPGA) network to utilize temporal and spatial
+information from coding priors. The CPGA mainly consists of an inter-frame
+temporal aggregation (ITA) module and a multi-scale non-local aggregation (MNA)
+module. Specifically, the ITA module aggregates temporal information from
+consecutive frames and coding priors, while the MNA module globally captures
+spatial information guided by residual frames. In addition, to facilitate
+research in VQE task, we newly construct the Video Coding Priors (VCP) dataset,
+comprising 300 videos with various coding priors extracted from corresponding
+bitstreams. It remedies the shortage of previous datasets on the lack of coding
+information. Experimental results demonstrate the superiority of our method
+compared to existing state-of-the-art methods. The code and dataset will be
+released at https://github.com/CPGA/CPGA.git.",eess.IV,"['eess.IV', 'cs.CV']"
+WonderJourney: Going from Anywhere to Everywhere,Hong-Xing Yu · Haoyi Duan · Junhwa Hur · Kyle Sargent · Michael Rubinstein · William Freeman · Forrester Cole · Deqing Sun · Noah Snavely · Jiajun Wu · Charles Herrmann, ,https://arxiv.org/abs/2312.03884,,,WonderJourney: Going from Anywhere to Everywhere,"We introduce WonderJourney, a modularized framework for perpetual 3D scene
+generation. Unlike prior work on view generation that focuses on a single type
+of scenes, we start at any user-provided location (by a text description or an
+image) and generate a journey through a long sequence of diverse yet coherently
+connected 3D scenes. We leverage an LLM to generate textual descriptions of the
+scenes in this journey, a text-driven point cloud generation pipeline to make a
+compelling and coherent sequence of 3D scenes, and a large VLM to verify the
+generated scenes. We show compelling, diverse visual results across various
+scene types and styles, forming imaginary ""wonderjourneys"". Project website:
+https://kovenyu.com/WonderJourney/",cs.CV,"['cs.CV', 'cs.GR']"
+Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge,Dongjin Kim · Sung Jin Um · Sangmin Lee · Jung Uk Kim, ,https://arxiv.org/abs/2403.17420,,2403.17420.pdf,Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge,"The goal of the multi-sound source localization task is to localize sound
+sources from the mixture individually. While recent multi-sound source
+localization methods have shown improved performance, they face challenges due
+to their reliance on prior information about the number of objects to be
+separated. In this paper, to overcome this limitation, we present a novel
+multi-sound source localization method that can perform localization without
+prior knowledge of the number of sound sources. To achieve this goal, we
+propose an iterative object identification (IOI) module, which can recognize
+sound-making objects in an iterative manner. After finding the regions of
+sound-making objects, we devise object similarity-aware clustering (OSC) loss
+to guide the IOI module to effectively combine regions of the same object but
+also distinguish between different objects and backgrounds. It enables our
+method to perform accurate localization of sound-making objects without any
+prior knowledge. Extensive experimental results on the MUSIC and VGGSound
+benchmarks show the significant performance improvements of the proposed method
+over the existing methods for both single and multi-source. Our code is
+available at: https://github.com/VisualAIKHU/NoPrior_MultiSSL",cs.CV,"['cs.CV', 'cs.MM', 'cs.SD', 'eess.AS']"
+Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation,Qiyuan Dai · Sibei Yang, ,,https://paperswithcode.com/paper/curriculum-point-prompting-for-weakly,,,,,nan
+Osprey: Pixel Understanding with Visual Instruction Tuning,Yuqian Yuan · Wentong Li · Jian liu · Dongqi Tang · Xinjie Luo · Chi Qin · Lei Zhang · Jianke Zhu,https://github.com/CircleRadon/Osprey,https://arxiv.org/abs/2312.10032,,2312.10032.pdf,Osprey: Pixel Understanding with Visual Instruction Tuning,"Multimodal large language models (MLLMs) have recently achieved impressive
+general-purpose vision-language capabilities through visual instruction tuning.
+However, current MLLMs primarily focus on image-level or box-level
+understanding, falling short in achieving fine-grained vision-language
+alignment at pixel level. Besides, the lack of mask-based instruction data
+limits their advancements. In this paper, we propose Osprey, a mask-text
+instruction tuning approach, to extend MLLMs by incorporating fine-grained mask
+regions into language instruction, aiming at achieving pixel-wise visual
+understanding. To achieve this goal, we first meticulously curate a mask-based
+region-text dataset with 724K samples, and then design a vision-language model
+by injecting pixel-level representation into LLM. Specifically, Osprey adopts a
+convolutional CLIP backbone as the vision encoder and employs a mask-aware
+visual extractor to extract precise visual mask features from high resolution
+input. Experimental results demonstrate Osprey's superiority in various region
+understanding tasks, showcasing its new capability for pixel-level instruction
+tuning. In particular, Osprey can be integrated with Segment Anything Model
+(SAM) seamlessly to obtain multi-granularity semantics. The source code,
+dataset and demo can be found at https://github.com/CircleRadon/Osprey.",cs.CV,['cs.CV']
+Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs,Hao Fei · Shengqiong Wu · Wei Ji · Hanwang Zhang · Tat-seng Chua,http://haofei.vip/Dysen-VDM/,https://arxiv.org/abs/2308.13812,,2308.13812.pdf,Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs,"Text-to-video (T2V) synthesis has gained increasing attention in the
+community, in which the recently emerged diffusion models (DMs) have
+promisingly shown stronger performance than the past approaches. While existing
+state-of-the-art DMs are competent to achieve high-resolution video generation,
+they may largely suffer from key limitations (e.g., action occurrence
+disorders, crude video motions) with respect to the intricate temporal dynamics
+modeling, one of the crux of video synthesis. In this work, we investigate
+strengthening the awareness of video dynamics for DMs, for high-quality T2V
+generation. Inspired by human intuition, we design an innovative dynamic scene
+manager (dubbed as Dysen) module, which includes (step-1) extracting from input
+text the key actions with proper time-order arrangement, (step-2) transforming
+the action schedules into the dynamic scene graph (DSG) representations, and
+(step-3) enriching the scenes in the DSG with sufficient and reasonable
+details. Taking advantage of the existing powerful LLMs (e.g., ChatGPT) via
+in-context learning, Dysen realizes (nearly) human-level temporal dynamics
+understanding. Finally, the resulting video DSG with rich action scene details
+is encoded as fine-grained spatio-temporal features, integrated into the
+backbone T2V DM for video generating. Experiments on popular T2V datasets
+suggest that our Dysen-VDM consistently outperforms prior arts with significant
+margins, especially in scenarios with complex actions. Codes at
+https://haofei.vip/Dysen-VDM",cs.AI,"['cs.AI', 'cs.CV']"
+RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D,Lingteng Qiu · Guanying Chen · Xiaodong Gu · Qi Zuo · Mutian Xu · Yushuang Wu · Weihao Yuan · Zilong Dong · Liefeng Bo · Xiaoguang Han,https://aigc3d.github.io/richdreamer/,https://arxiv.org/abs/2311.16918v1,,2311.16918v1.pdf,RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D,"Lifting 2D diffusion for 3D generation is a challenging problem due to the
+lack of geometric prior and the complex entanglement of materials and lighting
+in natural images. Existing methods have shown promise by first creating the
+geometry through score-distillation sampling (SDS) applied to rendered surface
+normals, followed by appearance modeling. However, relying on a 2D RGB
+diffusion model to optimize surface normals is suboptimal due to the
+distribution discrepancy between natural images and normals maps, leading to
+instability in optimization. In this paper, recognizing that the normal and
+depth information effectively describe scene geometry and be automatically
+estimated from images, we propose to learn a generalizable Normal-Depth
+diffusion model for 3D generation. We achieve this by training on the
+large-scale LAION dataset together with the generalizable image-to-depth and
+normal prior models. In an attempt to alleviate the mixed illumination effects
+in the generated materials, we introduce an albedo diffusion model to impose
+data-driven constraints on the albedo component. Our experiments show that when
+integrated into existing text-to-3D pipelines, our models significantly enhance
+the detail richness, achieving state-of-the-art results. Our project page is
+https://lingtengqiu.github.io/RichDreamer/.",cs.CV,"['cs.CV', 'cs.AI']"
+Towards Generalizing to Unseen Domains with Few Labels,Chamuditha Jayanga Galappaththige · Sanoojan Baliah · Malitha Gunawardhana · Muhammad Haris Khan, ,https://arxiv.org/abs/2403.11674,,2403.11674.pdf,Towards Generalizing to Unseen Domains with Few Labels,"We approach the challenge of addressing semi-supervised domain generalization
+(SSDG). Specifically, our aim is to obtain a model that learns
+domain-generalizable features by leveraging a limited subset of labelled data
+alongside a substantially larger pool of unlabeled data. Existing domain
+generalization (DG) methods which are unable to exploit unlabeled data perform
+poorly compared to semi-supervised learning (SSL) methods under SSDG setting.
+Nevertheless, SSL methods have considerable room for performance improvement
+when compared to fully-supervised DG training. To tackle this underexplored,
+yet highly practical problem of SSDG, we make the following core contributions.
+First, we propose a feature-based conformity technique that matches the
+posterior distributions from the feature space with the pseudo-label from the
+model's output space. Second, we develop a semantics alignment loss to learn
+semantically-compatible representations by regularizing the semantic structure
+in the feature space. Our method is plug-and-play and can be readily integrated
+with different SSL-based SSDG baselines without introducing any additional
+parameters. Extensive experimental results across five challenging DG
+benchmarks with four strong SSL baselines suggest that our method provides
+consistent and notable gains in two different SSDG settings.",cs.CV,['cs.CV']
+SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving,Yiming Xie · Henglu Wei · Zhenyi Liu · Xiaoyu Wang · Xiangyang Ji, ,https://arxiv.org/abs/2403.17094,,2403.17094.pdf,SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving,"To advance research in learning-based defogging algorithms, various synthetic
+fog datasets have been developed. However, existing datasets created using the
+Atmospheric Scattering Model (ASM) or real-time rendering engines often
+struggle to produce photo-realistic foggy images that accurately mimic the
+actual imaging process. This limitation hinders the effective generalization of
+models from synthetic to real data. In this paper, we introduce an end-to-end
+simulation pipeline designed to generate photo-realistic foggy images. This
+pipeline comprehensively considers the entire physically-based foggy scene
+imaging process, closely aligning with real-world image capture methods. Based
+on this pipeline, we present a new synthetic fog dataset named SynFog, which
+features both sky light and active lighting conditions, as well as three levels
+of fog density. Experimental results demonstrate that models trained on SynFog
+exhibit superior performance in visual perception and detection accuracy
+compared to others when applied to real-world foggy images.",cs.CV,"['cs.CV', 'cs.LG']"
+FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication,Eric Slyman · Stefan Lee · Scott Cohen · Kushal Kafle,https://ericslyman.com/fairdedup/,https://arxiv.org/abs/2404.16123,,2404.16123.pdf,FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication,"Recent dataset deduplication techniques have demonstrated that content-aware
+dataset pruning can dramatically reduce the cost of training Vision-Language
+Pretrained (VLP) models without significant performance losses compared to
+training on the original dataset. These results have been based on pruning
+commonly used image-caption datasets collected from the web -- datasets that
+are known to harbor harmful social biases that may then be codified in trained
+models. In this work, we evaluate how deduplication affects the prevalence of
+these biases in the resulting trained models and introduce an easy-to-implement
+modification to the recent SemDeDup algorithm that can reduce the negative
+effects that we observe. When examining CLIP-style models trained on
+deduplicated variants of LAION-400M, we find our proposed FairDeDup algorithm
+consistently leads to improved fairness metrics over SemDeDup on the FairFace
+and FACET datasets while maintaining zero-shot performance on CLIP benchmarks.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'I.4.10; I.2.7; E.0']"
+ESCAPE: Encoding Super-keypoints for Category-Agnostic Pose Estimation,Khoi D Nguyen · Chen Li · Gim Hee Lee, ,https://arxiv.org/abs/2403.13647,,,Meta-Point Learning and Refining for Category-Agnostic Pose Estimation,"Category-agnostic pose estimation (CAPE) aims to predict keypoints for
+arbitrary classes given a few support images annotated with keypoints. Existing
+methods only rely on the features extracted at support keypoints to predict or
+refine the keypoints on query image, but a few support feature vectors are
+local and inadequate for CAPE. Considering that human can quickly perceive
+potential keypoints of arbitrary objects, we propose a novel framework for CAPE
+based on such potential keypoints (named as meta-points). Specifically, we
+maintain learnable embeddings to capture inherent information of various
+keypoints, which interact with image feature maps to produce meta-points
+without any support. The produced meta-points could serve as meaningful
+potential keypoints for CAPE. Due to the inevitable gap between inherency and
+annotation, we finally utilize the identities and details offered by support
+keypoints to assign and refine meta-points to desired keypoints in query image.
+In addition, we propose a progressive deformable point decoder and a slacked
+regression loss for better prediction and supervision. Our novel framework not
+only reveals the inherency of keypoints but also outperforms existing methods
+of CAPE. Comprehensive experiments and in-depth studies on large-scale MP-100
+dataset demonstrate the effectiveness of our framework.",cs.CV,['cs.CV']
+SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation,Jiehong Lin · lihua liu · Dekun Lu · Kui Jia, ,https://arxiv.org/abs/2311.15707,,2311.15707.pdf,SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation,"Zero-shot 6D object pose estimation involves the detection of novel objects
+with their 6D poses in cluttered scenes, presenting significant challenges for
+model generalizability. Fortunately, the recent Segment Anything Model (SAM)
+has showcased remarkable zero-shot transfer performance, which provides a
+promising solution to tackle this task. Motivated by this, we introduce SAM-6D,
+a novel framework designed to realize the task through two steps, including
+instance segmentation and pose estimation. Given the target objects, SAM-6D
+employs two dedicated sub-networks, namely Instance Segmentation Model (ISM)
+and Pose Estimation Model (PEM), to perform these steps on cluttered RGB-D
+images. ISM takes SAM as an advanced starting point to generate all possible
+object proposals and selectively preserves valid ones through meticulously
+crafted object matching scores in terms of semantics, appearance and geometry.
+By treating pose estimation as a partial-to-partial point matching problem, PEM
+performs a two-stage point matching process featuring a novel design of
+background tokens to construct dense 3D-3D correspondence, ultimately yielding
+the pose estimates. Without bells and whistles, SAM-6D outperforms the existing
+methods on the seven core datasets of the BOP Benchmark for both instance
+segmentation and pose estimation of novel objects.",cs.CV,['cs.CV']
+Test-Time Zero-Shot Temporal Action Localization,Benedetta Liberatori · Alessandro Conti · Paolo Rota · Yiming Wang · Elisa Ricci, ,https://arxiv.org/abs/2404.05426,,2404.05426.pdf,Test-Time Zero-Shot Temporal Action Localization,"Zero-Shot Temporal Action Localization (ZS-TAL) seeks to identify and locate
+actions in untrimmed videos unseen during training. Existing ZS-TAL methods
+involve fine-tuning a model on a large amount of annotated training data. While
+effective, training-based ZS-TAL approaches assume the availability of labeled
+data for supervised learning, which can be impractical in some applications.
+Furthermore, the training process naturally induces a domain bias into the
+learned model, which may adversely affect the model's generalization ability to
+arbitrary videos. These considerations prompt us to approach the ZS-TAL problem
+from a radically novel perspective, relaxing the requirement for training data.
+To this aim, we introduce a novel method that performs Test-Time adaptation for
+Temporal Action Localization (T3AL). In a nutshell, T3AL adapts a pre-trained
+Vision and Language Model (VLM). T3AL operates in three steps. First, a
+video-level pseudo-label of the action category is computed by aggregating
+information from the entire video. Then, action localization is performed
+adopting a novel procedure inspired by self-supervised learning. Finally,
+frame-level textual descriptions extracted with a state-of-the-art captioning
+model are employed for refining the action region proposals. We validate the
+effectiveness of T3AL by conducting experiments on the THUMOS14 and the
+ActivityNet-v1.3 datasets. Our results demonstrate that T3AL significantly
+outperforms zero-shot baselines based on state-of-the-art VLMs, confirming the
+benefit of a test-time adaptation approach.",cs.CV,['cs.CV']
+De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts,Yuzheng Wang · Dingkang Yang · Zhaoyu Chen · Yang Liu · Siao Liu · Wenqiang Zhang · Lihua Zhang · Lizhe Qi, ,https://arxiv.org/abs/2403.19539,,,De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts,"Data-Free Knowledge Distillation (DFKD) is a promising task to train
+high-performance small models to enhance actual deployment without relying on
+the original training data. Existing methods commonly avoid relying on private
+data by utilizing synthetic or sampled data. However, a long-overlooked issue
+is that the severe distribution shifts between their substitution and original
+data, which manifests as huge differences in the quality of images and class
+proportions. The harmful shifts are essentially the confounder that
+significantly causes performance bottlenecks. To tackle the issue, this paper
+proposes a novel perspective with causal inference to disentangle the student
+models from the impact of such shifts. By designing a customized causal graph,
+we first reveal the causalities among the variables in the DFKD task.
+Subsequently, we propose a Knowledge Distillation Causal Intervention (KDCI)
+framework based on the backdoor adjustment to de-confound the confounder. KDCI
+can be flexibly combined with most existing state-of-the-art baselines.
+Experiments in combination with six representative DFKD methods demonstrate the
+effectiveness of our KDCI, which can obviously help existing methods under
+almost all settings, \textit{e.g.}, improving the baseline by up to 15.54\%
+accuracy on the CIFAR-100 dataset.",cs.CV,['cs.CV']
+DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields,Cheng-You Lu · Peisen Zhou · Angela Xing · Chandradeep Pokhariya · Arnab Dey · Ishaan Shah · Rugved Mavidipalli · Dylan Hu · Andrew Comport · Kefan Chen · Srinath Sridhar, ,https://arxiv.org/abs/2307.16897,,2307.16897.pdf,DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields,"Advances in neural fields are enabling high-fidelity capture of the shape and
+appearance of dynamic 3D scenes. However, their capabilities lag behind those
+offered by conventional representations such as 2D videos because of
+algorithmic challenges and the lack of large-scale multi-view real-world
+datasets. We address the dataset limitation with DiVa-360, a real-world 360
+dynamic visual dataset that contains synchronized high-resolution and
+long-duration multi-view video sequences of table-scale scenes captured using a
+customized low-cost system with 53 cameras. It contains 21 object-centric
+sequences categorized by different motion types, 25 intricate hand-object
+interaction sequences, and 8 long-duration sequences for a total of 17.4 M
+image frames. In addition, we provide foreground-background segmentation masks,
+synchronized audio, and text descriptions. We benchmark the state-of-the-art
+dynamic neural field methods on DiVa-360 and provide insights about existing
+methods and future challenges on long-duration neural field capture.",cs.CV,"['cs.CV', 'cs.AI']"
+When StyleGAN Meets Stable Diffusion: a ${\mathcal{W}_+}$ Adapter for Personalized Image Generation,Xiaoming Li · Xinyu Hou · Chen Change Loy,https://github.com/csxmli2016/w-plus-adapter,https://arxiv.org/abs/2311.17461v1,,2311.17461v1.pdf,When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation,"Text-to-image diffusion models have remarkably excelled in producing diverse,
+high-quality, and photo-realistic images. This advancement has spurred a
+growing interest in incorporating specific identities into generated content.
+Most current methods employ an inversion approach to embed a target visual
+concept into the text embedding space using a single reference image. However,
+the newly synthesized faces either closely resemble the reference image in
+terms of facial attributes, such as expression, or exhibit a reduced capacity
+for identity preservation. Text descriptions intended to guide the facial
+attributes of the synthesized face may fall short, owing to the intricate
+entanglement of identity information with identity-irrelevant facial attributes
+derived from the reference image. To address these issues, we present the novel
+use of the extended StyleGAN embedding space $\mathcal{W}_+$, to achieve
+enhanced identity preservation and disentanglement for diffusion models. By
+aligning this semantically meaningful human face latent space with
+text-to-image diffusion models, we succeed in maintaining high fidelity in
+identity preservation, coupled with the capacity for semantic editing.
+Additionally, we propose new training objectives to balance the influences of
+both prompt and identity conditions, ensuring that the identity-irrelevant
+background remains unaffected during facial attribute modifications. Extensive
+experiments reveal that our method adeptly generates personalized text-to-image
+outputs that are not only compatible with prompt descriptions but also amenable
+to common StyleGAN editing directions in diverse settings. Our source code will
+be available at \url{https://github.com/csxmli2016/w-plus-adapter}.",cs.CV,['cs.CV']
+Spectral and Polarization Vision: Spectro-polarimetric Real-world Dataset,Yujin Jeon · Eunsue Choi · Youngchan Kim · Yunseong Moon · Khalid Omer · Felix Heide · Seung-Hwan Baek, ,https://arxiv.org/abs/2311.17396,,2311.17396.pdf,Spectral and Polarization Vision: Spectro-polarimetric Real-world Dataset,"Image datasets are essential not only in validating existing methods in
+computer vision but also in developing new methods. Most existing image
+datasets focus on trichromatic intensity images to mimic human vision. However,
+polarization and spectrum, the wave properties of light that animals in harsh
+environments and with limited brain capacity often rely on, remain
+underrepresented in existing datasets. Although spectro-polarimetric datasets
+exist, these datasets have insufficient object diversity, limited illumination
+conditions, linear-only polarization data, and inadequate image count. Here, we
+introduce two spectro-polarimetric datasets: trichromatic Stokes images and
+hyperspectral Stokes images. These novel datasets encompass both linear and
+circular polarization; they introduce multiple spectral channels; and they
+feature a broad selection of real-world scenes. With our dataset in hand, we
+analyze the spectro-polarimetric image statistics, develop efficient
+representations of such high-dimensional data, and evaluate spectral dependency
+of shape-from-polarization methods. As such, the proposed dataset promises a
+foundation for data-driven spectro-polarimetric imaging and vision research.
+Dataset and code will be publicly available.",cs.CV,"['cs.CV', 'eess.IV']"
+LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation,Linfeng Yuan · Miaojing Shi · Zijie Yue · Qijun Chen, ,https://arxiv.org/abs/2306.08736,,2306.08736.pdf,LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation,"Referring video object segmentation (RVOS) aims to segment the target
+instance referred by a given text expression in a video clip. The text
+expression normally contains sophisticated description of the instance's
+appearance, action, and relation with others. It is therefore rather difficult
+for a RVOS model to capture all these attributes correspondingly in the video;
+in fact, the model often favours more on the action- and relation-related
+visual attributes of the instance. This can end up with partial or even
+incorrect mask prediction of the target instance. We tackle this problem by
+taking a subject-centric short text expression from the original long text
+expression. The short one retains only the appearance-related information of
+the target instance so that we can use it to focus the model's attention on the
+instance's appearance. We let the model make joint predictions using both long
+and short text expressions; and insert a long-short cross-attention module to
+interact the joint features and a long-short predictions intersection loss to
+regulate the joint predictions. Besides the improvement on the linguistic part,
+we also introduce a forward-backward visual consistency loss, which utilizes
+optical flows to warp visual features between the annotated frames and their
+temporal neighbors for consistency. We build our method on top of two state of
+the art pipelines. Extensive experiments on A2D-Sentences, Refer-YouTube-VOS,
+JHMDB-Sentences and Refer-DAVIS17 show impressive improvements of our
+method.Code is available at https://github.com/LinfengYuan1997/Losh.",cs.CV,['cs.CV']
+FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures,Lisa Mais · Peter Hirsch · Claire Managan · Ramya Kandarpa · Josef Rumberger · Annika Reinke · Lena Maier-Hein · Gudrun Ihrke · Dagmar Kainmueller, ,https://arxiv.org/abs/2404.00130,,2404.00130.pdf,FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures,"Instance segmentation of neurons in volumetric light microscopy images of
+nervous systems enables groundbreaking research in neuroscience by facilitating
+joint functional and morphological analyses of neural circuits at cellular
+resolution. Yet said multi-neuron light microscopy data exhibits extremely
+challenging properties for the task of instance segmentation: Individual
+neurons have long-ranging, thin filamentous and widely branching morphologies,
+multiple neurons are tightly inter-weaved, and partial volume effects, uneven
+illumination and noise inherent to light microscopy severely impede local
+disentangling as well as long-range tracing of individual neurons. These
+properties reflect a current key challenge in machine learning research, namely
+to effectively capture long-range dependencies in the data. While respective
+methodological research is buzzing, to date methods are typically benchmarked
+on synthetic datasets. To address this gap, we release the FlyLight Instance
+Segmentation Benchmark (FISBe) dataset, the first publicly available
+multi-neuron light microscopy dataset with pixel-wise annotations. In addition,
+we define a set of instance segmentation metrics for benchmarking that we
+designed to be meaningful with regard to downstream analyses. Lastly, we
+provide three baselines to kick off a competition that we envision to both
+advance the field of machine learning regarding methodology for capturing
+long-range data dependencies, and facilitate scientific discovery in basic
+neuroscience.",cs.CV,"['cs.CV', 'cs.LG']"
+Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos,Sagnik Majumder · Ziad Al-Halah · Kristen Grauman, ,https://arxiv.org/abs/2307.04760,,2307.04760.pdf,Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos,"We propose a self-supervised method for learning representations based on
+spatial audio-visual correspondences in egocentric videos. Our method uses a
+masked auto-encoding framework to synthesize masked binaural (multi-channel)
+audio through the synergy of audio and vision, thereby learning useful spatial
+relationships between the two modalities. We use our pretrained features to
+tackle two downstream video tasks requiring spatial understanding in social
+scenarios: active speaker detection and spatial audio denoising. Through
+extensive experiments, we show that our features are generic enough to improve
+over multiple state-of-the-art baselines on both tasks on two challenging
+egocentric video datasets that offer binaural audio, EgoCom and EasyCom.
+Project: http://vision.cs.utexas.edu/projects/ego_av_corr.",cs.CV,"['cs.CV', 'cs.SD', 'eess.AS']"
+Motion Blur Decomposition with Cross-shutter Guidance,Xiang Ji · Haiyang Jiang · Yinqiang Zheng,https://jixiang2016.github.io/dualBR_site/,https://arxiv.org/abs/2404.01120,,2404.01120.pdf,Motion Blur Decomposition with Cross-shutter Guidance,"Motion blur is a frequently observed image artifact, especially under
+insufficient illumination where exposure time has to be prolonged so as to
+collect more photons for a bright enough image. Rather than simply removing
+such blurring effects, recent researches have aimed at decomposing a blurry
+image into multiple sharp images with spatial and temporal coherence. Since
+motion blur decomposition itself is highly ambiguous, priors from neighbouring
+frames or human annotation are usually needed for motion disambiguation. In
+this paper, inspired by the complementary exposure characteristics of a global
+shutter (GS) camera and a rolling shutter (RS) camera, we propose to utilize
+the ordered scanline-wise delay in a rolling shutter image to robustify motion
+decomposition of a single blurry image. To evaluate this novel dual imaging
+setting, we construct a triaxial system to collect realistic data, as well as a
+deep network architecture that explicitly addresses temporal and contextual
+information through reciprocal branches for cross-shutter motion blur
+decomposition. Experiment results have verified the effectiveness of our
+proposed algorithm, as well as the validity of our dual imaging setting.",cs.CV,['cs.CV']
+LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition,Zhonglin Sun · Chen Feng · Ioannis Patras · Georgios Tzimiropoulos, ,https://arxiv.org/abs/2403.08161,,2403.08161.pdf,LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition,"In this work we focus on learning facial representations that can be adapted
+to train effective face recognition models, particularly in the absence of
+labels. Firstly, compared with existing labelled face datasets, a vastly larger
+magnitude of unlabeled faces exists in the real world. We explore the learning
+strategy of these unlabeled facial images through self-supervised pretraining
+to transfer generalized face recognition performance. Moreover, motivated by
+one recent finding, that is, the face saliency area is critical for face
+recognition, in contrast to utilizing random cropped blocks of images for
+constructing augmentations in pretraining, we utilize patches localized by
+extracted facial landmarks. This enables our method - namely LAndmark-based
+Facial Self-supervised learning LAFS), to learn key representation that is more
+critical for face recognition. We also incorporate two landmark-specific
+augmentations which introduce more diversity of landmark information to further
+regularize the learning. With learned landmark-based facial representations, we
+further adapt the representation for face recognition with regularization
+mitigating variations in landmark positions. Our method achieves significant
+improvement over the state-of-the-art on multiple face recognition benchmarks,
+especially on more challenging few-shot scenarios.",cs.CV,"['cs.CV', 'cs.AI']"
+Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation,Wenxuan Wang · Tongtian Yue · Yisi Zhang · Longteng Guo · Xingjian He · Xinlong Wang · Jing Liu, ,https://arxiv.org/abs/2312.08007,,2312.08007.pdf,Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation,"Referring expression segmentation (RES) aims at segmenting the foreground
+masks of the entities that match the descriptive natural language expression.
+Previous datasets and methods for classic RES task heavily rely on the prior
+assumption that one expression must refer to object-level targets. In this
+paper, we take a step further to finer-grained part-level RES task. To promote
+the object-level RES task towards finer-grained vision-language understanding,
+we put forward a new multi-granularity referring expression segmentation (MRES)
+task and construct an evaluation benchmark called RefCOCOm by manual
+annotations. By employing our automatic model-assisted data engine, we build
+the largest visual grounding dataset namely MRES-32M, which comprises over
+32.2M high-quality masks and captions on the provided 1M images. Besides, a
+simple yet strong model named UniRES is designed to accomplish the unified
+object-level and part-level grounding task. Extensive experiments on our
+RefCOCOm for MRES and three datasets (i.e., RefCOCO(+/g) for classic RES task
+demonstrate the superiority of our method over previous state-of-the-art
+methods. To foster future research into fine-grained visual grounding, our
+benchmark RefCOCOm, the MRES-32M dataset and model UniRES will be publicly
+available at https://github.com/Rubics-Xuan/MRES",cs.CV,['cs.CV']
+Event-based Visible and Infrared Fusion via Multi-task Collaboration,Mengyue Geng · Lin Zhu · Lizhi Wang · Wei Zhang · Ruiqin Xiong · Yonghong Tian, ,https://arxiv.org/abs/2312.04328,,2312.04328.pdf,A Multi-scale Information Integration Framework for Infrared and Visible Image Fusion,"Infrared and visible image fusion aims at generating a fused image containing
+the intensity and detail information of source images, and the key issue is
+effectively measuring and integrating the complementary information of
+multi-modality images from the same scene. Existing methods mostly adopt a
+simple weight in the loss function to decide the information retention of each
+modality rather than adaptively measuring complementary information for
+different image pairs. In this study, we propose a multi-scale dual attention
+(MDA) framework for infrared and visible image fusion, which is designed to
+measure and integrate complementary information in both structure and loss
+function at the image and patch level. In our method, the residual downsample
+block decomposes source images into three scales first. Then, dual attention
+fusion block integrates complementary information and generates a spatial and
+channel attention map at each scale for feature fusion. Finally, the output
+image is reconstructed by the residual reconstruction block. Loss function
+consists of image-level, feature-level and patch-level three parts, of which
+the calculation of the image-level and patch-level two parts are based on the
+weights generated by the complementary information measurement. Indeed, to
+constrain the pixel intensity distribution between the output and infrared
+image, a style loss is added. Our fusion results perform robust and informative
+across different scenarios. Qualitative and quantitative results on two
+datasets illustrate that our method is able to preserve both thermal radiation
+and detailed information from two modalities and achieve comparable results
+compared with the other state-of-the-art methods. Ablation experiments show the
+effectiveness of our information integration architecture and adaptively
+measure complementary information retention in the loss function.",cs.CV,['cs.CV']
+PFStorer: Personalized Face Restoration and Super-Resolution,Tuomas Varanka · Tapani Toivonen · Soumya Tripathy · Guoying Zhao · Erman Acar, ,https://arxiv.org/abs/2403.08436,,2403.08436.pdf,PFStorer: Personalized Face Restoration and Super-Resolution,"Recent developments in face restoration have achieved remarkable results in
+producing high-quality and lifelike outputs. The stunning results however often
+fail to be faithful with respect to the identity of the person as the models
+lack necessary context. In this paper, we explore the potential of personalized
+face restoration with diffusion models. In our approach a restoration model is
+personalized using a few images of the identity, leading to tailored
+restoration with respect to the identity while retaining fine-grained details.
+By using independent trainable blocks for personalization, the rich prior of a
+base restoration model can be exploited to its fullest. To avoid the model
+relying on parts of identity left in the conditioning low-quality images, a
+generative regularizer is employed. With a learnable parameter, the model
+learns to balance between the details generated based on the input image and
+the degree of personalization. Moreover, we improve the training pipeline of
+face restoration models to enable an alignment-free approach. We showcase the
+robust capabilities of our approach in several real-world scenarios with
+multiple identities, demonstrating our method's ability to generate
+fine-grained details with faithful restoration. In the user study we evaluate
+the perceptual quality and faithfulness of the genereated details, with our
+method being voted best 61% of the time compared to the second best with 25% of
+the votes.",cs.CV,['cs.CV']
+UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and Unfavorable Sets,Youngju Na · Woo Jae Kim · Kyu Han · Suhyeon Ha · Sung-Eui Yoon, ,https://arxiv.org/abs/2403.05086,,2403.05086.pdf,UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Sets,"Generalizable neural implicit surface reconstruction aims to obtain an
+accurate underlying geometry given a limited number of multi-view images from
+unseen scenes. However, existing methods select only informative and relevant
+views using predefined scores for training and testing phases. This constraint
+renders the model impractical in real-world scenarios, where the availability
+of favorable combinations cannot always be ensured. We introduce and validate a
+view-combination score to indicate the effectiveness of the input view
+combination. We observe that previous methods output degenerate solutions under
+arbitrary and unfavorable sets. Building upon this finding, we propose
+UFORecon, a robust view-combination generalizable surface reconstruction
+framework. To achieve this, we apply cross-view matching transformers to model
+interactions between source images and build correlation frustums to capture
+global correlations. Additionally, we explicitly encode pairwise feature
+similarities as view-consistent priors. Our proposed framework significantly
+outperforms previous methods in terms of view-combination generalizability and
+also in the conventional generalizable protocol trained with favorable
+view-combinations. The code is available at
+https://github.com/Youngju-Na/UFORecon.",cs.CV,['cs.CV']
+Generalizable Face Landmarking Guided by Conditional Face Warping,Jiayi Liang · Haotian Liu · Hongteng Xu · Dixin Luo,https://plustwo0.github.io/project-face-landmarker/,https://arxiv.org/abs/2404.12322,,2404.12322.pdf,Generalizable Face Landmarking Guided by Conditional Face Warping,"As a significant step for human face modeling, editing, and generation, face
+landmarking aims at extracting facial keypoints from images. A generalizable
+face landmarker is required in practice because real-world facial images, e.g.,
+the avatars in animations and games, are often stylized in various ways.
+However, achieving generalizable face landmarking is challenging due to the
+diversity of facial styles and the scarcity of labeled stylized faces. In this
+study, we propose a simple but effective paradigm to learn a generalizable face
+landmarker based on labeled real human faces and unlabeled stylized faces. Our
+method learns the face landmarker as the key module of a conditional face
+warper. Given a pair of real and stylized facial images, the conditional face
+warper predicts a warping field from the real face to the stylized one, in
+which the face landmarker predicts the ending points of the warping field and
+provides us with high-quality pseudo landmarks for the corresponding stylized
+facial images. Applying an alternating optimization strategy, we learn the face
+landmarker to minimize $i)$ the discrepancy between the stylized faces and the
+warped real ones and $ii)$ the prediction errors of both real and pseudo
+landmarks. Experiments on various datasets show that our method outperforms
+existing state-of-the-art domain adaptation methods in face landmarking tasks,
+leading to a face landmarker with better generalizability. Code is available at
+https://plustwo0.github.io/project-face-landmarker.",cs.CV,"['cs.CV', 'cs.AI']"
+PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs,Michael Dorkenwald · Nimrod Barazani · Cees G. M. Snoek · Yuki Asano,https://quva-lab.github.io/PIN/,https://arxiv.org/abs/2402.08657,,2402.08657.pdf,PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs,"Vision-Language Models (VLMs), such as Flamingo and GPT-4V, have shown
+immense potential by integrating large language models with vision systems.
+Nevertheless, these models face challenges in the fundamental computer vision
+task of object localisation, due to their training on multimodal data
+containing mostly captions without explicit spatial grounding. While it is
+possible to construct custom, supervised training pipelines with bounding box
+annotations that integrate with VLMs, these result in specialized and
+hard-to-scale models. In this paper, we aim to explore the limits of
+caption-based VLMs and instead propose to tackle the challenge in a simpler
+manner by i) keeping the weights of a caption-based VLM frozen and ii) not
+using any supervised detection data. To this end, we introduce an
+input-agnostic Positional Insert (PIN), a learnable spatial prompt, containing
+a minimal set of parameters that are slid inside the frozen VLM, unlocking
+object localisation capabilities. Our PIN module is trained with a simple
+next-token prediction task on synthetic data without requiring the introduction
+of new output heads. Our experiments demonstrate strong zero-shot localisation
+performances on a variety of images, including Pascal VOC, COCO, LVIS, and
+diverse images like paintings or cartoons.",cs.CV,['cs.CV']
+SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation,Keqi Chen · vinkle srivastav · Nicolas Padoy,https://github.com/CAMMA-public/SelfPose3d/,https://arxiv.org/abs/2404.02041,,,SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation,"We present a new self-supervised approach, SelfPose3d, for estimating 3d
+poses of multiple persons from multiple camera views. Unlike current
+state-of-the-art fully-supervised methods, our approach does not require any 2d
+or 3d ground-truth poses and uses only the multi-view input images from a
+calibrated camera setup and 2d pseudo poses generated from an off-the-shelf 2d
+human pose estimator. We propose two self-supervised learning objectives:
+self-supervised person localization in 3d space and self-supervised 3d pose
+estimation. We achieve self-supervised 3d person localization by training the
+model on synthetically generated 3d points, serving as 3d person root
+positions, and on the projected root-heatmaps in all the views. We then model
+the 3d poses of all the localized persons with a bottleneck representation, map
+them onto all views obtaining 2d joints, and render them using 2d Gaussian
+heatmaps in an end-to-end differentiable manner. Afterwards, we use the
+corresponding 2d joints and heatmaps from the pseudo 2d poses for learning. To
+alleviate the intrinsic inaccuracy of the pseudo labels, we propose an adaptive
+supervision attention mechanism to guide the self-supervision. Our experiments
+and analysis on three public benchmark datasets, including Panoptic, Shelf, and
+Campus, show the effectiveness of our approach, which is comparable to
+fully-supervised methods. Code is available at
+\url{https://github.com/CAMMA-public/SelfPose3D}",cs.CV,['cs.CV']
+PEM: Prototype-based Efficient MaskFormer for Image Segmentation,Niccolò Cavagnero · Gabriele Rosi · Claudia Cuttano · Francesca Pistilli · Marco Ciccone · Giuseppe Averta · Fabio Cermelli,https://niccolocavagnero.github.io/PEM/,https://arxiv.org/abs/2402.19422,,2402.19422.pdf,PEM: Prototype-based Efficient MaskFormer for Image Segmentation,"Recent transformer-based architectures have shown impressive results in the
+field of image segmentation. Thanks to their flexibility, they obtain
+outstanding performance in multiple segmentation tasks, such as semantic and
+panoptic, under a single unified framework. To achieve such impressive
+performance, these architectures employ intensive operations and require
+substantial computational resources, which are often not available, especially
+on edge devices. To fill this gap, we propose Prototype-based Efficient
+MaskFormer (PEM), an efficient transformer-based architecture that can operate
+in multiple segmentation tasks. PEM proposes a novel prototype-based
+cross-attention which leverages the redundancy of visual features to restrict
+the computation and improve the efficiency without harming the performance. In
+addition, PEM introduces an efficient multi-scale feature pyramid network,
+capable of extracting features that have high semantic content in an efficient
+way, thanks to the combination of deformable convolutions and context-based
+self-modulation. We benchmark the proposed PEM architecture on two tasks,
+semantic and panoptic segmentation, evaluated on two different datasets,
+Cityscapes and ADE20K. PEM demonstrates outstanding performance on every task
+and dataset, outperforming task-specific architectures while being comparable
+and even better than computationally-expensive baselines.",cs.CV,"['cs.CV', 'cs.AI']"
+Improving Distant 3D Object Detection Using 2D Box Supervision,Zetong Yang · Zhiding Yu · Christopher Choy · Renhao Wang · Anima Anandkumar · Jose M. Alvarez, ,https://arxiv.org/abs/2403.09230,,2403.09230.pdf,Improving Distant 3D Object Detection Using 2D Box Supervision,"Improving the detection of distant 3d objects is an important yet challenging
+task. For camera-based 3D perception, the annotation of 3d bounding relies
+heavily on LiDAR for accurate depth information. As such, the distance of
+annotation is often limited due to the sparsity of LiDAR points on distant
+objects, which hampers the capability of existing detectors for long-range
+scenarios. We address this challenge by considering only 2D box supervision for
+distant objects since they are easy to annotate. We propose LR3D, a framework
+that learns to recover the missing depth of distant objects. LR3D adopts an
+implicit projection head to learn the generation of mapping between 2D boxes
+and depth using the 3D supervision on close objects. This mapping allows the
+depth estimation of distant objects conditioned on their 2D boxes, making
+long-range 3D detection with 2D supervision feasible. Experiments show that
+without distant 3D annotations, LR3D allows camera-based methods to detect
+distant objects (over 200m) with comparable accuracy to full 3D supervision.
+Our framework is general, and could widely benefit 3D detection methods to a
+large extent.",cs.CV,['cs.CV']
+Visual Point Cloud Forecasting enables Scalable Autonomous Driving,Zetong Yang · Li Chen · Yanan Sun · Hongyang Li,https://github.com/OpenDriveLab/ViDAR,https://arxiv.org/abs/2312.17655,,2312.17655.pdf,Visual Point Cloud Forecasting enables Scalable Autonomous Driving,"In contrast to extensive studies on general vision, pre-training for scalable
+visual autonomous driving remains seldom explored. Visual autonomous driving
+applications require features encompassing semantics, 3D geometry, and temporal
+information simultaneously for joint perception, prediction, and planning,
+posing dramatic challenges for pre-training. To resolve this, we bring up a new
+pre-training task termed as visual point cloud forecasting - predicting future
+point clouds from historical visual input. The key merit of this task captures
+the synergic learning of semantics, 3D structures, and temporal dynamics. Hence
+it shows superiority in various downstream tasks. To cope with this new
+problem, we present ViDAR, a general model to pre-train downstream visual
+encoders. It first extracts historical embeddings by the encoder. These
+representations are then transformed to 3D geometric space via a novel Latent
+Rendering operator for future point cloud prediction. Experiments show
+significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10%
+error reduction on motion forecasting, and ~15% less collision rate on
+planning.",cs.CV,['cs.CV']
+Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting,Haipeng Liu · Yang Wang · Biao Qian · Meng Wang · Yong Rui,https://github.com/htyjers/StrDiffusion,https://arxiv.org/abs/2403.19898,,2403.19898.pdf,Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting,"Denoising diffusion probabilistic models for image inpainting aim to add the
+noise to the texture of image during the forward process and recover masked
+regions with unmasked ones of the texture via the reverse denoising process.
+Despite the meaningful semantics generation, the existing arts suffer from the
+semantic discrepancy between masked and unmasked regions, since the
+semantically dense unmasked texture fails to be completely degraded while the
+masked regions turn to the pure noise in diffusion process, leading to the
+large discrepancy between them. In this paper, we aim to answer how unmasked
+semantics guide texture denoising process;together with how to tackle the
+semantic discrepancy, to facilitate the consistent and meaningful semantics
+generation. To this end, we propose a novel structure-guided diffusion model
+named StrDiffusion, to reformulate the conventional texture denoising process
+under structure guidance to derive a simplified denoising objective for image
+inpainting, while revealing: 1) the semantically sparse structure is beneficial
+to tackle semantic discrepancy in early stage, while dense texture generates
+reasonable semantics in late stage; 2) the semantics from unmasked regions
+essentially offer the time-dependent structure guidance for the texture
+denoising process, benefiting from the time-dependent sparsity of the structure
+semantics. For the denoising process, a structure-guided neural network is
+trained to estimate the simplified denoising objective by exploiting the
+consistency of the denoised structure between masked and unmasked regions.
+Besides, we devise an adaptive resampling strategy as a formal criterion as
+whether structure is competent to guide the texture denoising process, while
+regulate their semantic correlations. Extensive experiments validate the merits
+of StrDiffusion over the state-of-the-arts. Our code is available at
+https://github.com/htyjers/StrDiffusion.",cs.CV,['cs.CV']
+Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models,Kota Sueyoshi · Takashi Matsubara, ,https://arxiv.org/abs/2311.16117,,2311.16117.pdf,Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models,"Diffusion models have achieved remarkable results in generating high-quality,
+diverse, and creative images. However, when it comes to text-based image
+generation, they often fail to capture the intended meaning presented in the
+text. For instance, a specified object may not be generated, an unnecessary
+object may be generated, and an adjective may alter objects it was not intended
+to modify. Moreover, we found that relationships indicating possession between
+objects are often overlooked. While users' intentions in text are diverse,
+existing methods tend to specialize in only some aspects of these. In this
+paper, we propose Predicated Diffusion, a unified framework to express users'
+intentions. We consider that the root of the above issues lies in the text
+encoder, which often focuses only on individual words and neglects the logical
+relationships between them. The proposed method does not solely rely on the
+text encoder, but instead, represents the intended meaning in the text as
+propositions using predicate logic and treats the pixels in the attention maps
+as the fuzzy predicates. This enables us to obtain a differentiable loss
+function that makes the image fulfill the proposition by minimizing it. When
+compared to several existing methods, we demonstrated that Predicated Diffusion
+can generate images that are more faithful to various text prompts, as verified
+by human evaluators and pretrained image-text models.",cs.CV,['cs.CV']
+Probabilistic Human Mesh Estimation with Hypothesis Scoring,Yuan Xu · Xiaoxuan Ma · Jiajun Su · Wentao Zhu · Yu Qiao · Yizhou Wang, ,https://arxiv.org/abs/2308.02963,,2308.02963.pdf,Generative Approach for Probabilistic Human Mesh Recovery using Diffusion Models,"This work focuses on the problem of reconstructing a 3D human body mesh from
+a given 2D image. Despite the inherent ambiguity of the task of human mesh
+recovery, most existing works have adopted a method of regressing a single
+output. In contrast, we propose a generative approach framework, called
+""Diffusion-based Human Mesh Recovery (Diff-HMR)"" that takes advantage of the
+denoising diffusion process to account for multiple plausible outcomes. During
+the training phase, the SMPL parameters are diffused from ground-truth
+parameters to random distribution, and Diff-HMR learns the reverse process of
+this diffusion. In the inference phase, the model progressively refines the
+given random SMPL parameters into the corresponding parameters that align with
+the input image. Diff-HMR, being a generative approach, is capable of
+generating diverse results for the same input image as the input noise varies.
+We conduct validation experiments, and the results demonstrate that the
+proposed framework effectively models the inherent ambiguity of the task of
+human mesh recovery in a probabilistic manner. The code is available at
+https://github.com/hanbyel0105/Diff-HMR",cs.CV,['cs.CV']
+TexVocab: Texture Vocabulary-conditioned Human Avatars,Yuxiao Liu · Zhe Li · Yebin Liu · Haoqian Wang, ,https://arxiv.org/abs/2404.00524,,2404.00524.pdf,TexVocab: Texture Vocabulary-conditioned Human Avatars,"To adequately utilize the available image evidence in multi-view video-based
+avatar modeling, we propose TexVocab, a novel avatar representation that
+constructs a texture vocabulary and associates body poses with texture maps for
+animation. Given multi-view RGB videos, our method initially back-projects all
+the available images in the training videos to the posed SMPL surface,
+producing texture maps in the SMPL UV domain. Then we construct pairs of human
+poses and texture maps to establish a texture vocabulary for encoding dynamic
+human appearances under various poses. Unlike the commonly used joint-wise
+manner, we further design a body-part-wise encoding strategy to learn the
+structural effects of the kinematic chain. Given a driving pose, we query the
+pose feature hierarchically by decomposing the pose vector into several body
+parts and interpolating the texture features for synthesizing fine-grained
+human dynamics. Overall, our method is able to create animatable human avatars
+with detailed and dynamic appearances from RGB videos, and the experiments show
+that our method outperforms state-of-the-art approaches. The project page can
+be found at https://texvocab.github.io/.",cs.CV,['cs.CV']
+LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network,Hao Yang · Liyuan Pan · Yan Yang · Richard Hartley · Miaomiao Liu, ,https://arxiv.org/abs/2307.09815,,2307.09815.pdf,LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network,"Recovering sharp images from dual-pixel (DP) pairs with disparity-dependent
+blur is a challenging task.~Existing blur map-based deblurring methods have
+demonstrated promising results. In this paper, we propose, to the best of our
+knowledge, the first framework that introduces the contrastive language-image
+pre-training framework (CLIP) to accurately estimate the blur map from a DP
+pair unsupervisedly. To achieve this, we first carefully design text prompts to
+enable CLIP to understand blur-related geometric prior knowledge from the DP
+pair. Then, we propose a format to input a stereo DP pair to CLIP without any
+fine-tuning, despite the fact that CLIP is pre-trained on monocular images.
+Given the estimated blur map, we introduce a blur-prior attention block, a
+blur-weighting loss, and a blur-aware loss to recover the all-in-focus image.
+Our method achieves state-of-the-art performance in extensive experiments (see
+Fig.~\ref{fig:teaser}).",cs.CV,['cs.CV']
+VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection,Zihua Liu · Hiroki Sakuma · Masatoshi Okutomi,http://www.ok.sc.e.titech.ac.jp/res/VSRD/index.html,https://arxiv.org/abs/2404.00149,,2404.00149.pdf,VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection,"Monocular 3D object detection poses a significant challenge in 3D scene
+understanding due to its inherently ill-posed nature in monocular depth
+estimation. Existing methods heavily rely on supervised learning using abundant
+3D labels, typically obtained through expensive and labor-intensive annotation
+on LiDAR point clouds. To tackle this problem, we propose a novel weakly
+supervised 3D object detection framework named VSRD (Volumetric Silhouette
+Rendering for Detection) to train 3D object detectors without any 3D
+supervision but only weak 2D supervision. VSRD consists of multi-view 3D
+auto-labeling and subsequent training of monocular 3D object detectors using
+the pseudo labels generated in the auto-labeling stage. In the auto-labeling
+stage, we represent the surface of each instance as a signed distance field
+(SDF) and render its silhouette as an instance mask through our proposed
+instance-aware volumetric silhouette rendering. To directly optimize the 3D
+bounding boxes through rendering, we decompose the SDF of each instance into
+the SDF of a cuboid and the residual distance field (RDF) that represents the
+residual from the cuboid. This mechanism enables us to optimize the 3D bounding
+boxes in an end-to-end manner by comparing the rendered instance masks with the
+ground truth instance masks. The optimized 3D bounding boxes serve as effective
+training data for 3D object detection. We conduct extensive experiments on the
+KITTI-360 dataset, demonstrating that our method outperforms the existing
+weakly supervised 3D object detection methods. The code is available at
+https://github.com/skmhrk1209/VSRD.",cs.CV,['cs.CV']
+Real-World Mobile Image Denoising Dataset with Efficient Baselines,Roman Flepp · Andrey Ignatov · Radu Timofte · Luc Van Gool, ,https://arxiv.org/html/2404.08514v2,,2404.08514v2.pdf,NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Datase,"Despite the significant progress in image denoising, it is still challenging
+to restore fine-scale details while removing noise, especially in extremely
+low-light environments. Leveraging near-infrared (NIR) images to assist visible
+RGB image denoising shows the potential to address this issue, becoming a
+promising technology. Nonetheless, existing works still struggle with taking
+advantage of NIR information effectively for real-world image denoising, due to
+the content inconsistency between NIR-RGB images and the scarcity of real-world
+paired datasets. To alleviate the problem, we propose an efficient Selective
+Fusion Module (SFM), which can be plug-and-played into the advanced denoising
+networks to merge the deep NIR-RGB features. Specifically, we sequentially
+perform the global and local modulation for NIR and RGB features, and then
+integrate the two modulated features. Furthermore, we present a Real-world
+NIR-Assisted Image Denoising (Real-NAID) dataset, which covers diverse
+scenarios as well as various noise levels. Extensive experiments on both
+synthetic and our real-world datasets demonstrate that the proposed method
+achieves better results than state-of-the-art ones.",cs.CV,['cs.CV']
+Exploiting Style Latent Flows for Generalizing Video Deepfake Detection,Jongwook Choi · Taehoon Kim · Yonghyun Jeong · Seungryul Baek · Jongwon Choi, ,https://arxiv.org/abs/2403.06592v1,,2403.06592v1.pdf,Exploiting Style Latent Flows for Generalizing Deepfake Detection Video Detection,"This paper presents a new approach for the detection of fake videos, based on
+the analysis of style latent vectors and their abnormal behavior in temporal
+changes in the generated videos. We discovered that the generated facial videos
+suffer from the temporal distinctiveness in the temporal changes of style
+latent vectors, which are inevitable during the generation of temporally stable
+videos with various facial expressions and geometric transformations. Our
+framework utilizes the StyleGRU module, trained by contrastive learning, to
+represent the dynamic properties of style latent vectors. Additionally, we
+introduce a style attention module that integrates StyleGRU-generated features
+with content-based features, enabling the detection of visual and temporal
+artifacts. We demonstrate our approach across various benchmark scenarios in
+deepfake detection, showing its superiority in cross-dataset and
+cross-manipulation scenarios. Through further analysis, we also validate the
+importance of using temporal changes of style latent vectors to improve the
+generality of deepfake video detection.",cs.CV,"['cs.CV', 'cs.AI']"
+3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation,Zidu Wang · Xiangyu Zhu · Tianshuo Zhang · baiqin wang · Zhen Lei,https://github.com/wang-zidu/3DDFA-V3,https://arxiv.org/abs/2312.00311,,2312.00311.pdf,3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation,"3D Morphable Models (3DMMs) provide promising 3D face reconstructions in
+various applications. However, existing methods struggle to reconstruct faces
+with extreme expressions due to deficiencies in supervisory signals, such as
+sparse or inaccurate landmarks. Segmentation information contains effective
+geometric contexts for face reconstruction. Certain attempts intuitively depend
+on differentiable renderers to compare the rendered silhouettes of
+reconstruction with segmentation, which is prone to issues like local optima
+and gradient instability. In this paper, we fully utilize the facial part
+segmentation geometry by introducing Part Re-projection Distance Loss (PRDL).
+Specifically, PRDL transforms facial part segmentation into 2D points and
+re-projects the reconstruction onto the image plane. Subsequently, by
+introducing grid anchors and computing different statistical distances from
+these anchors to the point sets, PRDL establishes geometry descriptors to
+optimize the distribution of the point sets for face reconstruction. PRDL
+exhibits a clear gradient compared to the renderer-based methods and presents
+state-of-the-art reconstruction performance in extensive quantitative and
+qualitative experiments. Our project is available at
+https://github.com/wang-zidu/3DDFA-V3 .",cs.CV,['cs.CV']
+PerceptionGPT: Effectively Fusing Visual Perception into LLM,Renjie Pi · Lewei Yao · Jiahui Gao · Jipeng Zhang · Tong Zhang, ,https://arxiv.org/abs/2311.06612,,2311.06612.pdf,PerceptionGPT: Effectively Fusing Visual Perception into LLM,"The integration of visual inputs with large language models (LLMs) has led to
+remarkable advancements in multi-modal capabilities, giving rise to visual
+large language models (VLLMs). However, effectively harnessing VLLMs for
+intricate visual perception tasks remains a challenge. In this paper, we
+present a novel end-to-end framework named PerceptionGPT, which efficiently and
+effectively equips the VLLMs with visual perception abilities by leveraging the
+representation power of LLMs' token embedding. Our proposed method treats the
+token embedding of the LLM as the carrier of spatial information, then leverage
+lightweight visual task encoders and decoders to perform visual perception
+tasks (e.g., detection, segmentation). Our approach significantly alleviates
+the training difficulty suffered by previous approaches that formulate the
+visual outputs as discrete tokens, and enables achieving superior performance
+with fewer trainable parameters, less training data and shorted training time.
+Moreover, as only one token embedding is required to decode the visual outputs,
+the resulting sequence length during inference is significantly reduced.
+Consequently, our approach enables accurate and flexible representations,
+seamless integration of visual perception tasks, and efficient handling of a
+multiple of visual outputs. We validate the effectiveness and efficiency of our
+approach through extensive experiments. The results demonstrate significant
+improvements over previous methods with much fewer trainable parameters and GPU
+hours, which facilitates future research in enabling LLMs with visual
+perception abilities.",cs.CV,"['cs.CV', 'cs.CL']"
+In Search of a Data Transformation That Accelerates Neural Field Training,Junwon Seo · Sangyoon Lee · Kwang In Kim · Jaeho Lee, ,https://arxiv.org/abs/2311.17094,,2311.17094.pdf,In Search of a Data Transformation That Accelerates Neural Field Training,"Neural field is an emerging paradigm in data representation that trains a
+neural network to approximate the given signal. A key obstacle that prevents
+its widespread adoption is the encoding speed-generating neural fields requires
+an overfitting of a neural network, which can take a significant number of SGD
+steps to reach the desired fidelity level. In this paper, we delve into the
+impacts of data transformations on the speed of neural field training,
+specifically focusing on how permuting pixel locations affect the convergence
+speed of SGD. Counterintuitively, we find that randomly permuting the pixel
+locations can considerably accelerate the training. To explain this phenomenon,
+we examine the neural field training through the lens of PSNR curves, loss
+landscapes, and error patterns. Our analyses suggest that the random pixel
+permutations remove the easy-to-fit patterns, which facilitate easy
+optimization in the early stage but hinder capturing fine details of the
+signal.",cs.LG,"['cs.LG', 'cs.CV']"
+Multi-view Aggregation Network for Dichotomous Image Segmentation,Qian Yu · Xiaoqi Zhao · Youwei Pang · Lihe Zhang · Huchuan Lu, ,https://arxiv.org/abs/2404.07445,,2404.07445.pdf,Multi-view Aggregation Network for Dichotomous Image Segmentation,"Dichotomous Image Segmentation (DIS) has recently emerged towards
+high-precision object segmentation from high-resolution natural images.
+  When designing an effective DIS model, the main challenge is how to balance
+the semantic dispersion of high-resolution targets in the small receptive field
+and the loss of high-precision details in the large receptive field. Existing
+methods rely on tedious multiple encoder-decoder streams and stages to
+gradually complete the global localization and local refinement.
+  Human visual system captures regions of interest by observing them from
+multiple views. Inspired by it, we model DIS as a multi-view object perception
+problem and provide a parsimonious multi-view aggregation network (MVANet),
+which unifies the feature fusion of the distant view and close-up view into a
+single stream with one encoder-decoder structure. With the help of the proposed
+multi-view complementary localization and refinement modules, our approach
+established long-range, profound visual interactions across multiple views,
+allowing the features of the detailed close-up view to focus on highly slender
+structures.Experiments on the popular DIS-5K dataset show that our MVANet
+significantly outperforms state-of-the-art methods in both accuracy and speed.
+The source code and datasets will be publicly available at
+\href{https://github.com/qianyu-dlut/MVANet}{MVANet}.",cs.CV,['cs.CV']
+Three Pillars improving Vision Foundation Model Distillation for Lidar,Gilles Puy · Spyros Gidaris · Alexandre Boulch · Oriane Siméoni · Corentin Sautier · Patrick Pérez · Andrei Bursuc · Renaud Marlet,https://github.com/valeoai/ScaLR,https://arxiv.org/abs/2310.17504,,2310.17504.pdf,Three Pillars improving Vision Foundation Model Distillation for Lidar,"Self-supervised image backbones can be used to address complex 2D tasks
+(e.g., semantic segmentation, object discovery) very efficiently and with
+little or no downstream supervision. Ideally, 3D backbones for lidar should be
+able to inherit these properties after distillation of these powerful 2D
+features. The most recent methods for image-to-lidar distillation on autonomous
+driving data show promising results, obtained thanks to distillation methods
+that keep improving. Yet, we still notice a large performance gap when
+measuring the quality of distilled and fully supervised features by linear
+probing. In this work, instead of focusing only on the distillation method, we
+study the effect of three pillars for distillation: the 3D backbone, the
+pretrained 2D backbones, and the pretraining dataset. In particular, thanks to
+our scalable distillation method named ScaLR, we show that scaling the 2D and
+3D backbones and pretraining on diverse datasets leads to a substantial
+improvement of the feature quality. This allows us to significantly reduce the
+gap between the quality of distilled and fully-supervised 3D features, and to
+improve the robustness of the pretrained backbones to domain gaps and
+perturbations.",cs.CV,['cs.CV']
+Cloud-Device Collaborative Learning for Multimodal Large Language Models,Guanqun Wang · Jiaming Liu · Chenxuan Li · Yuan Zhang · Ma Junpeng · Xinyu Wei · Kevin Zhang · Maurice Chong · Renrui Zhang · Yijiang Liu · Shanghang Zhang,https://github.com/2644521362/Cdcca/tree/main,https://arxiv.org/abs/2312.16279,,2312.16279.pdf,Cloud-Device Collaborative Learning for Multimodal Large Language Models,"The burgeoning field of Multimodal Large Language Models (MLLMs) has
+exhibited remarkable performance in diverse tasks such as captioning,
+commonsense reasoning, and visual scene understanding. However, the deployment
+of these large-scale MLLMs on client devices is hindered by their extensive
+model parameters, leading to a notable decline in generalization capabilities
+when these models are compressed for device deployment. Addressing this
+challenge, we introduce a Cloud-Device Collaborative Continual Adaptation
+framework, designed to enhance the performance of compressed, device-deployed
+MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs.
+Our framework is structured into three key components: a device-to-cloud uplink
+for efficient data transmission, cloud-based knowledge adaptation, and an
+optimized cloud-to-device downlink for model deployment. In the uplink phase,
+we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively
+filter out-of-distribution tokens, thereby reducing transmission costs and
+improving training efficiency. On the cloud side, we propose Adapter-based
+Knowledge Distillation (AKD) method to transfer refined knowledge from
+large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic
+Weight update Compression (DWC) strategy for the downlink, which adaptively
+selects and quantizes updated weight parameters, enhancing transmission
+efficiency and reducing the representational disparity between cloud and device
+models. Extensive experiments on several multimodal benchmarks demonstrate the
+superiority of our proposed framework over prior Knowledge Distillation and
+device-cloud collaboration methods. Notably, we also validate the feasibility
+of our approach to real-world experiments.",cs.CV,['cs.CV']
+Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors,Ziqin Zhou · Hai-Ming Xu · Yangyang Shu · Lingqiao Liu, ,https://arxiv.org/abs/2404.02117,,2404.02117.pdf,Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners,"Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model
+to learn new classes incrementally without forgetting when only a few samples
+for each class are given. FSCIL encounters two significant challenges:
+catastrophic forgetting and overfitting, and these challenges have driven prior
+studies to primarily rely on shallow models, such as ResNet-18. Even though
+their limited capacity can mitigate both forgetting and overfitting issues, it
+leads to inadequate knowledge transfer during few-shot incremental sessions. In
+this paper, we argue that large models such as vision and language transformers
+pre-trained on large datasets can be excellent few-shot incremental learners.
+To this end, we propose a novel FSCIL framework called PriViLege, Pre-trained
+Vision and Language transformers with prompting functions and knowledge
+distillation. Our framework effectively addresses the challenges of
+catastrophic forgetting and overfitting in large models through new pre-trained
+knowledge tuning (PKT) and two losses: entropy-based divergence loss and
+semantic knowledge distillation loss. Experimental results show that the
+proposed PriViLege significantly outperforms the existing state-of-the-art
+methods with a large margin, e.g., +9.38% in CUB200, +20.58% in CIFAR-100, and
++13.36% in miniImageNet. Our implementation code is available at
+https://github.com/KHU-AGI/PriViLege.",cs.CV,['cs.CV']
+LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernels,Tuo Feng · Wenguan Wang · Fan Ma · Yi Yang,https://github.com/FengZicai/LSK3DNet,https://arxiv.org/abs/2403.15173,,2403.15173.pdf,LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernels,"Autonomous systems need to process large-scale, sparse, and irregular point
+clouds with limited compute resources. Consequently, it is essential to develop
+LiDAR perception methods that are both efficient and effective. Although
+naively enlarging 3D kernel size can enhance performance, it will also lead to
+a cubically-increasing overhead. Therefore, it is crucial to develop
+streamlined 3D large kernel designs that eliminate redundant weights and work
+effectively with larger kernels. In this paper, we propose an efficient and
+effective Large Sparse Kernel 3D Neural Network (LSK3DNet) that leverages
+dynamic pruning to amplify the 3D kernel size. Our method comprises two core
+components: Spatial-wise Dynamic Sparsity (SDS) and Channel-wise Weight
+Selection (CWS). SDS dynamically prunes and regrows volumetric weights from the
+beginning to learn a large sparse 3D kernel. It not only boosts performance but
+also significantly reduces model size and computational cost. Moreover, CWS
+selects the most important channels for 3D convolution during training and
+subsequently prunes the redundant channels to accelerate inference for 3D
+vision tasks. We demonstrate the effectiveness of LSK3DNet on three benchmark
+datasets and five tracks compared with classical models and large kernel
+designs. Notably, LSK3DNet achieves the state-of-the-art performance on
+SemanticKITTI (i.e., 75.6% on single-scan and 63.4% on multi-scan), with
+roughly 40% model size reduction and 60% computing operations reduction
+compared to the naive large 3D kernel model.",cs.CV,['cs.CV']
+On the Robustness of Large Multimodal Models Against Image Adversarial Attacks,Xuanming Cui · Alejandro Aparcedo · Young Kyun Jang · Ser-Nam Lim, ,https://arxiv.org/abs/2312.03777,,2312.03777.pdf,On the Robustness of Large Multimodal Models Against Image Adversarial Attacks,"Recent advances in instruction tuning have led to the development of
+State-of-the-Art Large Multimodal Models (LMMs). Given the novelty of these
+models, the impact of visual adversarial attacks on LMMs has not been
+thoroughly examined. We conduct a comprehensive study of the robustness of
+various LMMs against different adversarial attacks, evaluated across tasks
+including image classification, image captioning, and Visual Question Answer
+(VQA). We find that in general LMMs are not robust to visual adversarial
+inputs. However, our findings suggest that context provided to the model via
+prompts, such as questions in a QA pair helps to mitigate the effects of visual
+adversarial inputs. Notably, the LMMs evaluated demonstrated remarkable
+resilience to such attacks on the ScienceQA task with only an 8.10% drop in
+performance compared to their visual counterparts which dropped 99.73%. We also
+propose a new approach to real-world image classification which we term query
+decomposition. By incorporating existence queries into our input prompt we
+observe diminished attack effectiveness and improvements in image
+classification accuracy. This research highlights a previously under-explored
+facet of LMM robustness and sets the stage for future work aimed at
+strengthening the resilience of multimodal systems in adversarial environments.",cs.CV,['cs.CV']
+Amodal Ground Truth and Completion in the Wild,Guanqi Zhan · Chuanxia Zheng · Weidi Xie · Andrew Zisserman,https://www.robots.ox.ac.uk/~vgg/research/amodal/,https://arxiv.org/abs/2312.17247,,2312.17247.pdf,Amodal Ground Truth and Completion in the Wild,"This paper studies amodal image segmentation: predicting entire object
+segmentation masks including both visible and invisible (occluded) parts. In
+previous work, the amodal segmentation ground truth on real images is usually
+predicted by manual annotaton and thus is subjective. In contrast, we use 3D
+data to establish an automatic pipeline to determine authentic ground truth
+amodal masks for partially occluded objects in real images. This pipeline is
+used to construct an amodal completion evaluation benchmark, MP3D-Amodal,
+consisting of a variety of object categories and labels. To better handle the
+amodal completion task in the wild, we explore two architecture variants: a
+two-stage model that first infers the occluder, followed by amodal mask
+completion; and a one-stage model that exploits the representation power of
+Stable Diffusion for amodal segmentation across many categories. Without bells
+and whistles, our method achieves a new state-of-the-art performance on Amodal
+segmentation datasets that cover a large variety of objects, including COCOA
+and our new MP3D-Amodal dataset. The dataset, model, and code are available at
+https://www.robots.ox.ac.uk/~vgg/research/amodal/.",cs.CV,['cs.CV']
+MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction,Xiaolu Liu · Song Wang · Wentong Li · Ruizi Yang · Junbo Chen · Jianke Zhu,https://github.com/xiaolul2/MGMap,https://arxiv.org/abs/2404.00876,,2404.00876.pdf,MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction,"Currently, high-definition (HD) map construction leans towards a lightweight
+online generation tendency, which aims to preserve timely and reliable road
+scene information. However, map elements contain strong shape priors. Subtle
+and sparse annotations make current detection-based frameworks ambiguous in
+locating relevant feature scopes and cause the loss of detailed structures in
+prediction. To alleviate these problems, we propose MGMap, a mask-guided
+approach that effectively highlights the informative regions and achieves
+precise map element localization by introducing the learned masks.
+Specifically, MGMap employs learned masks based on the enhanced multi-scale BEV
+features from two perspectives. At the instance level, we propose the
+Mask-activated instance (MAI) decoder, which incorporates global instance and
+structural information into instance queries by the activation of instance
+masks. At the point level, a novel position-guided mask patch refinement
+(PG-MPR) module is designed to refine point locations from a finer-grained
+perspective, enabling the extraction of point-specific patch information.
+Compared to the baselines, our proposed MGMap achieves a notable improvement of
+around 10 mAP for different input modalities. Extensive experiments also
+demonstrate that our approach showcases strong robustness and generalization
+capabilities. Our code can be found at https://github.com/xiaolul2/MGMap.",cs.CV,['cs.CV']
+SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder,Dihan Zheng · Yihang Zou · Xiaowen Zhang · Chenglong Bao, ,https://arxiv.org/abs/2403.17502,,2403.17502.pdf,SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder,"The data bottleneck has emerged as a fundamental challenge in learning based
+image restoration methods. Researchers have attempted to generate synthesized
+training data using paired or unpaired samples to address this challenge. This
+study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages
+both paired and unpaired datasets to generate realistic degraded data. Our
+approach is based on modeling the conditional distribution of degraded and
+clean images with a specially designed graphical model. Under the variational
+inference framework, we develop an objective function for handling both paired
+and unpaired data. We employ our method to generate paired training samples for
+real-world image denoising and super-resolution tasks. Our approach excels in
+the quality of synthetic degraded images compared to other unpaired and paired
+noise modeling methods. Furthermore, our approach demonstrates remarkable
+performance in downstream image restoration tasks, even with limited paired
+data. With more paired data, our method achieves the best performance on the
+SIDD dataset.",cs.CV,['cs.CV']
+What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation,Yihua Cheng · Yaning Zhu · Zongji Wang · hongquan hao · Liu wei · Shiqing Cheng · Xi Wang · Hyung Jin Chang,https://yihua.zone/work/ivgaze/,https://arxiv.org/abs/2403.15664,,2403.15664.pdf,What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation,"Driver's eye gaze holds a wealth of cognitive and intentional cues crucial
+for intelligent vehicles. Despite its significance, research on in-vehicle gaze
+estimation remains limited due to the scarcity of comprehensive and
+well-annotated datasets in real driving scenarios. In this paper, we present
+three novel elements to advance in-vehicle gaze research. Firstly, we introduce
+IVGaze, a pioneering dataset capturing in-vehicle gaze, collected from 125
+subjects and covering a large range of gaze and head poses within vehicles.
+Conventional gaze collection systems are inadequate for in-vehicle use. In this
+dataset, we propose a new vision-based solution for in-vehicle gaze collection,
+introducing a refined gaze target calibration method to tackle annotation
+challenges. Second, our research focuses on in-vehicle gaze estimation
+leveraging the IVGaze. In-vehicle face images often suffer from low resolution,
+prompting our introduction of a gaze pyramid transformer that leverages
+transformer-based multilevel features integration. Expanding upon this, we
+introduce the dual-stream gaze pyramid transformer (GazeDPTR). Employing
+perspective transformation, we rotate virtual cameras to normalize images,
+utilizing camera pose to merge normalized and original images for accurate gaze
+estimation. GazeDPTR shows state-of-the-art performance on the IVGaze dataset.
+Thirdly, we explore a novel strategy for gaze zone classification by extending
+the GazeDPTR. A foundational tri-plane and project gaze onto these planes are
+newly defined. Leveraging both positional features from the projection points
+and visual attributes from images, we achieve superior performance compared to
+relying solely on visual features, substantiating the advantage of gaze
+estimation. Our project is available at https://yihua.zone/work/ivgaze.",cs.CV,['cs.CV']
+LangSplat: 3D Language Gaussian Splatting,Minghan Qin · Wanhua Li · Jiawei ZHOU · Haoqian Wang · Hanspeter Pfister,https://langsplat.github.io/,https://arxiv.org/abs/2312.16084,,2312.16084.pdf,LangSplat: 3D Language Gaussian Splatting,"Humans live in a 3D world and commonly use natural language to interact with
+a 3D scene. Modeling a 3D language field to support open-ended language queries
+in 3D has gained increasing attention recently. This paper introduces
+LangSplat, which constructs a 3D language field that enables precise and
+efficient open-vocabulary querying within 3D spaces. Unlike existing methods
+that ground CLIP language embeddings in a NeRF model, LangSplat advances the
+field by utilizing a collection of 3D Gaussians, each encoding language
+features distilled from CLIP, to represent the language field. By employing a
+tile-based splatting technique for rendering language features, we circumvent
+the costly rendering process inherent in NeRF. Instead of directly learning
+CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
+then learns language features on the scene-specific latent space, thereby
+alleviating substantial memory demands imposed by explicit modeling. Existing
+methods struggle with imprecise and vague 3D language fields, which fail to
+discern clear boundaries between objects. We delve into this issue and propose
+to learn hierarchical semantics using SAM, thereby eliminating the need for
+extensively querying the language field across various scales and the
+regularization of DINO features. Extensive experimental results show that
+LangSplat significantly outperforms the previous state-of-the-art method LERF
+by a large margin. Notably, LangSplat is extremely efficient, achieving a 199
+$\times$ speedup compared to LERF at the resolution of 1440 $\times$ 1080. We
+strongly recommend readers to check out our video results at
+https://langsplat.github.io/",cs.CV,['cs.CV']
+DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching,Shuzhe Wang · Juho Kannala · Daniel Barath, ,https://arxiv.org/abs/2306.12547,,2306.12547.pdf,DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching,"Matching 2D keypoints in an image to a sparse 3D point cloud of the scene
+without requiring visual descriptors has garnered increased interest due to its
+low memory requirements, inherent privacy preservation, and reduced need for
+expensive 3D model maintenance compared to visual descriptor-based methods.
+However, existing algorithms often compromise on performance, resulting in a
+significant deterioration compared to their descriptor-based counterparts. In
+this paper, we introduce DGC-GNN, a novel algorithm that employs a
+global-to-local Graph Neural Network (GNN) that progressively exploits
+geometric and color cues to represent keypoints, thereby improving matching
+accuracy. Our procedure encodes both Euclidean and angular relations at a
+coarse level, forming the geometric embedding to guide the point matching. We
+evaluate DGC-GNN on both indoor and outdoor datasets, demonstrating that it not
+only doubles the accuracy of the state-of-the-art visual descriptor-free
+algorithm but also substantially narrows the performance gap between
+descriptor-based and descriptor-free methods.",cs.CV,['cs.CV']
+DiffForensics: Leveraging Diffusion Prior to Image Forgery Detection and Localization,Zeqin Yu · Jiangqun Ni · Yuzhen Lin · Haoyi Deng · Bin Li, ,https://arxiv.org/abs/2401.15859,,2401.15859.pdf,Diffusion Facial Forgery Detection,"Detecting diffusion-generated images has recently grown into an emerging
+research area. Existing diffusion-based datasets predominantly focus on general
+image generation. However, facial forgeries, which pose a more severe social
+risk, have remained less explored thus far. To address this gap, this paper
+introduces DiFF, a comprehensive dataset dedicated to face-focused
+diffusion-generated images. DiFF comprises over 500,000 images that are
+synthesized using thirteen distinct generation methods under four conditions.
+In particular, this dataset leverages 30,000 carefully collected textual and
+visual prompts, ensuring the synthesis of images with both high fidelity and
+semantic consistency. We conduct extensive experiments on the DiFF dataset via
+a human test and several representative forgery detection methods. The results
+demonstrate that the binary detection accuracy of both human observers and
+automated detectors often falls below 30%, shedding light on the challenges in
+detecting diffusion-generated facial forgeries. Furthermore, we propose an edge
+graph regularization approach to effectively enhance the generalization
+capability of existing detectors.",cs.CV,"['cs.CV', 'cs.AI']"
+Affine Equivariant Networks Based on Differential Invariants,Yikang Li · Yeqing Qiu · Yuxuan Chen · Lingshen He · Zhouchen Lin, ,,https://www.semanticscholar.org/paper/Lie-Group-Decompositions-for-Equivariant-Neural-Mironenco-Forr'e/5302620834b3969b11097f66375cadbf9ee9c817,,,,,nan
+EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World,Yifei Huang · Guo Chen · Jilan Xu · Mingfang Zhang · Lijin Yang · Baoqi Pei · Hongjie Zhang · Lu Dong · Yali Wang · Limin Wang · Yu Qiao,https://github.com/OpenGVLab/EgoExoLearn,https://arxiv.org/abs/2403.16182,,2403.16182.pdf,EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World,"Being able to map the activities of others into one's own point of view is
+one fundamental human skill even from a very early age. Taking a step toward
+understanding this human ability, we introduce EgoExoLearn, a large-scale
+dataset that emulates the human demonstration following process, in which
+individuals record egocentric videos as they execute tasks guided by
+demonstration videos. Focusing on the potential applications in daily
+assistance and professional support, EgoExoLearn contains egocentric and
+demonstration video data spanning 120 hours captured in daily life scenarios
+and specialized laboratories. Along with the videos we record high-quality gaze
+data and provide detailed multimodal annotations, formulating a playground for
+modeling the human ability to bridge asynchronous procedural actions from
+different viewpoints. To this end, we present benchmarks such as cross-view
+association, cross-view action planning, and cross-view referenced skill
+assessment, along with detailed analysis. We expect EgoExoLearn can serve as an
+important resource for bridging the actions across views, thus paving the way
+for creating AI agents capable of seamlessly learning by observing humans in
+the real world. Code and data can be found at:
+https://github.com/OpenGVLab/EgoExoLearn",cs.CV,['cs.CV']
+Learning from Observer Gaze: Zero-shot Attention Prediction Oriented by Human-Object Interaction Recognition,Yuchen Zhou · Linkai Liu · Chao Gou,https://yuchen2199.github.io/Interactive-Gaze/,https://arxiv.org/abs/2405.09931,,2405.09931.pdf,Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition,"Most existing attention prediction research focuses on salient instances like
+humans and objects. However, the more complex interaction-oriented attention,
+arising from the comprehension of interactions between instances by human
+observers, remains largely unexplored. This is equally crucial for advancing
+human-machine interaction and human-centered artificial intelligence. To bridge
+this gap, we first collect a novel gaze fixation dataset named IG, comprising
+530,000 fixation points across 740 diverse interaction categories, capturing
+visual attention during human observers cognitive processes of interactions.
+Subsequently, we introduce the zero-shot interaction-oriented attention
+prediction task ZeroIA, which challenges models to predict visual cues for
+interactions not encountered during training. Thirdly, we present the
+Interactive Attention model IA, designed to emulate human observers cognitive
+processes to tackle the ZeroIA problem. Extensive experiments demonstrate that
+the proposed IA outperforms other state-of-the-art approaches in both ZeroIA
+and fully supervised settings. Lastly, we endeavor to apply
+interaction-oriented attention to the interaction recognition task itself.
+Further experimental results demonstrate the promising potential to enhance the
+performance and interpretability of existing state-of-the-art HOI models by
+incorporating real human attention data from IG and attention labels generated
+by IA.",cs.CV,['cs.CV']
+EFHQ: Multi-purpose ExtremePose-Face-HQ dataset,Trung Dao · Duc H Vu · Cuong Pham · Anh Tran,https://bomcon123456.github.io/efhq/,https://arxiv.org/abs/2312.17205,,2312.17205.pdf,EFHQ: Multi-purpose ExtremePose-Face-HQ dataset,"The existing facial datasets, while having plentiful images at near frontal
+views, lack images with extreme head poses, leading to the downgraded
+performance of deep learning models when dealing with profile or pitched faces.
+This work aims to address this gap by introducing a novel dataset named Extreme
+Pose Face High-Quality Dataset (EFHQ), which includes a maximum of 450k
+high-quality images of faces at extreme poses. To produce such a massive
+dataset, we utilize a novel and meticulous dataset processing pipeline to
+curate two publicly available datasets, VFHQ and CelebV-HQ, which contain many
+high-resolution face videos captured in various settings. Our dataset can
+complement existing datasets on various facial-related tasks, such as facial
+synthesis with 2D/3D-aware GAN, diffusion-based text-to-image face generation,
+and face reenactment. Specifically, training with EFHQ helps models generalize
+well across diverse poses, significantly improving performance in scenarios
+involving extreme views, confirmed by extensive experiments. Additionally, we
+utilize EFHQ to define a challenging cross-view face verification benchmark, in
+which the performance of SOTA face recognition models drops 5-37% compared to
+frontal-to-frontal scenarios, aiming to stimulate studies on face recognition
+under severe pose conditions in the wild.",cs.CV,['cs.CV']
+Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling,Xinhang Liu · Yu-Wing Tai · Chi-Keung Tang · Pedro Miraldo · Suhas Lohit · Moitreya Chatterjee, ,https://arxiv.org/abs/2405.06214,,2405.06214.pdf,Aerial-NeRF: Adaptive Spatial Partitioning and Sampling for Large-Scale Aerial Rendering,"Recent progress in large-scale scene rendering has yielded Neural Radiance
+Fields (NeRF)-based models with an impressive ability to synthesize scenes
+across small objects and indoor scenes. Nevertheless, extending this idea to
+large-scale aerial rendering poses two critical problems. Firstly, a single
+NeRF cannot render the entire scene with high-precision for complex large-scale
+aerial datasets since the sampling range along each view ray is insufficient to
+cover buildings adequately. Secondly, traditional NeRFs are infeasible to train
+on one GPU to enable interactive fly-throughs for modeling massive images.
+Instead, existing methods typically separate the whole scene into multiple
+regions and train a NeRF on each region, which are unaccustomed to different
+flight trajectories and difficult to achieve fast rendering. To that end, we
+propose Aerial-NeRF with three innovative modifications for jointly adapting
+NeRF in large-scale aerial rendering: (1) Designing an adaptive spatial
+partitioning and selection method based on drones' poses to adapt different
+flight trajectories; (2) Using similarity of poses instead of (expert) network
+for rendering speedup to determine which region a new viewpoint belongs to; (3)
+Developing an adaptive sampling approach for rendering performance improvement
+to cover the entire buildings at different heights. Extensive experiments have
+conducted to verify the effectiveness and efficiency of Aerial-NeRF, and new
+state-of-the-art results have been achieved on two public large-scale aerial
+datasets and presented SCUTic dataset. Note that our model allows us to perform
+rendering over 4 times as fast as compared to multiple competitors. Our
+dataset, code, and model are publicly available at https://drliuqi.github.io/.",cs.CV,['cs.CV']
+PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models,Yiming Zhang · Zhening Xing · Yanhong Zeng · Youqing Fang · Kai Chen, ,https://arxiv.org/abs/2312.13964,,2312.13964.pdf,PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models,"Recent advancements in personalized text-to-image (T2I) models have
+revolutionized content creation, empowering non-experts to generate stunning
+images with unique styles. While promising, adding realistic motions into these
+personalized images by text poses significant challenges in preserving distinct
+styles, high-fidelity details, and achieving motion controllability by text. In
+this paper, we present PIA, a Personalized Image Animator that excels in
+aligning with condition images, achieving motion controllability by text, and
+the compatibility with various personalized T2I models without specific tuning.
+To achieve these goals, PIA builds upon a base T2I model with well-trained
+temporal alignment layers, allowing for the seamless transformation of any
+personalized T2I model into an image animation model. A key component of PIA is
+the introduction of the condition module, which utilizes the condition frame
+and inter-frame affinity as input to transfer appearance information guided by
+the affinity hint for individual frame synthesis in the latent space. This
+design mitigates the challenges of appearance-related image alignment within
+and allows for a stronger focus on aligning with motion-related guidance.",cs.CV,"['cs.CV', 'cs.AI']"
+Weakly Supervised Video Individual Counting,Xinyan Liu · Guorong Li · Yuankai Qi · Ziheng Yan · Zhenjun Han · Anton van den Hengel · Ming-Hsuan Yang · Qingming Huang, ,https://arxiv.org/abs/2312.05923,,2312.05923.pdf,Weakly Supervised Video Individual CountingWeakly Supervised Video Individual Counting,"Video Individual Counting (VIC) aims to predict the number of unique
+individuals in a single video. % Existing methods learn representations based
+on trajectory labels for individuals, which are annotation-expensive. % To
+provide a more realistic reflection of the underlying practical challenge, we
+introduce a weakly supervised VIC task, wherein trajectory labels are not
+provided. Instead, two types of labels are provided to indicate traffic
+entering the field of view (inflow) and leaving the field view (outflow). % We
+also propose the first solution as a baseline that formulates the task as a
+weakly supervised contrastive learning problem under group-level matching. In
+doing so, we devise an end-to-end trainable soft contrastive loss to drive the
+network to distinguish inflow, outflow, and the remaining. % To facilitate
+future study in this direction, we generate annotations from the existing VIC
+datasets SenseCrowd and CroHD and also build a new dataset, UAVVIC. % Extensive
+results show that our baseline weakly supervised method outperforms supervised
+methods, and thus, little information is lost in the transition to the more
+practically relevant weakly supervised task. The code and trained model will be
+public at \href{https://github.com/streamer-AP/CGNet}{CGNet}",cs.CV,['cs.CV']
+Model Inversion Robustness: Can Transfer Learning Help?,Sy-Tuyen Ho · Koh Jun Hao · Keshigeyan Chandrasegaran · Ngoc-Bao Nguyen · Ngai-Man Cheung, ,https://arxiv.org/abs/2405.05588,,2405.05588.pdf,Model Inversion Robustness: Can Transfer Learning Help?,"Model Inversion (MI) attacks aim to reconstruct private training data by
+abusing access to machine learning models. Contemporary MI attacks have
+achieved impressive attack performance, posing serious threats to privacy.
+Meanwhile, all existing MI defense methods rely on regularization that is in
+direct conflict with the training objective, resulting in noticeable
+degradation in model utility. In this work, we take a different perspective,
+and propose a novel and simple Transfer Learning-based Defense against Model
+Inversion (TL-DMI) to render MI-robust models. Particularly, by leveraging TL,
+we limit the number of layers encoding sensitive information from private
+training dataset, thereby degrading the performance of MI attack. We conduct an
+analysis using Fisher Information to justify our method. Our defense is
+remarkably simple to implement. Without bells and whistles, we show in
+extensive experiments that TL-DMI achieves state-of-the-art (SOTA) MI
+robustness. Our code, pre-trained models, demo and inverted data are available
+at: https://hosytuyen.github.io/projects/TL-DMI",cs.LG,"['cs.LG', 'cs.CR', 'cs.CV']"
+$M^3$-UDA: A New Benchmark for Unsupervised Domain Adaptive Fetal Cardiac Structure Detection,Bin Pu · Liwen Wang · Jiewen Yang · He Guannan · Xingbo Dong · Shengli Li · Ying Tan · Ming Chen · Zhe Jin · Kenli Li · Xiaomeng Li, ,https://arxiv.org/abs/2310.14172,,2310.14172.pdf,ASC: Appearance and Structure Consistency for Unsupervised Domain Adaptation in Fetal Brain MRI Segmentation,"Automatic tissue segmentation of fetal brain images is essential for the
+quantitative analysis of prenatal neurodevelopment. However, producing
+voxel-level annotations of fetal brain imaging is time-consuming and expensive.
+To reduce labeling costs, we propose a practical unsupervised domain adaptation
+(UDA) setting that adapts the segmentation labels of high-quality fetal brain
+atlases to unlabeled fetal brain MRI data from another domain. To address the
+task, we propose a new UDA framework based on Appearance and Structure
+Consistency, named ASC. We adapt the segmentation model to the appearances of
+different domains by constraining the consistency before and after a
+frequency-based image transformation, which is to swap the appearance between
+brain MRI data and atlases. Consider that even in the same domain, the fetal
+brain images of different gestational ages could have significant variations in
+the anatomical structures. To make the model adapt to the structural variations
+in the target domain, we further encourage prediction consistency under
+different structural perturbations. Extensive experiments on FeTA 2021
+benchmark demonstrate the effectiveness of our ASC in comparison to
+registration-based, semi-supervised learning-based, and existing UDA-based
+methods.",eess.IV,"['eess.IV', 'cs.CV']"
+A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?,Galadrielle Humblot-Renaux · Sergio Escalera · Thomas B. Moeslund, ,https://arxiv.org/abs/2404.01775,,2404.01775.pdf,A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?,"The ability to detect unfamiliar or unexpected images is essential for safe
+deployment of computer vision systems. In the context of classification, the
+task of detecting images outside of a model's training domain is known as
+out-of-distribution (OOD) detection. While there has been a growing research
+interest in developing post-hoc OOD detection methods, there has been
+comparably little discussion around how these methods perform when the
+underlying classifier is not trained on a clean, carefully curated dataset. In
+this work, we take a closer look at 20 state-of-the-art OOD detection methods
+in the (more realistic) scenario where the labels used to train the underlying
+classifier are unreliable (e.g. crowd-sourced or web-scraped labels). Extensive
+experiments across different datasets, noise types & levels, architectures and
+checkpointing strategies provide insights into the effect of class label noise
+on OOD detection, and show that poor separation between incorrectly classified
+ID samples vs. OOD samples is an overlooked yet important limitation of
+existing methods. Code: https://github.com/glhr/ood-labelnoise",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models,Mengcheng Li · Hongwen Zhang · Yuxiang Zhang · Ruizhi Shao · Tao Yu · Yebin Liu,https://www.liuyebin.com/HHMR/HHMR.html,https://arxiv.org/abs/2402.14654,,2402.14654.pdf,Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot,"We present Multi-HMR, a strong single-shot model for multi-person 3D human
+mesh recovery from a single RGB image. Predictions encompass the whole body,
+i.e, including hands and facial expressions, using the SMPL-X parametric model
+and spatial location in the camera coordinate system. Our model detects people
+by predicting coarse 2D heatmaps of person centers, using features produced by
+a standard Vision Transformer (ViT) backbone. It then predicts their whole-body
+pose, shape and spatial location using a new cross-attention module called the
+Human Prediction Head (HPH), with one query per detected center token,
+attending to the entire set of features. As direct prediction of SMPL-X
+parameters yields suboptimal results, we introduce CUFFS; the Close-Up Frames
+of Full-Body Subjects dataset, containing humans close to the camera with
+diverse hand poses. We show that incorporating this dataset into training
+further enhances predictions, particularly for hands, enabling us to achieve
+state-of-the-art performance. Multi-HMR also optionally accounts for camera
+intrinsics, if available, by encoding camera ray directions for each image
+token. This simple design achieves strong performance on whole-body and
+body-only benchmarks simultaneously. We train models with various backbone
+sizes and input resolutions. In particular, using a ViT-S backbone and
+$448\times448$ input images already yields a fast and competitive model with
+respect to state-of-the-art methods, while considering larger models and higher
+resolutions further improve performance.",cs.CV,['cs.CV']
+C$^\text{2}$RV: Cross-Regional and Cross-View Learning for Sparse-View CBCT Reconstruction,Yiqun Lin · Jiewen Yang · hualiang wang · Xinpeng Ding · Wei Zhao · Xiaomeng Li,https://github.com/xmed-lab/C2RV-CBCT,https://arxiv.org/abs/2312.01689,,2312.01689.pdf,Fast and accurate sparse-view CBCT reconstruction using meta-learned neural attenuation field and hash-encoding regularization,"Cone beam computed tomography (CBCT) is an emerging medical imaging technique
+to visualize the internal anatomical structures of patients. During a CBCT
+scan, several projection images of different angles or views are collectively
+utilized to reconstruct a tomographic image. However, reducing the number of
+projections in a CBCT scan while preserving the quality of a reconstructed
+image is challenging due to the nature of an ill-posed inverse problem.
+Recently, a neural attenuation field (NAF) method was proposed by adopting a
+neural radiance field algorithm as a new way for CBCT reconstruction,
+demonstrating fast and promising results using only 50 views. However,
+decreasing the number of projections is still preferable to reduce potential
+radiation exposure, and a faster reconstruction time is required considering a
+typical scan time. In this work, we propose a fast and accurate sparse-view
+CBCT reconstruction (FACT) method to provide better reconstruction quality and
+faster optimization speed in the minimal number of view acquisitions ($<$ 50
+views). In the FACT method, we meta-trained a neural network and a hash-encoder
+using a few scans (= 15), and a new regularization technique is utilized to
+reconstruct the details of an anatomical structure. In conclusion, we have
+shown that the FACT method produced better, and faster reconstruction results
+over the other conventional algorithms based on CBCT scans of different body
+parts (chest, head, and abdomen) and CT vendors (Siemens, Phillips, and GE).",eess.IV,"['eess.IV', 'cs.CV']"
+Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions,Namitha Padmanabhan · Matthew A Gwilliam · Pulkit Kumar · Shishira R Maiya · Max Ehrlich · Abhinav Shrivastava,https://namithap10.github.io/xinc/,https://arxiv.org/abs/2401.10217,,2401.10217.pdf,Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions,"The many variations of Implicit Neural Representations (INRs), where a neural
+network is trained as a continuous representation of a signal, have tremendous
+practical utility for downstream tasks including novel view synthesis, video
+compression, and image superresolution. Unfortunately, the inner workings of
+these networks are seriously under-studied. Our work, eXplaining the Implicit
+Neural Canvas (XINC), is a unified framework for explaining properties of INRs
+by examining the strength of each neuron's contribution to each output pixel.
+We call the aggregate of these contribution maps the Implicit Neural Canvas and
+we use this concept to demonstrate that the INRs which we study learn to
+''see'' the frames they represent in surprising ways. For example, INRs tend to
+have highly distributed representations. While lacking high-level object
+semantics, they have a significant bias for color and edges, and are almost
+entirely space-agnostic. We arrive at our conclusions by examining how objects
+are represented across time in video INRs, using clustering to visualize
+similar neurons across layers and architectures, and show that this is
+dominated by motion. These insights demonstrate the general usefulness of our
+analysis framework. Our project page is available at
+https://namithap10.github.io/xinc.",cs.CV,['cs.CV']
+Posterior Distillation Sampling,Juil Koo · Chanho Park · Minhyuk Sung,https://posterior-distillation-sampling.github.io/,https://arxiv.org/abs/2311.13831,,2311.13831.pdf,Posterior Distillation Sampling,"We introduce Posterior Distillation Sampling (PDS), a novel optimization
+method for parametric image editing based on diffusion models. Existing
+optimization-based methods, which leverage the powerful 2D prior of diffusion
+models to handle various parametric images, have mainly focused on generation.
+Unlike generation, editing requires a balance between conforming to the target
+attribute and preserving the identity of the source content. Recent 2D image
+editing methods have achieved this balance by leveraging the stochastic latent
+encoded in the generative process of diffusion models. To extend the editing
+capabilities of diffusion models shown in pixel space to parameter space, we
+reformulate the 2D image editing method into an optimization form named PDS.
+PDS matches the stochastic latents of the source and the target, enabling the
+sampling of targets in diverse parameter spaces that align with a desired
+attribute while maintaining the source's identity. We demonstrate that this
+optimization resembles running a generative process with the target attribute,
+but aligning this process with the trajectory of the source's generative
+process. Extensive editing results in Neural Radiance Fields and Scalable
+Vector Graphics representations demonstrate that PDS is capable of sampling
+targets to fulfill the aforementioned balance across various parameter spaces.",cs.CV,['cs.CV']
+Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices,Huancheng Chen · Haris Vikalo, ,https://arxiv.org/abs/2311.18129,,2311.18129.pdf,Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices,"While federated learning (FL) systems often utilize quantization to battle
+communication and computational bottlenecks, they have heretofore been limited
+to deploying fixed-precision quantization schemes. Meanwhile, the concept of
+mixed-precision quantization (MPQ), where different layers of a deep learning
+model are assigned varying bit-width, remains unexplored in the FL settings. We
+present a novel FL algorithm, FedMPQ, which introduces mixed-precision
+quantization to resource-heterogeneous FL systems. Specifically, local models,
+quantized so as to satisfy bit-width constraint, are trained by optimizing an
+objective function that includes a regularization term which promotes reduction
+of precision in some of the layers without significant performance degradation.
+The server collects local model updates, de-quantizes them into full-precision
+models, and then aggregates them into a global model. To initialize the next
+round of local training, the server relies on the information learned in the
+previous training round to customize bit-width assignments of the models
+delivered to different clients. In extensive benchmarking experiments on
+several model architectures and different datasets in both iid and non-iid
+settings, FedMPQ outperformed the baseline FL schemes that utilize
+fixed-precision quantization while incurring only a minor computational
+overhead on the participating devices.",cs.LG,"['cs.LG', 'cs.DC']"
+Coherent Temporal Synthesis for Incremental Action Segmentation,Guodong Ding · Hans Golong · Angela Yao,https://guodongding.cn/projects/itas/itas.html,https://arxiv.org/abs/2403.06102,,2403.06102.pdf,Coherent Temporal Synthesis for Incremental Action Segmentation,"Data replay is a successful incremental learning technique for images. It
+prevents catastrophic forgetting by keeping a reservoir of previous data,
+original or synthesized, to ensure the model retains past knowledge while
+adapting to novel concepts. However, its application in the video domain is
+rudimentary, as it simply stores frame exemplars for action recognition. This
+paper presents the first exploration of video data replay techniques for
+incremental action segmentation, focusing on action temporal modeling. We
+propose a Temporally Coherent Action (TCA) model, which represents actions
+using a generative model instead of storing individual frames. The integration
+of a conditioning variable that captures temporal coherence allows our model to
+understand the evolution of action features over time. Therefore, action
+segments generated by TCA for replay are diverse and temporally coherent. In a
+10-task incremental setup on the Breakfast dataset, our approach achieves
+significant increases in accuracy for up to 22% compared to the baselines.",cs.CV,['cs.CV']
+GLACE: Global Local Accelerated Coordinate Encoding,Fangjinhua Wang · Xudong Jiang · Silvano Galliani · Christoph Vogel · Marc Pollefeys, ,,https://ieeexplore.ieee.org/document/10204902/figures,,,,,nan
+Text-Guided Variational Image Generation for Industrial Anomaly Detection and Segmentation,Mingyu Lee · Jongwon Choi,https://github.com/MingyuLee82/TGI_AD_v1,https://arxiv.org/abs/2403.06247,,2403.06247.pdf,Text-Guided Variational Image Generation for Industrial Anomaly Detection and Segmentation,"We propose a text-guided variational image generation method to address the
+challenge of getting clean data for anomaly detection in industrial
+manufacturing. Our method utilizes text information about the target object,
+learned from extensive text library documents, to generate non-defective data
+images resembling the input image. The proposed framework ensures that the
+generated non-defective images align with anticipated distributions derived
+from textual and image-based knowledge, ensuring stability and generality.
+Experimental results demonstrate the effectiveness of our approach, surpassing
+previous methods even with limited non-defective data. Our approach is
+validated through generalization tests across four baseline models and three
+distinct datasets. We present an additional analysis to enhance the
+effectiveness of anomaly detection models by utilizing the generated images.",cs.CV,"['cs.CV', 'cs.AI']"
+Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models,Bin Fu · Fanghua Yu · Anran Liu · Zixuan Wang · Jie Wen · Junjun He · Yu Qiao, ,https://arxiv.org/abs/2312.12142,,2312.12142.pdf,FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning,"Automatic font generation is an imitation task, which aims to create a font
+library that mimics the style of reference images while preserving the content
+from source images. Although existing font generation methods have achieved
+satisfactory performance, they still struggle with complex characters and large
+style variations. To address these issues, we propose FontDiffuser, a
+diffusion-based image-to-image one-shot font generation method, which
+innovatively models the font imitation task as a noise-to-denoise paradigm. In
+our method, we introduce a Multi-scale Content Aggregation (MCA) block, which
+effectively combines global and local content cues across different scales,
+leading to enhanced preservation of intricate strokes of complex characters.
+Moreover, to better manage the large variations in style transfer, we propose a
+Style Contrastive Refinement (SCR) module, which is a novel structure for style
+representation learning. It utilizes a style extractor to disentangle styles
+from images, subsequently supervising the diffusion model via a meticulously
+designed style contrastive loss. Extensive experiments demonstrate
+FontDiffuser's state-of-the-art performance in generating diverse characters
+and styles. It consistently excels on complex characters and large style
+changes compared to previous methods. The code is available at
+https://github.com/yeungchenwa/FontDiffuser.",cs.CV,"['cs.CV', 'cs.AI']"
+PTQ4SAM: Post-Training Quantization for Segment Anything,Chengtao Lv · Hong Chen · Jinyang Guo · Yifu Ding · Xianglong Liu, ,https://arxiv.org/abs/2405.03144,,2405.03144.pdf,PTQ4SAM: Post-Training Quantization for Segment Anything,"Segment Anything Model (SAM) has achieved impressive performance in many
+computer vision tasks. However, as a large-scale model, the immense memory and
+computation costs hinder its practical deployment. In this paper, we propose a
+post-training quantization (PTQ) framework for Segment Anything Model, namely
+PTQ4SAM. First, we investigate the inherent bottleneck of SAM quantization
+attributed to the bimodal distribution in post-Key-Linear activations. We
+analyze its characteristics from both per-tensor and per-channel perspectives,
+and propose a Bimodal Integration strategy, which utilizes a mathematically
+equivalent sign operation to transform the bimodal distribution into a
+relatively easy-quantized normal distribution offline. Second, SAM encompasses
+diverse attention mechanisms (i.e., self-attention and two-way
+cross-attention), resulting in substantial variations in the post-Softmax
+distributions. Therefore, we introduce an Adaptive Granularity Quantization for
+Softmax through searching the optimal power-of-two base, which is
+hardware-friendly. Extensive experimental results across various vision tasks
+(instance segmentation, semantic segmentation and object detection), datasets
+and model variants show the superiority of PTQ4SAM. For example, when
+quantizing SAM-L to 6-bit, we achieve lossless accuracy for instance
+segmentation, about 0.5\% drop with theoretical 3.9$\times$ acceleration. The
+code is available at \url{https://github.com/chengtao-lv/PTQ4SAM}.",cs.CV,"['cs.CV', 'cs.LG']"
+Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach,Mir Hossain Hossain · Mennatullah Siam · Leonid Sigal · Jim Little, ,https://arxiv.org/abs/2404.11732,,2404.11732.pdf,Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach,"The emergence of attention-based transformer models has led to their
+extensive use in various tasks, due to their superior generalization and
+transfer properties. Recent research has demonstrated that such models, when
+prompted appropriately, are excellent for few-shot inference. However, such
+techniques are under-explored for dense prediction tasks like semantic
+segmentation. In this work, we examine the effectiveness of prompting a
+transformer-decoder with learned visual prompts for the generalized few-shot
+segmentation (GFSS) task. Our goal is to achieve strong performance not only on
+novel categories with limited examples, but also to retain performance on base
+categories. We propose an approach to learn visual prompts with limited
+examples. These learned visual prompts are used to prompt a multiscale
+transformer decoder to facilitate accurate dense predictions. Additionally, we
+introduce a unidirectional causal attention mechanism between the novel
+prompts, learned with limited examples, and the base prompts, learned with
+abundant data. This mechanism enriches the novel prompts without deteriorating
+the base class performance. Overall, this form of prompting helps us achieve
+state-of-the-art performance for GFSS on two different benchmark datasets:
+COCO-$20^i$ and Pascal-$5^i$, without the need for test-time optimization (or
+transduction). Furthermore, test-time optimization leveraging unlabelled test
+data can be used to improve the prompts, which we refer to as transductive
+prompt tuning.",cs.CV,['cs.CV']
+Precise Image Editing via Recognition and Generation Tasks,Shelly Sheynin · Adam Polyak · Uriel Singer · Yuval Kirstain · Amit Zohar · Oron Ashual · Devi Parikh · Yaniv Taigman,https://emu-edit.metademolab.com/,https://arxiv.org/abs/2311.10089,,2311.10089.pdf,Emu Edit: Precise Image Editing via Recognition and Generation Tasks,"Instruction-based image editing holds immense potential for a variety of
+applications, as it enables users to perform any editing operation using a
+natural language instruction. However, current models in this domain often
+struggle with accurately executing user instructions. We present Emu Edit, a
+multi-task image editing model which sets state-of-the-art results in
+instruction-based image editing. To develop Emu Edit we train it to multi-task
+across an unprecedented range of tasks, such as region-based editing, free-form
+editing, and Computer Vision tasks, all of which are formulated as generative
+tasks. Additionally, to enhance Emu Edit's multi-task learning abilities, we
+provide it with learned task embeddings which guide the generation process
+towards the correct edit type. Both these elements are essential for Emu Edit's
+outstanding performance. Furthermore, we show that Emu Edit can generalize to
+new tasks, such as image inpainting, super-resolution, and compositions of
+editing tasks, with just a few labeled examples. This capability offers a
+significant advantage in scenarios where high-quality samples are scarce.
+Lastly, to facilitate a more rigorous and informed assessment of instructable
+image editing models, we release a new challenging and versatile benchmark that
+includes seven different image editing tasks.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+UniVS: Unified and Universal Video Segmentation with Prompts as Queries,Minghan LI · Shuai Li · Xindong Zhang · Lei Zhang, ,https://arxiv.org/abs/2402.18115,,2402.18115.pdf,UniVS: Unified and Universal Video Segmentation with Prompts as Queries,"Despite the recent advances in unified image segmentation (IS), developing a
+unified video segmentation (VS) model remains a challenge. This is mainly
+because generic category-specified VS tasks need to detect all objects and
+track them across consecutive frames, while prompt-guided VS tasks require
+re-identifying the target with visual/text prompts throughout the entire video,
+making it hard to handle the different tasks with the same architecture. We
+make an attempt to address these issues and present a novel unified VS
+architecture, namely UniVS, by using prompts as queries. UniVS averages the
+prompt features of the target from previous frames as its initial query to
+explicitly decode masks, and introduces a target-wise prompt cross-attention
+layer in the mask decoder to integrate prompt features in the memory pool. By
+taking the predicted masks of entities from previous frames as their visual
+prompts, UniVS converts different VS tasks into prompt-guided target
+segmentation, eliminating the heuristic inter-frame matching process. Our
+framework not only unifies the different VS tasks but also naturally achieves
+universal training and testing, ensuring robust performance across different
+scenarios. UniVS shows a commendable balance between performance and
+universality on 10 challenging VS benchmarks, covering video instance,
+semantic, panoptic, object, and referring segmentation tasks. Code can be found
+at \url{https://github.com/MinghanLi/UniVS}.",cs.CV,"['cs.CV', 'cs.CL']"
+A-Teacher: Asymmetric Network for 3D Semi-Supervised Object Detection,Hanshi Wang · Zhipeng Zhang · Jin Gao · Weiming Hu, ,https://arxiv.org/abs/2401.05011,,2401.05011.pdf,Dual-Perspective Knowledge Enrichment for Semi-Supervised 3D Object Detection,"Semi-supervised 3D object detection is a promising yet under-explored
+direction to reduce data annotation costs, especially for cluttered indoor
+scenes. A few prior works, such as SESS and 3DIoUMatch, attempt to solve this
+task by utilizing a teacher model to generate pseudo-labels for unlabeled
+samples. However, the availability of unlabeled samples in the 3D domain is
+relatively limited compared to its 2D counterpart due to the greater effort
+required to collect 3D data. Moreover, the loose consistency regularization in
+SESS and restricted pseudo-label selection strategy in 3DIoUMatch lead to
+either low-quality supervision or a limited amount of pseudo labels. To address
+these issues, we present a novel Dual-Perspective Knowledge Enrichment approach
+named DPKE for semi-supervised 3D object detection. Our DPKE enriches the
+knowledge of limited training data, particularly unlabeled data, from two
+perspectives: data-perspective and feature-perspective. Specifically, from the
+data-perspective, we propose a class-probabilistic data augmentation method
+that augments the input data with additional instances based on the varying
+distribution of class probabilities. Our DPKE achieves feature-perspective
+knowledge enrichment by designing a geometry-aware feature matching method that
+regularizes feature-level similarity between object proposals from the student
+and teacher models. Extensive experiments on the two benchmark datasets
+demonstrate that our DPKE achieves superior performance over existing
+state-of-the-art approaches under various label ratio conditions. The source
+code will be made available to the public.",cs.CV,['cs.CV']
+MRFS: Mutually Reinforcing Image Fusion and Segmentation,HAO ZHANG · Xuhui Zuo · Jie Jiang · Chunchao Guo · Jiayi Ma, ,,https://ojs.aaai.org/index.php/AAAI/article/view/28536,,,,,nan
+OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning,Haiyang Ying · Yixuan Yin · Jinzhi Zhang · Fan Wang · Tao Yu · Ruqi Huang · Lu Fang,https://oceanying.github.io/OmniSeg3D/,https://arxiv.org/abs/2311.11666,,2311.11666.pdf,OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning,"Towards holistic understanding of 3D scenes, a general 3D segmentation method
+is needed that can segment diverse objects without restrictions on object
+quantity or categories, while also reflecting the inherent hierarchical
+structure. To achieve this, we propose OmniSeg3D, an omniversal segmentation
+method aims for segmenting anything in 3D all at once. The key insight is to
+lift multi-view inconsistent 2D segmentations into a consistent 3D feature
+field through a hierarchical contrastive learning framework, which is
+accomplished by two steps. Firstly, we design a novel hierarchical
+representation based on category-agnostic 2D segmentations to model the
+multi-level relationship among pixels. Secondly, image features rendered from
+the 3D feature field are clustered at different levels, which can be further
+drawn closer or pushed apart according to the hierarchical relationship between
+different levels. In tackling the challenges posed by inconsistent 2D
+segmentations, this framework yields a global consistent 3D feature field,
+which further enables hierarchical segmentation, multi-object selection, and
+global discretization. Extensive experiments demonstrate the effectiveness of
+our method on high-quality 3D segmentation and accurate hierarchical structure
+understanding. A graphical user interface further facilitates flexible
+interaction for omniversal 3D segmentation.",cs.CV,['cs.CV']
+Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance,Kelvin C.K. Chan · Yang Zhao · Xuhui Jia · Ming-Hsuan Yang · Huisheng Wang, ,https://arxiv.org/abs/2405.01356,,2405.01356.pdf,Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance,"In subject-driven text-to-image synthesis, the synthesis process tends to be
+heavily influenced by the reference images provided by users, often overlooking
+crucial attributes detailed in the text prompt. In this work, we propose
+Subject-Agnostic Guidance (SAG), a simple yet effective solution to remedy the
+problem. We show that through constructing a subject-agnostic condition and
+applying our proposed dual classifier-free guidance, one could obtain outputs
+consistent with both the given subject and input text prompts. We validate the
+efficacy of our approach through both optimization-based and encoder-based
+methods. Additionally, we demonstrate its applicability in second-order
+customization methods, where an encoder-based model is fine-tuned with
+DreamBooth. Our approach is conceptually simple and requires only minimal code
+modifications, but leads to substantial quality improvements, as evidenced by
+our evaluations and user studies.",cs.CV,['cs.CV']
+DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement,Hao Wu · Huabin Liu · Yu Qiao · Xiao Sun, ,https://arxiv.org/abs/2404.02755,,2404.02755.pdf,DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement,"We present Dive Into the BoundarieS (DIBS), a novel pretraining framework for
+dense video captioning (DVC), that elaborates on improving the quality of the
+generated event captions and their associated pseudo event boundaries from
+unlabeled videos. By leveraging the capabilities of diverse large language
+models (LLMs), we generate rich DVC-oriented caption candidates and optimize
+the corresponding pseudo boundaries under several meticulously designed
+objectives, considering diversity, event-centricity, temporal ordering, and
+coherence. Moreover, we further introduce a novel online boundary refinement
+strategy that iteratively improves the quality of pseudo boundaries during
+training. Comprehensive experiments have been conducted to examine the
+effectiveness of the proposed technique components. By leveraging a substantial
+amount of unlabeled video data, such as HowTo100M, we achieve a remarkable
+advancement on standard DVC datasets like YouCook2 and ActivityNet. We
+outperform the previous state-of-the-art Vid2Seq across a majority of metrics,
+achieving this with just 0.4% of the unlabeled video data used for pre-training
+by Vid2Seq.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM']"
+AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution,Cheeun Hong · Kyoung Mu Lee, ,https://arxiv.org/abs/2404.03296,,2404.03296.pdf,AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution,"Although image super-resolution (SR) problem has experienced unprecedented
+restoration accuracy with deep neural networks, it has yet limited versatile
+applications due to the substantial computational costs. Since different input
+images for SR face different restoration difficulties, adapting computational
+costs based on the input image, referred to as adaptive inference, has emerged
+as a promising solution to compress SR networks. Specifically, adapting the
+quantization bit-widths has successfully reduced the inference and memory cost
+without sacrificing the accuracy. However, despite the benefits of the
+resultant adaptive network, existing works rely on time-intensive
+quantization-aware training with full access to the original training pairs to
+learn the appropriate bit allocation policies, which limits its ubiquitous
+usage. To this end, we introduce the first on-the-fly adaptive quantization
+framework that accelerates the processing time from hours to seconds. We
+formulate the bit allocation problem with only two bit mapping modules: one to
+map the input image to the image-wise bit adaptation factor and one to obtain
+the layer-wise adaptation factors. These bit mappings are calibrated and
+fine-tuned using only a small number of calibration images. We achieve
+competitive performance with the previous adaptive quantization methods, while
+the processing time is accelerated by x2000. Codes are available at
+https://github.com/Cheeun/AdaBM.",cs.CV,"['cs.CV', 'eess.IV']"
+Residual Denoising Diffusion Models,Jiawei Liu · Qiang Wang · Huijie Fan · Yinong Wang · Yandong Tang · Liangqiong Qu,https://github.com/nachifur/RDDM,https://arxiv.org/abs/2308.13712,,,Residual Denoising Diffusion Models,"We propose residual denoising diffusion models (RDDM), a novel dual diffusion
+process that decouples the traditional single denoising diffusion process into
+residual diffusion and noise diffusion. This dual diffusion framework expands
+the denoising-based diffusion models, initially uninterpretable for image
+restoration, into a unified and interpretable model for both image generation
+and restoration by introducing residuals. Specifically, our residual diffusion
+represents directional diffusion from the target image to the degraded input
+image and explicitly guides the reverse generation process for image
+restoration, while noise diffusion represents random perturbations in the
+diffusion process. The residual prioritizes certainty, while the noise
+emphasizes diversity, enabling RDDM to effectively unify tasks with varying
+certainty or diversity requirements, such as image generation and restoration.
+We demonstrate that our sampling process is consistent with that of DDPM and
+DDIM through coefficient transformation, and propose a partially
+path-independent generation process to better understand the reverse process.
+Notably, our RDDM enables a generic UNet, trained with only an L1 loss and a
+batch size of 1, to compete with state-of-the-art image restoration methods. We
+provide code and pre-trained models to encourage further exploration,
+application, and development of our innovative framework
+(https://github.com/nachifur/RDDM).",cs.CV,"['cs.CV', 'cs.LG']"
+Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse,Yining Wang · Junjie Sun · Chenyue Wang · Mi Zhang · Min Yang, ,https://arxiv.org/abs/2405.05587,,2405.05587.pdf,Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse,"Recent studies have noted an intriguing phenomenon termed Neural Collapse,
+that is, when the neural networks establish the right correlation between
+feature spaces and the training targets, their last-layer features, together
+with the classifier weights, will collapse into a stable and symmetric
+structure. In this paper, we extend the investigation of Neural Collapse to the
+biased datasets with imbalanced attributes. We observe that models will easily
+fall into the pitfall of shortcut learning and form a biased, non-collapsed
+feature space at the early period of training, which is hard to reverse and
+limits the generalization capability. To tackle the root cause of biased
+classification, we follow the recent inspiration of prime training, and propose
+an avoid-shortcut learning framework without additional training complexity.
+With well-designed shortcut primes based on Neural Collapse structure, the
+models are encouraged to skip the pursuit of simple shortcuts and naturally
+capture the intrinsic correlations. Experimental results demonstrate that our
+method induces better convergence properties during training, and achieves
+state-of-the-art generalization performance on both synthetic and real-world
+biased datasets.",cs.CV,"['cs.CV', 'cs.LG']"
+TIGER: Time-Varying Denoising Model for 3D Point Cloud Generation with Diffusion Process,Zhiyuan Ren · Minchul Kim · Feng Liu · Xiaoming Liu, ,,https://link.springer.com/article/10.1007/s00371-024-03370-x,,,,,nan
+IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation,Yizhi Song · Zhifei Zhang · Zhe Lin · Scott Cohen · Brian Price · Jianming Zhang · Soo Ye Kim · He Zhang · Wei Xiong · Daniel Aliaga,https://song630.github.io/IMPRINT-Project-Page/,https://arxiv.org/abs/2403.10701,,2403.10701.pdf,IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation,"Generative object compositing emerges as a promising new avenue for
+compositional image editing. However, the requirement of object identity
+preservation poses a significant challenge, limiting practical usage of most
+existing methods. In response, this paper introduces IMPRINT, a novel
+diffusion-based generative model trained with a two-stage learning framework
+that decouples learning of identity preservation from that of compositing. The
+first stage is targeted for context-agnostic, identity-preserving pretraining
+of the object encoder, enabling the encoder to learn an embedding that is both
+view-invariant and conducive to enhanced detail preservation. The subsequent
+stage leverages this representation to learn seamless harmonization of the
+object composited to the background. In addition, IMPRINT incorporates a
+shape-guidance mechanism offering user-directed control over the compositing
+process. Extensive experiments demonstrate that IMPRINT significantly
+outperforms existing methods and various baselines on identity preservation and
+composition quality.",cs.CV,['cs.CV']
+Differentiable Micro-Mesh Construction,Yishun Dou · Zhong Zheng · Qiaoqiao Jin · Rui Shi · Yuhan Li · Bingbing Ni, ,http://export.arxiv.org/abs/2310.08332v1,,2310.08332v1.pdf,Real-Time Neural BRDF with Spherically Distributed Primitives,"We propose a novel compact and efficient neural BRDF offering highly
+versatile material representation, yet with very-light memory and neural
+computation consumption towards achieving real-time rendering. The results in
+Figure 1, rendered at full HD resolution on a current desktop machine, show
+that our system achieves real-time rendering with a wide variety of
+appearances, which is approached by the following two designs. On the one hand,
+noting that bidirectional reflectance is distributed in a very sparse
+high-dimensional subspace, we propose to project the BRDF into two
+low-dimensional components, i.e., two hemisphere feature-grids for incoming and
+outgoing directions, respectively. On the other hand, learnable neural
+reflectance primitives are distributed on our highly-tailored spherical surface
+grid, which offer informative features for each component and alleviate the
+conventional heavy feature learning network to a much smaller one, leading to
+very fast evaluation. These primitives are centrally stored in a codebook and
+can be shared across multiple grids and even across materials, based on the
+low-cost indices stored in material-specific spherical surface grids. Our
+neural BRDF, which is agnostic to the material, provides a unified framework
+that can represent a variety of materials in consistent manner. Comprehensive
+experimental results on measured BRDF compression, Monte Carlo simulated BRDF
+acceleration, and extension to spatially varying effect demonstrate the
+superior quality and generalizability achieved by the proposed scheme.",cs.CV,['cs.CV']
+FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization,Jiahui Zhang · Fangneng Zhan · MUYU XU · Shijian Lu · Eric P. Xing, ,https://arxiv.org/abs/2403.06908v1,,2403.06908v1.pdf,FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization,"3D Gaussian splatting has achieved very impressive performance in real-time
+novel view synthesis. However, it often suffers from over-reconstruction during
+Gaussian densification where high-variance image regions are covered by a few
+large Gaussians only, leading to blur and artifacts in the rendered images. We
+design a progressive frequency regularization (FreGS) technique to tackle the
+over-reconstruction issue within the frequency space. Specifically, FreGS
+performs coarse-to-fine Gaussian densification by exploiting low-to-high
+frequency components that can be easily extracted with low-pass and high-pass
+filters in the Fourier space. By minimizing the discrepancy between the
+frequency spectrum of the rendered image and the corresponding ground truth, it
+achieves high-quality Gaussian densification and alleviates the
+over-reconstruction of Gaussian splatting effectively. Experiments over
+multiple widely adopted benchmarks (e.g., Mip-NeRF360, Tanks-and-Temples and
+Deep Blending) show that FreGS achieves superior novel view synthesis and
+outperforms the state-of-the-art consistently.",cs.CV,['cs.CV']
+Parameter Efficient Self-Supervised Geospatial Domain Adaptation,Linus Scheibenreif · Michael Mommert · Damian Borth, ,https://arxiv.org/abs/2312.13066,,2312.13066.pdf,PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation,"Self-supervised monocular depth estimation is of significant importance with
+applications spanning across autonomous driving and robotics. However, the
+reliance on self-supervision introduces a strong static-scene assumption,
+thereby posing challenges in achieving optimal performance in dynamic scenes,
+which are prevalent in most real-world situations. To address these issues, we
+propose PPEA-Depth, a Progressive Parameter-Efficient Adaptation approach to
+transfer a pre-trained image model for self-supervised depth estimation. The
+training comprises two sequential stages: an initial phase trained on a dataset
+primarily composed of static scenes, succeeded by an expansion to more
+intricate datasets involving dynamic scenes. To facilitate this process, we
+design compact encoder and decoder adapters to enable parameter-efficient
+tuning, allowing the network to adapt effectively. They not only uphold
+generalized patterns from pre-trained image models but also retain knowledge
+gained from the preceding phase into the subsequent one. Extensive experiments
+demonstrate that PPEA-Depth achieves state-of-the-art performance on KITTI,
+CityScapes and DDAD datasets.",cs.CV,['cs.CV']
+Question Aware Vision Transformer for Multimodal Reasoning,Roy Ganz · Yair Kittenplon · Aviad Aberdam · Elad Ben Avraham · Oren Nuriel · Shai Mazor · Ron Litman, ,https://arxiv.org/abs/2402.05472,,2402.05472.pdf,Question Aware Vision Transformer for Multimodal Reasoning,"Vision-Language (VL) models have gained significant research focus, enabling
+remarkable advances in multimodal reasoning. These architectures typically
+comprise a vision encoder, a Large Language Model (LLM), and a projection
+module that aligns visual features with the LLM's representation space. Despite
+their success, a critical limitation persists: the vision encoding process
+remains decoupled from user queries, often in the form of image-related
+questions. Consequently, the resulting visual features may not be optimally
+attuned to the query-specific elements of the image. To address this, we
+introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
+reasoning, which embeds question awareness directly within the vision encoder.
+This integration results in dynamic visual features focusing on relevant image
+aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
+efficiently into any VL architecture. Extensive experiments demonstrate the
+effectiveness of applying our method to various multimodal architectures,
+leading to consistent improvement across diverse tasks and showcasing its
+potential for enhancing visual and scene-text understanding.",cs.CV,['cs.CV']
+Real-Time Neural BRDF with Spherically Distributed Primitives,Yishun Dou · Zhong Zheng · Qiaoqiao Jin · Bingbing Ni · Yugang Chen · Junxiang Ke, ,https://arxiv.org/abs/2310.08332,,2310.08332.pdf,Real-Time Neural BRDF with Spherically Distributed Primitives,"We propose a novel compact and efficient neural BRDF offering highly
+versatile material representation, yet with very-light memory and neural
+computation consumption towards achieving real-time rendering. The results in
+Figure 1, rendered at full HD resolution on a current desktop machine, show
+that our system achieves real-time rendering with a wide variety of
+appearances, which is approached by the following two designs. On the one hand,
+noting that bidirectional reflectance is distributed in a very sparse
+high-dimensional subspace, we propose to project the BRDF into two
+low-dimensional components, i.e., two hemisphere feature-grids for incoming and
+outgoing directions, respectively. On the other hand, learnable neural
+reflectance primitives are distributed on our highly-tailored spherical surface
+grid, which offer informative features for each component and alleviate the
+conventional heavy feature learning network to a much smaller one, leading to
+very fast evaluation. These primitives are centrally stored in a codebook and
+can be shared across multiple grids and even across materials, based on the
+low-cost indices stored in material-specific spherical surface grids. Our
+neural BRDF, which is agnostic to the material, provides a unified framework
+that can represent a variety of materials in consistent manner. Comprehensive
+experimental results on measured BRDF compression, Monte Carlo simulated BRDF
+acceleration, and extension to spatially varying effect demonstrate the
+superior quality and generalizability achieved by the proposed scheme.",cs.CV,['cs.CV']
+Dispel Darkness for Better Fusion: A Controllable Visual Enhancer based on Cross-modal Conditional Adversarial Learning,HAO ZHANG · Linfeng Tang · Xinyu Xiang · Xuhui Zuo · Jiayi Ma, ,,https://github.com/HaoZhang1018/DDBF,,,,,nan
+HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention,Xiaolong Tang · Meina Kan · Shiguang Shan · Zhilong Ji · Jinfeng Bai · Xilin Chen, ,https://arxiv.org/abs/2404.06351,,2404.06351.pdf,HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention,"Predicting the trajectories of road agents is essential for autonomous
+driving systems. The recent mainstream methods follow a static paradigm, which
+predicts the future trajectory by using a fixed duration of historical frames.
+These methods make the predictions independently even at adjacent time steps,
+which leads to potential instability and temporal inconsistency. As successive
+time steps have largely overlapping historical frames, their forecasting should
+have intrinsic correlation, such as overlapping predicted trajectories should
+be consistent, or be different but share the same motion goal depending on the
+road situation. Motivated by this, in this work, we introduce HPNet, a novel
+dynamic trajectory forecasting method. Aiming for stable and accurate
+trajectory forecasting, our method leverages not only historical frames
+including maps and agent states, but also historical predictions. Specifically,
+we newly design a Historical Prediction Attention module to automatically
+encode the dynamic relationship between successive predictions. Besides, it
+also extends the attention range beyond the currently visible window
+benefitting from the use of historical predictions. The proposed Historical
+Prediction Attention together with the Agent Attention and Mode Attention is
+further formulated as the Triple Factorized Attention module, serving as the
+core design of HPNet.Experiments on the Argoverse and INTERACTION datasets show
+that HPNet achieves state-of-the-art performance, and generates accurate and
+stable future trajectories. Our code are available at
+https://github.com/XiaolongTang23/HPNet.",cs.CV,['cs.CV']
+Scene-adaptive and Region-aware Multi-modal Prompt for Open Vocabulary Object Detection,Xiaowei Zhao · Xianglong Liu · Duorui Wang · Yajun Gao · Zhide Liu, ,https://arxiv.org/abs/2306.05493,,,Multi-Modal Classifiers for Open-Vocabulary Object Detection,"The goal of this paper is open-vocabulary object detection (OVOD)
+$\unicode{x2013}$ building a model that can detect objects beyond the set of
+categories seen at training, thus enabling the user to specify categories of
+interest at inference without the need for model retraining. We adopt a
+standard two-stage object detector architecture, and explore three ways for
+specifying novel categories: via language descriptions, via image exemplars, or
+via a combination of the two. We make three contributions: first, we prompt a
+large language model (LLM) to generate informative language descriptions for
+object classes, and construct powerful text-based classifiers; second, we
+employ a visual aggregator on image exemplars that can ingest any number of
+images as input, forming vision-based classifiers; and third, we provide a
+simple method to fuse information from language descriptions and image
+exemplars, yielding a multi-modal classifier. When evaluating on the
+challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our
+text-based classifiers outperform all previous OVOD works; (ii) our
+vision-based classifiers perform as well as text-based classifiers in prior
+work; (iii) using multi-modal classifiers perform better than either modality
+alone; and finally, (iv) our text-based and multi-modal classifiers yield
+better performance than a fully-supervised detector.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'I.4.6; I.4.8; I.4.9; I.2.10']"
+CAPE: CAM as a Probabilistic Ensemble for Enhanced DNN Interpretation,Townim Chowdhury · Kewen Liao · Vu Minh Hieu Phan · Minh-Son To · Yutong Xie · Kevin Hung · David Ross · Anton van den Hengel · Johan Verjans · Zhibin Liao, ,https://arxiv.org/abs/2404.02388,,2404.02388.pdf,CAPE: CAM as a Probabilistic Ensemble for Enhanced DNN Interpretation,"Deep Neural Networks (DNNs) are widely used for visual classification tasks,
+but their complex computation process and black-box nature hinder decision
+transparency and interpretability. Class activation maps (CAMs) and recent
+variants provide ways to visually explain the DNN decision-making process by
+displaying 'attention' heatmaps of the DNNs. Nevertheless, the CAM explanation
+only offers relative attention information, that is, on an attention heatmap,
+we can interpret which image region is more or less important than the others.
+However, these regions cannot be meaningfully compared across classes, and the
+contribution of each region to the model's class prediction is not revealed. To
+address these challenges that ultimately lead to better DNN Interpretation, in
+this paper, we propose CAPE, a novel reformulation of CAM that provides a
+unified and probabilistically meaningful assessment of the contributions of
+image regions. We quantitatively and qualitatively compare CAPE with
+state-of-the-art CAM methods on CUB and ImageNet benchmark datasets to
+demonstrate enhanced interpretability. We also test on a cytology imaging
+dataset depicting a challenging Chronic Myelomonocytic Leukemia (CMML)
+diagnosis problem. Code is available at: https://github.com/AIML-MED/CAPE.",cs.CV,['cs.CV']
+Focus on Hiders: Exploring Hidden Threats for Enhancing Adversarial Training,Qian Li · Yuxiao Hu · Yinpeng Dong · Dongxiao Zhang · Yuntian Chen, ,https://arxiv.org/abs/2312.07067,,2312.07067.pdf,Focus on Hiders: Exploring Hidden Threats for Enhancing Adversarial Training,"Adversarial training is often formulated as a min-max problem, however,
+concentrating only on the worst adversarial examples causes alternating
+repetitive confusion of the model, i.e., previously defended or correctly
+classified samples are not defensible or accurately classifiable in subsequent
+adversarial training. We characterize such non-ignorable samples as ""hiders"",
+which reveal the hidden high-risk regions within the secure area obtained
+through adversarial training and prevent the model from finding the real worst
+cases. We demand the model to prevent hiders when defending against adversarial
+examples for improving accuracy and robustness simultaneously. By rethinking
+and redefining the min-max optimization problem for adversarial training, we
+propose a generalized adversarial training algorithm called Hider-Focused
+Adversarial Training (HFAT). HFAT introduces the iterative evolution
+optimization strategy to simplify the optimization problem and employs an
+auxiliary model to reveal hiders, effectively combining the optimization
+directions of standard adversarial training and prevention hiders. Furthermore,
+we introduce an adaptive weighting mechanism that facilitates the model in
+adaptively adjusting its focus between adversarial examples and hiders during
+different training periods. We demonstrate the effectiveness of our method
+based on extensive experiments, and ensure that HFAT can provide higher
+robustness and accuracy.",cs.LG,"['cs.LG', 'cs.CR', 'cs.CV', 'stat.AP']"
+Multi-Space Alignments Towards Universal LiDAR Segmentation,Youquan Liu · Lingdong Kong · Xiaoyang Wu · Runnan Chen · Xin Li · Liang Pan · Ziwei Liu · Yuexin Ma, ,https://arxiv.org/abs/2405.01538,,2405.01538.pdf,Multi-Space Alignments Towards Universal LiDAR Segmentation,"A unified and versatile LiDAR segmentation model with strong robustness and
+generalizability is desirable for safe autonomous driving perception. This work
+presents M3Net, a one-of-a-kind framework for fulfilling multi-task,
+multi-dataset, multi-modality LiDAR segmentation in a universal manner using
+just a single set of parameters. To better exploit data volume and diversity,
+we first combine large-scale driving datasets acquired by different types of
+sensors from diverse scenes and then conduct alignments in three spaces, namely
+data, feature, and label spaces, during the training. As a result, M3Net is
+capable of taming heterogeneous data for training state-of-the-art LiDAR
+segmentation models. Extensive experiments on twelve LiDAR segmentation
+datasets verify our effectiveness. Notably, using a shared set of parameters,
+M3Net achieves 75.1%, 83.1%, and 72.4% mIoU scores, respectively, on the
+official benchmarks of SemanticKITTI, nuScenes, and Waymo Open.",cs.CV,"['cs.CV', 'cs.LG', 'cs.RO']"
+Fast ODE-based Sampling for Diffusion Models in Around 5 Steps,Zhenyu Zhou · Defang Chen · Can Wang · Chun Chen, ,https://arxiv.org/abs/2312.00094,,2312.00094.pdf,Fast ODE-based Sampling for Diffusion Models in Around 5 Steps,"Sampling from diffusion models can be treated as solving the corresponding
+ordinary differential equations (ODEs), with the aim of obtaining an accurate
+solution with as few number of function evaluations (NFE) as possible.
+Recently, various fast samplers utilizing higher-order ODE solvers have emerged
+and achieved better performance than the initial first-order one. However,
+these numerical methods inherently result in certain approximation errors,
+which significantly degrades sample quality with extremely small NFE (e.g.,
+around 5). In contrast, based on the geometric observation that each sampling
+trajectory almost lies in a two-dimensional subspace embedded in the ambient
+space, we propose Approximate MEan-Direction Solver (AMED-Solver) that
+eliminates truncation errors by directly learning the mean direction for fast
+diffusion sampling. Besides, our method can be easily used as a plugin to
+further improve existing ODE-based samplers. Extensive experiments on image
+synthesis with the resolution ranging from 32 to 512 demonstrate the
+effectiveness of our method. With only 5 NFE, we achieve 6.61 FID on CIFAR-10,
+10.74 FID on ImageNet 64$\times$64, and 13.20 FID on LSUN Bedroom. Our code is
+available at https://github.com/zju-pi/diff-sampler.",cs.CV,"['cs.CV', 'cs.AI']"
+OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies,Lingdong Kong · Youquan Liu · Lai Xing Ng · Benoit Cottereau · Wei Tsang Ooi,https://github.com/ldkong1205/OpenESS,http://export.arxiv.org/abs/2405.05259,,2405.05259.pdf,OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies,"Event-based semantic segmentation (ESS) is a fundamental yet challenging task
+for event camera sensing. The difficulties in interpreting and annotating event
+data limit its scalability. While domain adaptation from images to event data
+can help to mitigate this issue, there exist data representational differences
+that require additional effort to resolve. In this work, for the first time, we
+synergize information from image, text, and event-data domains and introduce
+OpenESS to enable scalable ESS in an open-world, annotation-efficient manner.
+We achieve this goal by transferring the semantically rich CLIP knowledge from
+image-text pairs to event streams. To pursue better cross-modality adaptation,
+we propose a frame-to-event contrastive distillation and a text-to-event
+semantic consistency regularization. Experimental results on popular ESS
+benchmarks showed our approach outperforms existing methods. Notably, we
+achieve 53.93% and 43.31% mIoU on DDD17 and DSEC-Semantic without using either
+event or frame labels.",cs.CV,"['cs.CV', 'cs.RO']"
+Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior,Zike Wu · Pan Zhou · YI Xuanyu · Xiaoding Yuan · Hanwang Zhang, ,,https://paperswithcode.com/paper/consistent3d-towards-consistent-high-fidelity,,,,,nan
+VMINer: Versatile Multi-view Inverse Rendering with Near- and Far-field Light Sources,Fan Fei · Jiajun Tang · Ping Tan · Boxin Shi,https://costrice.github.io/vminer/,https://arxiv.org/abs/2402.06136,,2402.06136.pdf,SIR: Multi-view Inverse Rendering with Decomposable Shadow for Indoor Scenes,"We propose SIR, an efficient method to decompose differentiable shadows for
+inverse rendering on indoor scenes using multi-view data, addressing the
+challenges in accurately decomposing the materials and lighting conditions.
+Unlike previous methods that struggle with shadow fidelity in complex lighting
+environments, our approach explicitly learns shadows for enhanced realism in
+material estimation under unknown light positions. Utilizing posed HDR images
+as input, SIR employs an SDF-based neural radiance field for comprehensive
+scene representation. Then, SIR integrates a shadow term with a three-stage
+material estimation approach to improve SVBRDF quality. Specifically, SIR is
+designed to learn a differentiable shadow, complemented by BRDF regularization,
+to optimize inverse rendering accuracy. Extensive experiments on both synthetic
+and real-world indoor scenes demonstrate the superior performance of SIR over
+existing methods in both quantitative metrics and qualitative analysis. The
+significant decomposing ability of SIR enables sophisticated editing
+capabilities like free-view relighting, object insertion, and material
+replacement. The code and data are available at
+https://xiaokangwei.github.io/SIR/.",cs.CV,['cs.CV']
+Weak-to-Strong 3D Object Detection with X-Ray Distillation,Alexander Gambashidze · Aleksandr Dadukin · Maksim Golyadkin · Maria Razzhivina · Ilya Makarov, ,https://arxiv.org/abs/2404.00679,,2404.00679.pdf,Weak-to-Strong 3D Object Detection with X-Ray Distillation,"This paper addresses the critical challenges of sparsity and occlusion in
+LiDAR-based 3D object detection. Current methods often rely on supplementary
+modules or specific architectural designs, potentially limiting their
+applicability to new and evolving architectures. To our knowledge, we are the
+first to propose a versatile technique that seamlessly integrates into any
+existing framework for 3D Object Detection, marking the first instance of
+Weak-to-Strong generalization in 3D computer vision. We introduce a novel
+framework, X-Ray Distillation with Object-Complete Frames, suitable for both
+supervised and semi-supervised settings, that leverages the temporal aspect of
+point cloud sequences. This method extracts crucial information from both
+previous and subsequent LiDAR frames, creating Object-Complete frames that
+represent objects from multiple viewpoints, thus addressing occlusion and
+sparsity. Given the limitation of not being able to generate Object-Complete
+frames during online inference, we utilize Knowledge Distillation within a
+Teacher-Student framework. This technique encourages the strong Student model
+to emulate the behavior of the weaker Teacher, which processes simple and
+informative Object-Complete frames, effectively offering a comprehensive view
+of objects as if seen through X-ray vision. Our proposed methods surpass
+state-of-the-art in semi-supervised learning by 1-1.5 mAP and enhance the
+performance of five established supervised models by 1-2 mAP on standard
+autonomous driving datasets, even with default hyperparameters. Code for
+Object-Complete frames is available here:
+https://github.com/sakharok13/X-Ray-Teacher-Patching-Tools.",cs.CV,['cs.CV']
+AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents,Jieming Cui · Tengyu Liu · Nian Liu · Yaodong Yang · Yixin Zhu · Siyuan Huang, ,https://arxiv.org/abs/2403.12835,,2403.12835.pdf,AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents,"Traditional approaches in physics-based motion generation, centered around
+imitation learning and reward shaping, often struggle to adapt to new
+scenarios. To tackle this limitation, we propose AnySkill, a novel hierarchical
+method that learns physically plausible interactions following open-vocabulary
+instructions. Our approach begins by developing a set of atomic actions via a
+low-level controller trained via imitation learning. Upon receiving an
+open-vocabulary textual instruction, AnySkill employs a high-level policy that
+selects and integrates these atomic actions to maximize the CLIP similarity
+between the agent's rendered images and the text. An important feature of our
+method is the use of image-based rewards for the high-level policy, which
+allows the agent to learn interactions with objects without manual reward
+engineering. We demonstrate AnySkill's capability to generate realistic and
+natural motion sequences in response to unseen instructions of varying lengths,
+marking it the first method capable of open-vocabulary physical skill learning
+for interactive humanoid agents.",cs.CV,"['cs.CV', 'cs.RO']"
+Learning Continuous 3D Words for Text-to-Image Generation,Ta-Ying Cheng · Matheus Gadelha · Thibault Groueix · Matthew Fisher · Radomir Mech · Andrew Markham · Niki Trigoni,https://ttchengab.github.io/continuous_3d_words/,https://arxiv.org/abs/2402.08654,,2402.08654.pdf,Learning Continuous 3D Words for Text-to-Image Generation,"Current controls over diffusion models (e.g., through text or ControlNet) for
+image generation fall short in recognizing abstract, continuous attributes like
+illumination direction or non-rigid shape change. In this paper, we present an
+approach for allowing users of text-to-image models to have fine-grained
+control of several attributes in an image. We do this by engineering special
+sets of input tokens that can be transformed in a continuous manner -- we call
+them Continuous 3D Words. These attributes can, for example, be represented as
+sliders and applied jointly with text prompts for fine-grained control over
+image generation. Given only a single mesh and a rendering engine, we show that
+our approach can be adopted to provide continuous user control over several
+3D-aware attributes, including time-of-day illumination, bird wing orientation,
+dollyzoom effect, and object poses. Our method is capable of conditioning image
+creation with multiple Continuous 3D Words and text descriptions simultaneously
+while adding no overhead to the generative process. Project Page:
+https://ttchengab.github.io/continuous_3d_words",cs.CV,['cs.CV']
+Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model,Xu He · Qiaochu Huang · Zhensong Zhang · Zhiwei Lin · Zhiyong Wu · Sicheng Yang · Minglei Li · Zhiyi Chen · Songcen Xu · Xiaofei Wu, ,https://arxiv.org/abs/2404.01862v1,,2404.01862v1.pdf,Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model,"Co-speech gestures, if presented in the lively form of videos, can achieve
+superior visual effects in human-machine interaction. While previous works
+mostly generate structural human skeletons, resulting in the omission of
+appearance information, we focus on the direct generation of audio-driven
+co-speech gesture videos in this work. There are two main challenges: 1) A
+suitable motion feature is needed to describe complex human movements with
+crucial appearance information. 2) Gestures and speech exhibit inherent
+dependencies and should be temporally aligned even of arbitrary length. To
+solve these problems, we present a novel motion-decoupled framework to generate
+co-speech gesture videos. Specifically, we first introduce a well-designed
+nonlinear TPS transformation to obtain latent motion features preserving
+essential appearance information. Then a transformer-based diffusion model is
+proposed to learn the temporal correlation between gestures and speech, and
+performs generation in the latent motion space, followed by an optimal motion
+selection module to produce long-term coherent and consistent gesture videos.
+For better visual perception, we further design a refinement network focusing
+on missing details of certain areas. Extensive experimental results show that
+our proposed framework significantly outperforms existing approaches in both
+motion and video-related evaluations. Our code, demos, and more resources are
+available at https://github.com/thuhcsi/S2G-MDDiffusion.",cs.CV,"['cs.CV', 'cs.HC', 'cs.MM']"
+Leveraging Camera Triplets for Efficient and Accurate Structure-from-Motion,Lalit Manam · Venu Madhav Govindu,https://ee.iisc.ac.in/cvlab/research/camtripsfm/,,,,,,,nan
+TextCraftor: Your Text Encoder Can be Image Quality Controller,Yanyu Li · Xian Liu · Anil Kag · Ju Hu · Yerlan Idelbayev · Dhritiman Sagar · Yanzhi Wang · Sergey Tulyakov · Jian Ren, ,https://arxiv.org/abs/2403.18978,,2403.18978.pdf,TextCraftor: Your Text Encoder Can be Image Quality Controller,"Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
+revolutionized the field of content generation, enabling significant
+advancements in areas like image editing and video synthesis. Despite their
+formidable capabilities, these models are not without their limitations. It is
+still challenging to synthesize an image that aligns well with the input text,
+and multiple runs with carefully crafted prompts are required to achieve
+satisfactory results. To mitigate these limitations, numerous studies have
+endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
+various technologies. Yet, amidst these efforts, a pivotal question of
+text-to-image diffusion model training has remained largely unexplored: Is it
+possible and feasible to fine-tune the text encoder to improve the performance
+of text-to-image diffusion models? Our findings reveal that, instead of
+replacing the CLIP text encoder used in Stable Diffusion with other large
+language models, we can enhance it through our proposed fine-tuning approach,
+TextCraftor, leading to substantial improvements in quantitative benchmarks and
+human assessments. Interestingly, our technique also empowers controllable
+image generation through the interpolation of different text encoders
+fine-tuned with various rewards. We also demonstrate that TextCraftor is
+orthogonal to UNet finetuning, and can be combined to further improve
+generative quality.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining,Jiahao Nie · Yun Xing · Gongjie Zhang · Pei Yan · Aoran Xiao · Yap-peng Tan · Alex C. Kot · Shijian Lu, ,https://arxiv.org/abs/2401.08407,,,Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining,"Cross-Domain Few-Shot Segmentation (CD-FSS) poses the challenge of segmenting
+novel categories from a distinct domain using only limited exemplars. In this
+paper, we undertake a comprehensive study of CD-FSS and uncover two crucial
+insights: (i) the necessity of a fine-tuning stage to effectively transfer the
+learned meta-knowledge across domains, and (ii) the overfitting risk during the
+na\""ive fine-tuning due to the scarcity of novel category examples. With these
+insights, we propose a novel cross-domain fine-tuning strategy that addresses
+the challenging CD-FSS tasks. We first design Bi-directional Few-shot
+Prediction (BFP), which establishes support-query correspondence in a
+bi-directional manner, crafting augmented supervision to reduce the overfitting
+risk. Then we further extend BFP into Iterative Few-shot Adaptor (IFA), which
+is a recursive framework to capture the support-query correspondence
+iteratively, targeting maximal exploitation of supervisory signals from the
+sparse novel category samples. Extensive empirical evaluations show that our
+method significantly outperforms the state-of-the-arts (+7.8\%), which verifies
+that IFA tackles the cross-domain challenges and mitigates the overfitting
+simultaneously. The code is available at: https://github.com/niejiahao1998/IFA.",cs.CV,['cs.CV']
+Learning Large-Factor EM Image Super-Resolution with Generative Priors,Jiateng Shou · Zeyu Xiao · Shiyu Deng · Wei Huang · ShiPeiyao · Ruobing Zhang · Zhiwei Xiong · Feng Wu,https://github.com/jtshou/GPEMSR,https://arxiv.org/html/2405.07044v1,,2405.07044v1.pdf,Semantic Guided Large Scale Factor Remote Sensing Image Super-resolution with Generative Diffusion Prior,"Remote sensing images captured by different platforms exhibit significant
+disparities in spatial resolution. Large scale factor super-resolution (SR)
+algorithms are vital for maximizing the utilization of low-resolution (LR)
+satellite data captured from orbit. However, existing methods confront
+challenges in recovering SR images with clear textures and correct ground
+objects. We introduce a novel framework, the Semantic Guided Diffusion Model
+(SGDM), designed for large scale factor remote sensing image super-resolution.
+The framework exploits a pre-trained generative model as a prior to generate
+perceptually plausible SR images. We further enhance the reconstruction by
+incorporating vector maps, which carry structural and semantic cues. Moreover,
+pixel-level inconsistencies in paired remote sensing images, stemming from
+sensor-specific imaging characteristics, may hinder the convergence of the
+model and diversity in generated results. To address this problem, we propose
+to extract the sensor-specific imaging characteristics and model the
+distribution of them, allowing diverse SR images generation based on imaging
+characteristics provided by reference images or sampled from the imaging
+characteristic probability distributions. To validate and evaluate our
+approach, we create the Cross-Modal Super-Resolution Dataset (CMSRD).
+Qualitative and quantitative experiments on CMSRD showcase the superiority and
+broad applicability of our method. Experimental results on downstream vision
+tasks also demonstrate the utilitarian of the generated SR images. The dataset
+and code will be publicly available at https://github.com/wwangcece/SGDM",cs.CV,['cs.CV']
+FairCLIP: Harnessing Fairness in Vision-Language Learning,Yan Luo · MIN SHI · Muhammad Osama Khan · Muhammad Muneeb Afzal · Hao Huang · Shuaihang Yuan · Yu Tian · Luo Song · Ava Kouhana · Tobias Elze · Yi Fang · Mengyu Wang, ,https://arxiv.org/abs/2403.19949,,2403.19949.pdf,FairCLIP: Harnessing Fairness in Vision-Language Learning,"Fairness is a critical concern in deep learning, especially in healthcare,
+where these models influence diagnoses and treatment decisions. Although
+fairness has been investigated in the vision-only domain, the fairness of
+medical vision-language (VL) models remains unexplored due to the scarcity of
+medical VL datasets for studying fairness. To bridge this research gap, we
+introduce the first fair vision-language medical dataset Harvard-FairVLMed that
+provides detailed demographic attributes, ground-truth labels, and clinical
+notes to facilitate an in-depth examination of fairness within VL foundation
+models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis
+of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and
+medical domains, across four different protected attributes. Our results
+highlight significant biases in all VL models, with Asian, Male, Non-Hispanic,
+and Spanish being the preferred subgroups across the protected attributes of
+race, gender, ethnicity, and language, respectively. In order to alleviate
+these biases, we propose FairCLIP, an optimal-transport-based approach that
+achieves a favorable trade-off between performance and fairness by reducing the
+Sinkhorn distance between the overall sample distribution and the distributions
+corresponding to each demographic group. As the first VL dataset of its kind,
+Harvard-FairVLMed holds the potential to catalyze advancements in the
+development of machine learning models that are both ethically aware and
+clinically effective. Our dataset and code are available at
+https://ophai.hms.harvard.edu/datasets/harvard-fairvlmed10k.",cs.CV,['cs.CV']
+Distributionally Generative Augmentation for Fair Facial Attribute Classification,Fengda Zhang · Qianpei He · Kun Kuang · Jiashuo Liu · Long Chen · Chao Wu · Jun Xiao · Hanwang Zhang,https://github.com/heqianpei/DiGA,https://arxiv.org/abs/2403.06606,,2403.06606.pdf,Distributionally Generative Augmentation for Fair Facial Attribute Classification,"Facial Attribute Classification (FAC) holds substantial promise in widespread
+applications. However, FAC models trained by traditional methodologies can be
+unfair by exhibiting accuracy inconsistencies across varied data
+subpopulations. This unfairness is largely attributed to bias in data, where
+some spurious attributes (e.g., Male) statistically correlate with the target
+attribute (e.g., Smiling). Most of existing fairness-aware methods rely on the
+labels of spurious attributes, which may be unavailable in practice. This work
+proposes a novel, generation-based two-stage framework to train a fair FAC
+model on biased data without additional annotation. Initially, we identify the
+potential spurious attributes based on generative models. Notably, it enhances
+interpretability by explicitly showing the spurious attributes in image space.
+Following this, for each image, we first edit the spurious attributes with a
+random degree sampled from a uniform distribution, while keeping target
+attribute unchanged. Then we train a fair FAC model by fostering model
+invariance to these augmentation. Extensive experiments on three common
+datasets demonstrate the effectiveness of our method in promoting fairness in
+FAC without compromising accuracy. Codes are in
+https://github.com/heqianpei/DiGA.",cs.CV,"['cs.CV', 'cs.LG']"
+RobustSAM: Segment Anything Robustly on Degraded Images,Wei-Ting Chen · Yu Jiet Vong · Sy-Yen Kuo · Sizhuo Ma · Jian Wang, ,https://arxiv.org/abs/2306.07713,,2306.07713.pdf,Robustness of SAM: Segment Anything Under Corruptions and Beyond,"Segment anything model (SAM), as the name suggests, is claimed to be capable
+of cutting out any object and demonstrates impressive zero-shot transfer
+performance with the guidance of prompts. However, there is currently a lack of
+comprehensive evaluation regarding its robustness under various corruptions.
+Understanding the robustness of SAM across different corruption scenarios is
+crucial for its real-world deployment. Prior works show that SAM is biased
+towards texture (style) rather than shape, motivated by which we start by
+investigating its robustness against style transfer, which is synthetic
+corruption. Following by interpreting the effects of synthetic corruption as
+style changes, we proceed to conduct a comprehensive evaluation for its
+robustness against 15 types of common corruption. These corruptions mainly fall
+into categories such as digital, noise, weather, and blur, and within each
+corruption category, we explore 5 severity levels to simulate real-world
+corruption scenarios. Beyond the corruptions, we further assess the robustness
+of SAM against local occlusion and local adversarial patch attacks. To the best
+of our knowledge, our work is the first of its kind to evaluate the robustness
+of SAM under style change, local occlusion, and local adversarial patch
+attacks. Given that patch attacks visible to human eyes are easily detectable,
+we further assess its robustness against global adversarial attacks that are
+imperceptible to human eyes. Overall, this work provides a comprehensive
+empirical study of the robustness of SAM, evaluating its performance under
+various corruptions and extending the assessment to critical aspects such as
+local occlusion, local adversarial patch attacks, and global adversarial
+attacks. These evaluations yield valuable insights into the practical
+applicability and effectiveness of SAM in addressing real-world challenges.",cs.CV,['cs.CV']
+ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation,Dar-Yen Chen · Hamish Tennent · Ching-Wen Hsu,https://cardinalblue.github.io/artadapter.github.io/,https://arxiv.org/abs/2312.02109v1,,2312.02109v1.pdf,ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation,"This work introduces ArtAdapter, a transformative text-to-image (T2I) style
+transfer framework that transcends traditional limitations of color,
+brushstrokes, and object shape, capturing high-level style elements such as
+composition and distinctive artistic expression. The integration of a
+multi-level style encoder with our proposed explicit adaptation mechanism
+enables ArtAdapte to achieve unprecedented fidelity in style transfer, ensuring
+close alignment with textual descriptions. Additionally, the incorporation of
+an Auxiliary Content Adapter (ACA) effectively separates content from style,
+alleviating the borrowing of content from style references. Moreover, our novel
+fast finetuning approach could further enhance zero-shot style representation
+while mitigating the risk of overfitting. Comprehensive evaluations confirm
+that ArtAdapter surpasses current state-of-the-art methods.",cs.CV,['cs.CV']
+NAPGuard: Towards Detecting Naturalistic Adversarial Patches,Siyang Wu · Jiakai Wang · Jiejie Zhao · Yazhe Wang · Xianglong Liu,https://github.com/wsynuiag/NAPGaurd,https://arxiv.org/abs/2307.08076,,2307.08076.pdf,Diffusion to Confusion: Naturalistic Adversarial Patch Generation Based on Diffusion Model for Object Detector,"Many physical adversarial patch generation methods are widely proposed to
+protect personal privacy from malicious monitoring using object detectors.
+However, they usually fail to generate satisfactory patch images in terms of
+both stealthiness and attack performance without making huge efforts on careful
+hyperparameter tuning. To address this issue, we propose a novel naturalistic
+adversarial patch generation method based on the diffusion models (DM). Through
+sampling the optimal image from the DM model pretrained upon natural images, it
+allows us to stably craft high-quality and naturalistic physical adversarial
+patches to humans without suffering from serious mode collapse problems as
+other deep generative models. To the best of our knowledge, we are the first to
+propose DM-based naturalistic adversarial patch generation for object
+detectors. With extensive quantitative, qualitative, and subjective
+experiments, the results demonstrate the effectiveness of the proposed approach
+to generate better-quality and more naturalistic adversarial patches while
+achieving acceptable attack performance than other state-of-the-art patch
+generation methods. We also show various generation trade-offs under different
+conditions.",cs.CV,['cs.CV']
+DSL-FIQA: Assessing Facial Image Quality via Dual-Set Degradation Learning and Landmark-Guided Transformer,Wei-Ting Chen · Gurunandan Krishnan · Qiang Gao · Sy-Yen Kuo · Sizhuo Ma · Jian Wang, ,,https://ieeexplore.ieee.org/abstract/document/10381809/authors,,,,,nan
+PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios,Jingbo Wang · Zhengyi Luo · Ye Yuan · Yixuan LI · Bo Dai, ,https://arxiv.org/html/2404.19722v1,,2404.19722v1.pdf,PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios,"We address the challenge of content diversity and controllability in
+pedestrian simulation for driving scenarios. Recent pedestrian animation
+frameworks have a significant limitation wherein they primarily focus on either
+following trajectory [46] or the content of the reference video [57],
+consequently overlooking the potential diversity of human motion within such
+scenarios. This limitation restricts the ability to generate pedestrian
+behaviors that exhibit a wider range of variations and realistic motions and
+therefore restricts its usage to provide rich motion content for other
+components in the driving simulation system, e.g., suddenly changed motion to
+which the autonomous vehicle should respond. In our approach, we strive to
+surpass the limitation by showcasing diverse human motions obtained from
+various sources, such as generated human motions, in addition to following the
+given trajectory. The fundamental contribution of our framework lies in
+combining the motion tracking task with trajectory following, which enables the
+tracking of specific motion parts (e.g., upper body) while simultaneously
+following the given trajectory by a single policy. This way, we significantly
+enhance both the diversity of simulated human motion within the given scenario
+and the controllability of the content, including language-based control. Our
+framework facilitates the generation of a wide range of human motions,
+contributing to greater realism and adaptability in pedestrian simulations for
+driving scenarios. More information is on our project page
+https://wangjingbo1219.github.io/papers/CVPR2024_PACER_PLUS/PACERPLUSPage.html .",cs.CV,['cs.CV']
+Cache Me if You Can: Accelerating Diffusion Models through Block Caching,Felix Wimbauer · Bichen Wu · Edgar Schoenfeld · Xiaoliang Dai · Ji Hou · Zijian He · Artsiom Sanakoyeu · Peizhao Zhang · Sam Tsai · Jonas Kohler · Christian Rupprecht · Daniel Cremers · Peter Vajda · Jialiang Wang, ,https://arxiv.org/abs/2312.03209,,2312.03209.pdf,Cache Me if You Can: Accelerating Diffusion Models through Block Caching,"Diffusion models have recently revolutionized the field of image synthesis
+due to their ability to generate photorealistic images. However, one of the
+major drawbacks of diffusion models is that the image generation process is
+costly. A large image-to-image network has to be applied many times to
+iteratively refine an image from random noise. While many recent works propose
+techniques to reduce the number of required steps, they generally treat the
+underlying denoising network as a black box. In this work, we investigate the
+behavior of the layers within the network and find that 1) the layers' output
+changes smoothly over time, 2) the layers show distinct patterns of change, and
+3) the change from step to step is often very small. We hypothesize that many
+layer computations in the denoising network are redundant. Leveraging this, we
+introduce block caching, in which we reuse outputs from layer blocks of
+previous steps to speed up inference. Furthermore, we propose a technique to
+automatically determine caching schedules based on each block's changes over
+timesteps. In our experiments, we show through FID, human evaluation and
+qualitative analysis that Block Caching allows to generate images with higher
+visual quality at the same computational cost. We demonstrate this for
+different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).",cs.CV,['cs.CV']
+Multi-Modal Hallucination Control by Visual Information Grounding,Alessandro Favero · Luca Zancato · Matthew Trager · Siddharth Choudhary · Pramuditha Perera · Alessandro Achille · Ashwin Swaminathan · Stefano Soatto, ,https://arxiv.org/abs/2403.14003,,2403.14003.pdf,Multi-Modal Hallucination Control by Visual Information Grounding,"Generative Vision-Language Models (VLMs) are prone to generate
+plausible-sounding textual answers that, however, are not always grounded in
+the input image. We investigate this phenomenon, usually referred to as
+""hallucination"" and show that it stems from an excessive reliance on the
+language prior. In particular, we show that as more tokens are generated, the
+reliance on the visual prompt decreases, and this behavior strongly correlates
+with the emergence of hallucinations. To reduce hallucinations, we introduce
+Multi-Modal Mutual-Information Decoding (M3ID), a new sampling method for
+prompt amplification. M3ID amplifies the influence of the reference image over
+the language prior, hence favoring the generation of tokens with higher mutual
+information with the visual prompt. M3ID can be applied to any pre-trained
+autoregressive VLM at inference time without necessitating further training and
+with minimal computational overhead. If training is an option, we show that
+M3ID can be paired with Direct Preference Optimization (DPO) to improve the
+model's reliance on the prompt image without requiring any labels. Our
+empirical findings show that our algorithms maintain the fluency and linguistic
+capabilities of pre-trained VLMs while reducing hallucinations by mitigating
+visually ungrounded answers. Specifically, for the LLaVA 13B model, M3ID and
+M3ID+DPO reduce the percentage of hallucinated objects in captioning tasks by
+25% and 28%, respectively, and improve the accuracy on VQA benchmarks such as
+POPE by 21% and 24%.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Diffusion Time-step Curriculum for One Image to 3D Generation,YI Xuanyu · Zike Wu · Qingshan Xu · Pan Zhou · Joo Lim · Hanwang Zhang, ,https://arxiv.org/abs/2404.04562,,2404.04562.pdf,Diffusion Time-step Curriculum for One Image to 3D Generation,"Score distillation sampling~(SDS) has been widely adopted to overcome the
+absence of unseen views in reconstructing 3D objects from a \textbf{single}
+image. It leverages pre-trained 2D diffusion models as teacher to guide the
+reconstruction of student 3D models. Despite their remarkable success,
+SDS-based methods often encounter geometric artifacts and texture saturation.
+We find out the crux is the overlooked indiscriminate treatment of diffusion
+time-steps during optimization: it unreasonably treats the student-teacher
+knowledge distillation to be equal at all time-steps and thus entangles
+coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion
+Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the
+teacher and student models collaborating with the time-step curriculum in a
+coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and
+Level50 benchmark demonstrate that DTC123 can produce multi-view consistent,
+high-quality, and diverse 3D assets. Codes and more generation demos will be
+released in https://github.com/yxymessi/DTC123.",cs.CV,['cs.CV']
+3DToonify: Creating Your High-Fidelity 3D Stylized Avatar Easily from 2D Portrait Images,Yifang Men · Hanxi Liu · Yuan Yao · Miaomiao Cui · Xuansong Xie · Zhouhui Lian, ,https://arxiv.org/abs/2311.17917,,2311.17917.pdf,AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text,"We study the problem of creating high-fidelity and animatable 3D avatars from
+only textual descriptions. Existing text-to-avatar methods are either limited
+to static avatars which cannot be animated or struggle to generate animatable
+avatars with promising quality and precise pose control. To address these
+limitations, we propose AvatarStudio, a coarse-to-fine generative model that
+generates explicit textured 3D meshes for animatable human avatars.
+Specifically, AvatarStudio begins with a low-resolution NeRF-based
+representation for coarse generation, followed by incorporating SMPL-guided
+articulation into the explicit mesh representation to support avatar animation
+and high resolution rendering. To ensure view consistency and pose
+controllability of the resulting avatars, we introduce a 2D diffusion model
+conditioned on DensePose for Score Distillation Sampling supervision. By
+effectively leveraging the synergy between the articulated mesh representation
+and the DensePose-conditional diffusion model, AvatarStudio can create
+high-quality avatars from text that are ready for animation, significantly
+outperforming previous methods. Moreover, it is competent for many
+applications, e.g., multimodal avatar animations and style-guided avatar
+creation. For more results, please refer to our project page:
+http://jeff95.me/projects/avatarstudio.html",cs.GR,"['cs.GR', 'cs.CV']"
+Unleashing Network Potentials for Semantic Scene Completion,Fengyun Wang · Qianru Sun · Dong Zhang · Jinhui Tang,https://github.com/fereenwong/AMMNet,https://arxiv.org/abs/2403.07560v2,,2403.07560v2.pdf,Unleashing Network Potentials for Semantic Scene Completion,"Semantic scene completion (SSC) aims to predict complete 3D voxel occupancy
+and semantics from a single-view RGB-D image, and recent SSC methods commonly
+adopt multi-modal inputs. However, our investigation reveals two limitations:
+ineffective feature learning from single modalities and overfitting to limited
+datasets. To address these issues, this paper proposes a novel SSC framework -
+Adversarial Modality Modulation Network (AMMNet) - with a fresh perspective of
+optimizing gradient updates. The proposed AMMNet introduces two core modules: a
+cross-modal modulation enabling the interdependence of gradient flows between
+modalities, and a customized adversarial training scheme leveraging dynamic
+gradient competition. Specifically, the cross-modal modulation adaptively
+re-calibrates the features to better excite representation potentials from each
+single modality. The adversarial training employs a minimax game of evolving
+gradients, with customized guidance to strengthen the generator's perception of
+visual fidelity from both geometric completeness and semantic correctness.
+Extensive experimental results demonstrate that AMMNet outperforms
+state-of-the-art SSC methods by a large margin, providing a promising direction
+for improving the effectiveness and generalization of SSC methods.",cs.CV,['cs.CV']
+NeRF Director: Revisiting View Selection in Neural Volume Rendering,Wenhui Xiao · Rodrigo Santa Cruz · David Ahmedt-Aristizabal · Olivier Salvado · Clinton Fookes · Leo Lebrat,https://wenwhx.github.io/nerfdirector/,https://arxiv.org/abs/2310.20685,,2310.20685.pdf,NeRF Revisited: Fixing Quadrature Instability in Volume Rendering,"Neural radiance fields (NeRF) rely on volume rendering to synthesize novel
+views. Volume rendering requires evaluating an integral along each ray, which
+is numerically approximated with a finite sum that corresponds to the exact
+integral along the ray under piecewise constant volume density. As a
+consequence, the rendered result is unstable w.r.t. the choice of samples along
+the ray, a phenomenon that we dub quadrature instability. We propose a
+mathematically principled solution by reformulating the sample-based rendering
+equation so that it corresponds to the exact integral under piecewise linear
+volume density. This simultaneously resolves multiple issues: conflicts between
+samples along different rays, imprecise hierarchical sampling, and
+non-differentiability of quantiles of ray termination distances w.r.t. model
+parameters. We demonstrate several benefits over the classical sample-based
+rendering equation, such as sharper textures, better geometric reconstruction,
+and stronger depth supervision. Our proposed formulation can be also be used as
+a drop-in replacement to the volume rendering equation of existing NeRF-based
+methods. Our project page can be found at pl-nerf.github.io.",cs.CV,['cs.CV']
+Exploring Efficient Asymmetric Blind-Spots for Self-Supervised Denoising in Real-World Scenarios,Shiyan Chen · Jiyuan Zhang · Zhaofei Yu · Tiejun Huang, ,https://ar5iv.labs.arxiv.org/html/2303.16783,,2303.16783.pdf,Exploring Efficient Asymmetric Blind-Spots for Self-Supervised Denoising in Real-World Scenarios,"Self-supervised denoising has attracted widespread attention due to its
+ability to train without clean images. However, noise in real-world scenarios
+is often spatially correlated, which causes many self-supervised algorithms
+that assume pixel-wise independent noise to perform poorly. Recent works have
+attempted to break noise correlation with downsampling or neighborhood masking.
+However, denoising on downsampled subgraphs can lead to aliasing effects and
+loss of details due to a lower sampling rate. Furthermore, the neighborhood
+masking methods either come with high computational complexity or do not
+consider local spatial preservation during inference. Through the analysis of
+existing methods, we point out that the key to obtaining high-quality and
+texture-rich results in real-world self-supervised denoising tasks is to train
+at the original input resolution structure and use asymmetric operations during
+training and inference. Based on this, we propose Asymmetric Tunable Blind-Spot
+Network (AT-BSN), where the blind-spot size can be freely adjusted, thus better
+balancing noise correlation suppression and image local spatial destruction
+during training and inference. In addition, we regard the pre-trained AT-BSN as
+a meta-teacher network capable of generating various teacher networks by
+sampling different blind-spots. We propose a blind-spot based multi-teacher
+distillation strategy to distill a lightweight network, significantly improving
+performance. Experimental results on multiple datasets prove that our method
+achieves state-of-the-art, and is superior to other self-supervised algorithms
+in terms of computational overhead and visual effects.",cs.CV,['cs.CV']
+Interpretable Measures of Conceptual Similarity by  Complexity-Constrained Descriptive Auto-Encoding,Alessandro Achille · Greg Ver Steeg · Tian Yu Liu · Matthew Trager · Carson Klingenberg · Stefano Soatto, ,https://arxiv.org/abs/2402.08919v1,,2402.08919v1.pdf,Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding,"Quantifying the degree of similarity between images is a key copyright issue
+for image-based machine learning. In legal doctrine however, determining the
+degree of similarity between works requires subjective analysis, and
+fact-finders (judges and juries) can demonstrate considerable variability in
+these subjective judgement calls. Images that are structurally similar can be
+deemed dissimilar, whereas images of completely different scenes can be deemed
+similar enough to support a claim of copying. We seek to define and compute a
+notion of ""conceptual similarity"" among images that captures high-level
+relations even among images that do not share repeated elements or visually
+similar components. The idea is to use a base multi-modal model to generate
+""explanations"" (captions) of visual data at increasing levels of complexity.
+Then, similarity can be measured by the length of the caption needed to
+discriminate between the two images: Two highly dissimilar images can be
+discriminated early in their description, whereas conceptually dissimilar ones
+will need more detail to be distinguished. We operationalize this definition
+and show that it correlates with subjective (averaged human evaluation)
+assessment, and beats existing baselines on both image-to-image and
+text-to-text similarity benchmarks. Beyond just providing a number, our method
+also offers interpretability by pointing to the specific level of granularity
+of the description where the source data are differentiated.",cs.CV,"['cs.CV', 'cs.LG']"
+Attack To Defend: Exploiting Adversarial Attacks for Detecting Poisoned Models,Samar Fares · Karthik Nandakumar, ,https://arxiv.org/abs/2312.06230,,2312.06230.pdf,Activation Gradient based Poisoned Sample Detection Against Backdoor Attacks,"This work studies the task of poisoned sample detection for defending against
+data poisoning based backdoor attacks. Its core challenge is finding a
+generalizable and discriminative metric to distinguish between clean and
+various types of poisoned samples (e.g., various triggers, various poisoning
+ratios). Inspired by a common phenomenon in backdoor attacks that the
+backdoored model tend to map significantly different poisoned and clean samples
+within the target class to similar activation areas, we introduce a novel
+perspective of the circular distribution of the gradients w.r.t. sample
+activation, dubbed gradient circular distribution (GCD). And, we find two
+interesting observations based on GCD. One is that the GCD of samples in the
+target class is much more dispersed than that in the clean class. The other is
+that in the GCD of target class, poisoned and clean samples are clearly
+separated. Inspired by above two observations, we develop an innovative
+three-stage poisoned sample detection approach, called Activation Gradient
+based Poisoned sample Detection (AGPD). First, we calculate GCDs of all classes
+from the model trained on the untrustworthy dataset. Then, we identify the
+target class(es) based on the difference on GCD dispersion between target and
+clean classes. Last, we filter out poisoned samples within the identified
+target class(es) based on the clear separation between poisoned and clean
+samples. Extensive experiments under various settings of backdoor attacks
+demonstrate the superior detection performance of the proposed method to
+existing poisoned detection approaches according to sample activation-based
+metrics.",cs.CR,['cs.CR']
+YOLO-World: Real-Time Open-Vocabulary Object Detection,Tianheng Cheng · Lin Song · Yixiao Ge · Wenyu Liu · Xinggang Wang · Ying Shan,https://github.com/AILab-CVC/YOLO-World,https://arxiv.org/abs/2401.17270,,2401.17270.pdf,YOLO-World: Real-Time Open-Vocabulary Object Detection,"The You Only Look Once (YOLO) series of detectors have established themselves
+as efficient and practical tools. However, their reliance on predefined and
+trained object categories limits their applicability in open scenarios.
+Addressing this limitation, we introduce YOLO-World, an innovative approach
+that enhances YOLO with open-vocabulary detection capabilities through
+vision-language modeling and pre-training on large-scale datasets.
+Specifically, we propose a new Re-parameterizable Vision-Language Path
+Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate
+the interaction between visual and linguistic information. Our method excels in
+detecting a wide range of objects in a zero-shot manner with high efficiency.
+On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on
+V100, which outperforms many state-of-the-art methods in terms of both accuracy
+and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable
+performance on several downstream tasks, including object detection and
+open-vocabulary instance segmentation.",cs.CV,['cs.CV']
+Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction,Devikalyan Das · Christopher Wewer · Raza Yunus · Eddy Ilg · Jan Lenssen,https://geometric-rl.mpi-inf.mpg.de/npg/,https://arxiv.org/abs/2312.01196,,2312.01196.pdf,Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction,"Reconstructing dynamic objects from monocular videos is a severely
+underconstrained and challenging problem, and recent work has approached it in
+various directions. However, owing to the ill-posed nature of this problem,
+there has been no solution that can provide consistent, high-quality novel
+views from camera positions that are significantly different from the training
+views. In this work, we introduce Neural Parametric Gaussians (NPGs) to take on
+this challenge by imposing a two-stage approach: first, we fit a low-rank
+neural deformation model, which then is used as regularization for non-rigid
+reconstruction in the second stage. The first stage learns the object's
+deformations such that it preserves consistency in novel views. The second
+stage obtains high reconstruction quality by optimizing 3D Gaussians that are
+driven by the coarse model. To this end, we introduce a local 3D Gaussian
+representation, where temporally shared Gaussians are anchored in and deformed
+by local oriented volumes. The resulting combined model can be rendered as
+radiance fields, resulting in high-quality photo-realistic reconstructions of
+the non-rigidly deforming objects. We demonstrate that NPGs achieve superior
+results compared to previous works, especially in challenging scenarios with
+few multi-view cues.",cs.CV,['cs.CV']
+AttriHuman-3D: Editable 3D Human Avatar Generation with Attribute Decomposition and Indexing,Fan Yang · Tianyi Chen · XIAOSHENG HE · Zhongang Cai · Lei Yang · Si Wu · Guosheng Lin, ,https://arxiv.org/abs/2312.02209,,2312.02209.pdf,AttriHuman-3D: Editable 3D Human Avatar Generation with Attribute Decomposition and Indexing,"Editable 3D-aware generation, which supports user-interacted editing, has
+witnessed rapid development recently. However, existing editable 3D GANs either
+fail to achieve high-accuracy local editing or suffer from huge computational
+costs. We propose AttriHuman-3D, an editable 3D human generation model, which
+address the aforementioned problems with attribute decomposition and indexing.
+The core idea of the proposed model is to generate all attributes (e.g. human
+body, hair, clothes and so on) in an overall attribute space with six feature
+planes, which are then decomposed and manipulated with different attribute
+indexes. To precisely extract features of different attributes from the
+generated feature planes, we propose a novel attribute indexing method as well
+as an orthogonal projection regularization to enhance the disentanglement. We
+also introduce a hyper-latent training strategy and an attribute-specific
+sampling strategy to avoid style entanglement and misleading punishment from
+the discriminator. Our method allows users to interactively edit selected
+attributes in the generated 3D human avatars while keeping others fixed. Both
+qualitative and quantitative experiments demonstrate that our model provides a
+strong disentanglement between different attributes, allows fine-grained image
+editing and generates high-quality 3D human avatars.",cs.CV,['cs.CV']
+GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting,Yiwen Chen · Zilong Chen · Chi Zhang · Feng Wang · Xiaofeng Yang · Yikai Wang · Zhongang Cai · Lei Yang · Huaping Liu · Guosheng Lin, ,https://arxiv.org/abs/2311.14521,,2311.14521.pdf,GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting,"3D editing plays a crucial role in many areas such as gaming and virtual
+reality. Traditional 3D editing methods, which rely on representations like
+meshes and point clouds, often fall short in realistically depicting complex
+scenes. On the other hand, methods based on implicit 3D representations, like
+Neural Radiance Field (NeRF), render complex scenes effectively but suffer from
+slow processing speeds and limited control over specific scene areas. In
+response to these challenges, our paper presents GaussianEditor, an innovative
+and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D
+representation. GaussianEditor enhances precision and control in editing
+through our proposed Gaussian semantic tracing, which traces the editing target
+throughout the training process. Additionally, we propose Hierarchical Gaussian
+splatting (HGS) to achieve stabilized and fine results under stochastic
+generative guidance from 2D diffusion models. We also develop editing
+strategies for efficient object removal and integration, a challenging task for
+existing methods. Our comprehensive experiments demonstrate GaussianEditor's
+superior control, efficacy, and rapid performance, marking a significant
+advancement in 3D editing. Project Page:
+https://buaacyw.github.io/gaussian-editor/",cs.CV,['cs.CV']
+AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation,Qingping SUN · Yanjun Wang · Ailing Zeng · Wanqi Yin · Chen Wei · Wenjia Wang · Haiy Mei · Chi LEUNG · Ziwei Liu · Lei Yang · Zhongang Cai, ,https://arxiv.org/abs/2403.17934,,2403.17934.pdf,AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation,"Expressive human pose and shape estimation (a.k.a. 3D whole-body mesh
+recovery) involves the human body, hand, and expression estimation. Most
+existing methods have tackled this task in a two-stage manner, first detecting
+the human body part with an off-the-shelf detection model and inferring the
+different human body parts individually. Despite the impressive results
+achieved, these methods suffer from 1) loss of valuable contextual information
+via cropping, 2) introducing distractions, and 3) lacking inter-association
+among different persons and body parts, inevitably causing performance
+degradation, especially for crowded scenes. To address these issues, we
+introduce a novel all-in-one-stage framework, AiOS, for multiple expressive
+human pose and shape recovery without an additional human detection step.
+Specifically, our method is built upon DETR, which treats multi-person
+whole-body mesh recovery task as a progressive set prediction problem with
+various sequential detection. We devise the decoder tokens and extend them to
+our task. Specifically, we first employ a human token to probe a human location
+in the image and encode global features for each instance, which provides a
+coarse location for the later transformer block. Then, we introduce a
+joint-related token to probe the human joint in the image and encoder a
+fine-grained local feature, which collaborates with the global feature to
+regress the whole-body mesh. This straightforward but effective model
+outperforms previous state-of-the-art methods by a 9% reduction in NMVE on
+AGORA, a 30% reduction in PVE on EHF, a 10% reduction in PVE on ARCTIC, and a
+3% reduction in PVE on EgoBody.",cs.CV,['cs.CV']
+Edge-Aware 3D Instance Segmentation Network with Intelligent Semantic Prior,Wonseok Roh · Hwanhee Jung · Giljoo Nam · Jinseop Yeom · Hyunje Park · Sang Ho Yoon · Sangpil Kim, ,https://arxiv.org/abs/2311.12291,,2311.12291.pdf,Instance-aware 3D Semantic Segmentation powered by Shape Generators and Classifiers,"Existing 3D semantic segmentation methods rely on point-wise or voxel-wise
+feature descriptors to output segmentation predictions. However, these
+descriptors are often supervised at point or voxel level, leading to
+segmentation models that can behave poorly at instance-level. In this paper, we
+proposed a novel instance-aware approach for 3D semantic segmentation. Our
+method combines several geometry processing tasks supervised at instance-level
+to promote the consistency of the learned feature representation. Specifically,
+our methods use shape generators and shape classifiers to perform shape
+reconstruction and classification tasks for each shape instance. This enforces
+the feature representation to faithfully encode both structural and local shape
+information, with an awareness of shape instances. In the experiments, our
+method significantly outperform existing approaches in 3D semantic segmentation
+on several public benchmarks, such as Waymo Open Dataset, SemanticKITTI and
+ScanNetV2.",cs.CV,['cs.CV']
+ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation,Suraj Patni · Aradhye Agarwal · Chetan Arora,https://ecodepth-iitd.github.io/,https://arxiv.org/abs/2403.18807,,2403.18807.pdf,ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation,"In the absence of parallax cues, a learning-based single image depth
+estimation (SIDE) model relies heavily on shading and contextual cues in the
+image. While this simplicity is attractive, it is necessary to train such
+models on large and varied datasets, which are difficult to capture. It has
+been shown that using embeddings from pre-trained foundational models, such as
+CLIP, improves zero shot transfer in several applications. Taking inspiration
+from this, in our paper we explore the use of global image priors generated
+from a pre-trained ViT model to provide more detailed contextual information.
+We argue that the embedding vector from a ViT model, pre-trained on a large
+dataset, captures greater relevant information for SIDE than the usual route of
+generating pseudo image captions, followed by CLIP based text embeddings. Based
+on this idea, we propose a new SIDE model using a diffusion backbone which is
+conditioned on ViT embeddings. Our proposed design establishes a new
+state-of-the-art (SOTA) for SIDE on NYUv2 dataset, achieving Abs Rel error of
+0.059 (14% improvement) compared to 0.069 by the current SOTA (VPD). And on
+KITTI dataset, achieving Sq Rel error of 0.139 (2% improvement) compared to
+0.142 by the current SOTA (GEDepth). For zero-shot transfer with a model
+trained on NYUv2, we report mean relative improvement of (20%, 23%, 81%, 25%)
+over NeWCRFs on (Sun-RGBD, iBims1, DIODE, HyperSim) datasets, compared to (16%,
+18%, 45%, 9%) by ZoeDepth. The project page is available at
+https://ecodepth-iitd.github.io",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs,Lin Song · Yukang Chen · Shuai Yang · Xiaohan Ding · Yixiao Ge · Ying-Cong Chen · Ying Shan, ,https://arxiv.org/abs/2405.18572,,2405.18572.pdf,Low-rank finetuning for LLMs: A fairness perspective,"Low-rank approximation techniques have become the de facto standard for
+fine-tuning Large Language Models (LLMs) due to their reduced computational and
+memory requirements. This paper investigates the effectiveness of these methods
+in capturing the shift of fine-tuning datasets from the initial pre-trained
+data distribution. Our findings reveal that there are cases in which low-rank
+fine-tuning falls short in learning such shifts. This, in turn, produces
+non-negligible side effects, especially when fine-tuning is adopted for
+toxicity mitigation in pre-trained models, or in scenarios where it is
+important to provide fair models. Through comprehensive empirical evidence on
+several models, datasets, and tasks, we show that low-rank fine-tuning
+inadvertently preserves undesirable biases and toxic behaviors. We also show
+that this extends to sequential decision-making tasks, emphasizing the need for
+careful evaluation to promote responsible LLMs development.",cs.LG,"['cs.LG', 'cs.AI', 'cs.CL']"
+CG-HOI: Contact-Guided 3D Human-Object Interaction Generation,Christian Diller · Angela Dai,https://cg-hoi.christian-diller.de/#main,https://arxiv.org/abs/2311.16097v2,,2311.16097v2.pdf,CG-HOI: Contact-Guided 3D Human-Object Interaction Generation,"We propose CG-HOI, the first method to address the task of generating dynamic
+3D human-object interactions (HOIs) from text. We model the motion of both
+human and object in an interdependent fashion, as semantically rich human
+motion rarely happens in isolation without any interactions. Our key insight is
+that explicitly modeling contact between the human body surface and object
+geometry can be used as strong proxy guidance, both during training and
+inference. Using this guidance to bridge human and object motion enables
+generating more realistic and physically plausible interaction sequences, where
+the human body and corresponding object move in a coherent manner. Our method
+first learns to model human motion, object motion, and contact in a joint
+diffusion process, inter-correlated through cross-attention. We then leverage
+this learned contact for guidance during inference to synthesize realistic and
+coherent HOIs. Extensive evaluation shows that our joint contact-based
+human-object interaction approach generates realistic and physically plausible
+sequences, and we show two applications highlighting the capabilities of our
+method. Conditioned on a given object trajectory, we can generate the
+corresponding human motion without re-training, demonstrating strong
+human-object interdependency learning. Our approach is also flexible, and can
+be applied to static real-world 3D scene scans.",cs.CV,"['cs.CV', 'I.2.10; I.4.8; I.5.1; I.5.4']"
+Digital Life Project: Autonomous 3D Characters with Social Intelligence,Zhongang Cai · Jianping Jiang · Zhongfei Qing · Xinying Guo · Mingyuan Zhang · Zhengyu Lin · Haiy Mei · Chen Wei · Wang Ruisi · Wanqi Yin · Liang Pan · Xiangyu Fan · Han Du · Peng Gao · Zhitao Yang · Yang Gao · Jiaqi Li · Tianxiang Ren · YuKun Wei · Xiaogang Wang · Chen Change Loy · Lei Yang · Ziwei Liu,https://digital-life-project.com/,https://arxiv.org/abs/2312.04547,,2312.04547.pdf,Digital Life Project: Autonomous 3D Characters with Social Intelligence,"In this work, we present Digital Life Project, a framework utilizing language
+as the universal medium to build autonomous 3D characters, who are capable of
+engaging in social interactions and expressing with articulated body motions,
+thereby simulating life in a digital environment. Our framework comprises two
+primary components: 1) SocioMind: a meticulously crafted digital brain that
+models personalities with systematic few-shot exemplars, incorporates a
+reflection process based on psychology principles, and emulates autonomy by
+initiating dialogue topics; 2) MoMat-MoGen: a text-driven motion synthesis
+paradigm for controlling the character's digital body. It integrates motion
+matching, a proven industry technique to ensure motion quality, with
+cutting-edge advancements in motion generation for diversity. Extensive
+experiments demonstrate that each module achieves state-of-the-art performance
+in its respective domain. Collectively, they enable virtual characters to
+initiate and sustain dialogues autonomously, while evolving their
+socio-psychological states. Concurrently, these characters can perform
+contextually relevant bodily movements. Additionally, a motion captioning
+module further allows the virtual character to recognize and appropriately
+respond to human players' actions. Homepage: https://digital-life-project.com/",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.HC']"
+From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding,Yonglu Li · Xiaoqian Wu · Xinpeng Liu · Zehao Wang · Yiming Dou · Yikun Ji · Junyi Zhang · Yixing Li · Xudong LU · Jingru Tan · Cewu Lu, ,,https://synthical.com/article/a412be8a-adaa-450f-81ea-957ce0f2d0e4,,,,,nan
+FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations,Christian Diller · Thomas Funkhouser · Angela Dai,https://future-human-3d.christian-diller.de/#main,https://arxiv.org/abs/2312.11972,,,Expressive Forecasting of 3D Whole-body Human Motions,"Human motion forecasting, with the goal of estimating future human behavior
+over a period of time, is a fundamental task in many real-world applications.
+However, existing works typically concentrate on predicting the major joints of
+the human body without considering the delicate movements of the human hands.
+In practical applications, hand gesture plays an important role in human
+communication with the real world, and expresses the primary intention of human
+beings. In this work, we are the first to formulate a whole-body human pose
+forecasting task, which jointly predicts the future body and hand activities.
+Correspondingly, we propose a novel Encoding-Alignment-Interaction (EAI)
+framework that aims to predict both coarse (body joints) and fine-grained
+(gestures) activities collaboratively, enabling expressive and
+cross-facilitated forecasting of 3D whole-body human motions. Specifically, our
+model involves two key constituents: cross-context alignment (XCA) and
+cross-context interaction (XCI). Considering the heterogeneous information
+within the whole-body, XCA aims to align the latent features of various human
+components, while XCI focuses on effectively capturing the context interaction
+among the human components. We conduct extensive experiments on a
+newly-introduced large-scale benchmark and achieve state-of-the-art
+performance. The code is public for research purposes at
+https://github.com/Dingpx/EAI.",cs.CV,['cs.CV']
+"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition",Xiaohan Ding · Yiyuan Zhang · Yixiao Ge · Sijie Zhao · Lin Song · Xiangyu Yue · Ying Shan, ,https://arxiv.org/abs/2311.15599,,2311.15599.pdf,"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","Large-kernel convolutional neural networks (ConvNets) have recently received
+extensive research attention, but two unresolved and critical issues demand
+further investigation. 1) The architectures of existing large-kernel ConvNets
+largely follow the design principles of conventional ConvNets or transformers,
+while the architectural design for large-kernel ConvNets remains
+under-addressed. 2) As transformers have dominated multiple modalities, it
+remains to be investigated whether ConvNets also have a strong universal
+perception ability in domains beyond vision. In this paper, we contribute from
+two aspects. 1) We propose four architectural guidelines for designing
+large-kernel ConvNets, the core of which is to exploit the essential
+characteristics of large kernels that distinguish them from small kernels -
+they can see wide without going deep. Following such guidelines, our proposed
+large-kernel ConvNet shows leading performance in image recognition (ImageNet
+accuracy of 88.0%, ADE20K mIoU of 55.6%, and COCO box AP of 56.4%),
+demonstrating better performance and higher speed than the recent powerful
+competitors. 2) We discover large kernels are the key to unlocking the
+exceptional performance of ConvNets in domains where they were originally not
+proficient. With certain modality-related preprocessing approaches, the
+proposed model achieves state-of-the-art performance on time-series forecasting
+and audio recognition tasks even without modality-specific customization to the
+architecture. All the code and models are publicly available on GitHub and
+Huggingface.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning,Yuxiang Zhang · Hongwen Zhang · Liangxiao Hu · Jiajun Zhang · Hongwei Yi · Shengping Zhang · Yebin Liu,https://zhangyux15.github.io/ProxyCapV2,https://arxiv.org/abs/2307.01200,,2307.01200.pdf,ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning,"Learning-based approaches to monocular motion capture have recently shown
+promising results by learning to regress in a data-driven manner. However, due
+to the challenges in data collection and network designs, it remains
+challenging for existing solutions to achieve real-time full-body capture while
+being accurate in world space. In this work, we introduce ProxyCap, a
+human-centric proxy-to-motion learning scheme to learn world-space motions from
+a proxy dataset of 2D skeleton sequences and 3D rotational motions. Such proxy
+data enables us to build a learning-based network with accurate world-space
+supervision while also mitigating the generalization issues. For more accurate
+and physically plausible predictions in world space, our network is designed to
+learn human motions from a human-centric perspective, which enables the
+understanding of the same motion captured with different camera trajectories.
+Moreover, a contact-aware neural motion descent module is proposed in our
+network so that it can be aware of foot-ground contact and motion misalignment
+with the proxy observations. With the proposed learning-based solution, we
+demonstrate the first real-time monocular full-body capture system with
+plausible foot-ground contact in world space even using hand-held moving
+cameras. Our project page is https://zhangyux15.github.io/ProxyCapV2.",cs.CV,['cs.CV']
+DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation,Chenyang Wang · Zerong Zheng · Tao Yu · Xiaoqian Lv · Bineng Zhong · Shengping Zhang · Liqiang Nie, ,https://arxiv.org/abs/2312.00853,,2312.00853.pdf,Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution,"Real-world low-resolution (LR) videos have diverse and complex degradations,
+imposing great challenges on video super-resolution (VSR) algorithms to
+reproduce their high-resolution (HR) counterparts with high quality. Recently,
+the diffusion models have shown compelling performance in generating realistic
+details for image restoration tasks. However, the diffusion process has
+randomness, making it hard to control the contents of restored images. This
+issue becomes more serious when applying diffusion models to VSR tasks because
+temporal consistency is crucial to the perceptual quality of videos. In this
+paper, we propose an effective real-world VSR algorithm by leveraging the
+strength of pre-trained latent diffusion models. To ensure the content
+consistency among adjacent frames, we exploit the temporal dynamics in LR
+videos to guide the diffusion process by optimizing the latent sampling path
+with a motion-guided loss, ensuring that the generated HR video maintains a
+coherent and continuous visual flow. To further mitigate the discontinuity of
+generated details, we insert temporal module to the decoder and fine-tune it
+with an innovative sequence-oriented loss. The proposed motion-guided latent
+diffusion (MGLD) based VSR algorithm achieves significantly better perceptual
+quality than state-of-the-arts on real-world VSR benchmark datasets, validating
+the effectiveness of the proposed model design and training strategies.",cs.CV,['cs.CV']
+DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis,Yuming Gu · Hongyi Xu · You Xie · Guoxian Song · Yichun Shi · Di Chang · Jing Yang · Linjie Luo,https://freedomgu.github.io/DiffPortrait3D/,https://arxiv.org/abs/2312.13016,,2312.13016.pdf,DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis,"We present DiffPortrait3D, a conditional diffusion model that is capable of
+synthesizing 3D-consistent photo-realistic novel views from as few as a single
+in-the-wild portrait. Specifically, given a single RGB input, we aim to
+synthesize plausible but consistent facial details rendered from novel camera
+views with retained both identity and facial expression. In lieu of
+time-consuming optimization and fine-tuning, our zero-shot method generalizes
+well to arbitrary face portraits with unposed camera views, extreme facial
+expressions, and diverse artistic depictions. At its core, we leverage the
+generative prior of 2D diffusion models pre-trained on large-scale image
+datasets as our rendering backbone, while the denoising is guided with
+disentangled attentive control of appearance and camera pose. To achieve this,
+we first inject the appearance context from the reference image into the
+self-attention layers of the frozen UNets. The rendering view is then
+manipulated with a novel conditional control module that interprets the camera
+pose by watching a condition image of a crossed subject from the same view.
+Furthermore, we insert a trainable cross-view attention module to enhance view
+consistency, which is further strengthened with a novel 3D-aware noise
+generation process during inference. We demonstrate state-of-the-art results
+both qualitatively and quantitatively on our challenging in-the-wild and
+multi-view benchmarks.",cs.CV,['cs.CV']
+Tyche: Stochastic in Context Learning for Medical Image Segmentation,Marianne Rakic · Hallee Wong · Jose Javier Gonzalez Ortiz · Beth Cimini · John Guttag · Adrian V. Dalca, ,https://arxiv.org/abs/2401.13650,,2401.13650.pdf,Tyche: Stochastic In-Context Learning for Medical Image Segmentation,"Existing learning-based solutions to medical image segmentation have two
+important shortcomings. First, for most new segmentation task, a new model has
+to be trained or fine-tuned. This requires extensive resources and machine
+learning expertise, and is therefore often infeasible for medical researchers
+and clinicians. Second, most existing segmentation methods produce a single
+deterministic segmentation mask for a given image. In practice however, there
+is often considerable uncertainty about what constitutes the correct
+segmentation, and different expert annotators will often segment the same image
+differently. We tackle both of these problems with Tyche, a model that uses a
+context set to generate stochastic predictions for previously unseen tasks
+without the need to retrain. Tyche differs from other in-context segmentation
+methods in two important ways. (1) We introduce a novel convolution block
+architecture that enables interactions among predictions. (2) We introduce
+in-context test-time augmentation, a new mechanism to provide prediction
+stochasticity. When combined with appropriate model design and loss functions,
+Tyche can predict a set of plausible diverse segmentation candidates for new or
+unseen medical images and segmentation tasks without the need to retrain.",eess.IV,"['eess.IV', 'cs.CV']"
+Incremental Residual Concept Bottleneck Models,Chenming Shang · Shiji Zhou · Hengyuan Zhang · Xinzhe Ni · Yujiu Yang · Yuwang Wang, ,https://arxiv.org/abs/2404.08978,,2404.08978.pdf,Incremental Residual Concept Bottleneck Models,"Concept Bottleneck Models (CBMs) map the black-box visual representations
+extracted by deep neural networks onto a set of interpretable concepts and use
+the concepts to make predictions, enhancing the transparency of the
+decision-making process. Multimodal pre-trained models can match visual
+representations with textual concept embeddings, allowing for obtaining the
+interpretable concept bottleneck without the expertise concept annotations.
+Recent research has focused on the concept bank establishment and the
+high-quality concept selection. However, it is challenging to construct a
+comprehensive concept bank through humans or large language models, which
+severely limits the performance of CBMs. In this work, we propose the
+Incremental Residual Concept Bottleneck Model (Res-CBM) to address the
+challenge of concept completeness. Specifically, the residual concept
+bottleneck model employs a set of optimizable vectors to complete missing
+concepts, then the incremental concept discovery module converts the
+complemented vectors with unclear meanings into potential concepts in the
+candidate concept bank. Our approach can be applied to any user-defined concept
+bank, as a post-hoc processing method to enhance the performance of any CBMs.
+Furthermore, to measure the descriptive efficiency of CBMs, the Concept
+Utilization Efficiency (CUE) metric is proposed. Experiments show that the
+Res-CBM outperforms the current state-of-the-art methods in terms of both
+accuracy and efficiency and achieves comparable performance to black-box models
+across multiple datasets.",cs.LG,"['cs.LG', 'cs.AI']"
+RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features,Geonho Bang · Kwangjin Choi · Jisong Kim · Dongsuk Kum · Jun Won Choi, ,https://arxiv.org/abs/2403.05061,,2403.05061.pdf,RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features,"The inherent noisy and sparse characteristics of radar data pose challenges
+in finding effective representations for 3D object detection. In this paper, we
+propose RadarDistill, a novel knowledge distillation (KD) method, which can
+improve the representation of radar data by leveraging LiDAR data. RadarDistill
+successfully transfers desirable characteristics of LiDAR features into radar
+features using three key components: Cross-Modality Alignment (CMA),
+Activation-based Feature Distillation (AFD), and Proposal-based Feature
+Distillation (PFD). CMA enhances the density of radar features by employing
+multiple layers of dilation operations, effectively addressing the challenge of
+inefficient knowledge transfer from LiDAR to radar. AFD selectively transfers
+knowledge based on regions of the LiDAR features, with a specific focus on
+areas where activation intensity exceeds a predefined threshold. PFD similarly
+guides the radar network to selectively mimic features from the LiDAR network
+within the object proposals. Our comparative analyses conducted on the nuScenes
+datasets demonstrate that RadarDistill achieves state-of-the-art (SOTA)
+performance for radar-only object detection task, recording 20.5% in mAP and
+43.7% in NDS. Also, RadarDistill significantly improves the performance of the
+camera-radar fusion model.",cs.CV,['cs.CV']
+Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes,Gaurav Shrivastava · Abhinav Shrivastava,https://www.cs.umd.edu/~gauravsh/cvp/supp/website.html,https://arxiv.org/abs/2401.14718,,2401.14718.pdf,A Survey on Video Prediction: From Deterministic to Generative Approaches,"Video prediction, a fundamental task in computer vision, aims to enable
+models to generate sequences of future frames based on existing video content.
+This task has garnered widespread application across various domains. In this
+paper, we comprehensively survey both historical and contemporary works in this
+field, encompassing the most widely used datasets and algorithms. Our survey
+scrutinizes the challenges and evolving landscape of video prediction within
+the realm of computer vision. We propose a novel taxonomy centered on the
+stochastic nature of video prediction algorithms. This taxonomy accentuates the
+gradual transition from deterministic to generative prediction methodologies,
+underlining significant advancements and shifts in approach.",cs.CV,['cs.CV']
+Efficient Multi-scale Network with Learnable Discrete Wavelet Transform for Blind Motion Deblurring,Xin Gao · Tianheng Qiu · Xinyu Zhang · Hanlin Bai · Kang Liu · xuan huang · Hu Wei · Guoying Zhang · Huaping Liu, ,https://arxiv.org/abs/2401.00027,,2401.00027.pdf,Efficient Multi-scale Network with Learnable Discrete Wavelet Transform for Blind Motion Deblurring,"Coarse-to-fine schemes are widely used in traditional single-image motion
+deblur; however, in the context of deep learning, existing multi-scale
+algorithms not only require the use of complex modules for feature fusion of
+low-scale RGB images and deep semantics, but also manually generate
+low-resolution pairs of images that do not have sufficient confidence. In this
+work, we propose a multi-scale network based on single-input and
+multiple-outputs(SIMO) for motion deblurring. This simplifies the complexity of
+algorithms based on a coarse-to-fine scheme. To alleviate restoration defects
+impacting detail information brought about by using a multi-scale architecture,
+we combine the characteristics of real-world blurring trajectories with a
+learnable wavelet transform module to focus on the directional continuity and
+frequency features of the step-by-step transitions between blurred images to
+sharp images. In conclusion, we propose a multi-scale network with a learnable
+discrete wavelet transform (MLWNet), which exhibits state-of-the-art
+performance on multiple real-world deblurred datasets, in terms of both
+subjective and objective quality as well as computational efficiency.",cs.CV,['cs.CV']
+Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners,Keon Hee Park · Kyungwoo Song · Gyeong-Moon Park, ,https://arxiv.org/abs/2404.02117,,,Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners,"Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model
+to learn new classes incrementally without forgetting when only a few samples
+for each class are given. FSCIL encounters two significant challenges:
+catastrophic forgetting and overfitting, and these challenges have driven prior
+studies to primarily rely on shallow models, such as ResNet-18. Even though
+their limited capacity can mitigate both forgetting and overfitting issues, it
+leads to inadequate knowledge transfer during few-shot incremental sessions. In
+this paper, we argue that large models such as vision and language transformers
+pre-trained on large datasets can be excellent few-shot incremental learners.
+To this end, we propose a novel FSCIL framework called PriViLege, Pre-trained
+Vision and Language transformers with prompting functions and knowledge
+distillation. Our framework effectively addresses the challenges of
+catastrophic forgetting and overfitting in large models through new pre-trained
+knowledge tuning (PKT) and two losses: entropy-based divergence loss and
+semantic knowledge distillation loss. Experimental results show that the
+proposed PriViLege significantly outperforms the existing state-of-the-art
+methods with a large margin, e.g., +9.38% in CUB200, +20.58% in CIFAR-100, and
++13.36% in miniImageNet. Our implementation code is available at
+https://github.com/KHU-AGI/PriViLege.",cs.CV,['cs.CV']
+PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation,Jinfeng Xu · Siyuan Yang · Xianzhi Li · Yuan Tang · yixue Hao · Long Hu · Min Chen, ,https://arxiv.org/abs/2404.00979,,2404.00979.pdf,PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation,"Existing point cloud semantic segmentation networks cannot identify unknown
+classes and update their knowledge, due to a closed-set and static perspective
+of the real world, which would induce the intelligent agent to make bad
+decisions. To address this problem, we propose a Probability-Driven Framework
+(PDF) for open world semantic segmentation that includes (i) a lightweight
+U-decoder branch to identify unknown classes by estimating the uncertainties,
+(ii) a flexible pseudo-labeling scheme to supply geometry features along with
+probability distribution features of unknown classes by generating pseudo
+labels, and (iii) an incremental knowledge distillation strategy to incorporate
+novel classes into the existing knowledge base gradually. Our framework enables
+the model to behave like human beings, which could recognize unknown objects
+and incrementally learn them with the corresponding knowledge. Experimental
+results on the S3DIS and ScanNetv2 datasets demonstrate that the proposed PDF
+outperforms other methods by a large margin in both important tasks of open
+world semantic segmentation.",cs.CV,['cs.CV']
+Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle,Youtian Lin · Zuozhuo Dai · Siyu Zhu · Yao Yao, ,https://arxiv.org/abs/2312.03431,,2312.03431.pdf,Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle,"We introduce Gaussian-Flow, a novel point-based approach for fast dynamic
+scene reconstruction and real-time rendering from both multi-view and monocular
+videos. In contrast to the prevalent NeRF-based approaches hampered by slow
+training and rendering speeds, our approach harnesses recent advancements in
+point-based 3D Gaussian Splatting (3DGS). Specifically, a novel Dual-Domain
+Deformation Model (DDDM) is proposed to explicitly model attribute deformations
+of each Gaussian point, where the time-dependent residual of each attribute is
+captured by a polynomial fitting in the time domain, and a Fourier series
+fitting in the frequency domain. The proposed DDDM is capable of modeling
+complex scene deformations across long video footage, eliminating the need for
+training separate 3DGS for each frame or introducing an additional implicit
+neural field to model 3D dynamics. Moreover, the explicit deformation modeling
+for discretized Gaussian points ensures ultra-fast training and rendering of a
+4D scene, which is comparable to the original 3DGS designed for static 3D
+reconstruction. Our proposed approach showcases a substantial efficiency
+improvement, achieving a $5\times$ faster training speed compared to the
+per-frame 3DGS modeling. In addition, quantitative results demonstrate that the
+proposed Gaussian-Flow significantly outperforms previous leading methods in
+novel view rendering quality. Project page:
+https://nju-3dv.github.io/projects/Gaussian-Flow",cs.CV,['cs.CV']
+Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation,Zhipeng Du · Miaojing Shi · Jiankang Deng,https://github.com/ZPDu/Boosting-Object-Detection-with-Zero-Shot-Day-Night-Domain-Adaptation,https://arxiv.org/abs/2312.01220,,2312.01220.pdf,Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation,"Detecting objects in low-light scenarios presents a persistent challenge, as
+detectors trained on well-lit data exhibit significant performance degradation
+on low-light data due to low visibility. Previous methods mitigate this issue
+by exploring image enhancement or object detection techniques with real
+low-light image datasets. However, the progress is impeded by the inherent
+difficulties about collecting and annotating low-light images. To address this
+challenge, we propose to boost low-light object detection with zero-shot
+day-night domain adaptation, which aims to generalize a detector from well-lit
+scenarios to low-light ones without requiring real low-light data. Revisiting
+Retinex theory in the low-level vision, we first design a reflectance
+representation learning module to learn Retinex-based illumination invariance
+in images with a carefully designed illumination invariance reinforcement
+strategy. Next, an interchange-redecomposition-coherence procedure is
+introduced to improve over the vanilla Retinex image decomposition process by
+performing two sequential image decompositions and introducing a
+redecomposition cohering loss. Extensive experiments on ExDark, DARK FACE, and
+CODaN datasets show strong low-light generalizability of our method. Our code
+is available at https://github.com/ZPDu/DAI-Net.",cs.CV,['cs.CV']
+Clockwork Diffusion: Efficient Generation With Model-Step Distillation,Amirhossein Habibian · Amir Ghodrati · Noor Fathima · Guillaume Sautiere · Risheek Garrepalli · Fatih Porikli · Jens Petersen, ,https://arxiv.org/abs/2312.08128,,2312.08128.pdf,Clockwork Diffusion: Efficient Generation With Model-Step Distillation,"This work aims to improve the efficiency of text-to-image diffusion models.
+While diffusion models use computationally expensive UNet-based denoising
+operations in every generation step, we identify that not all operations are
+equally relevant for the final output quality. In particular, we observe that
+UNet layers operating on high-res feature maps are relatively sensitive to
+small perturbations. In contrast, low-res feature maps influence the semantic
+layout of the final image and can often be perturbed with no noticeable change
+in the output. Based on this observation, we propose Clockwork Diffusion, a
+method that periodically reuses computation from preceding denoising steps to
+approximate low-res feature maps at one or more subsequent steps. For multiple
+baselines, and for both text-to-image generation and image editing, we
+demonstrate that Clockwork leads to comparable or improved perceptual scores
+with drastically reduced computational complexity. As an example, for Stable
+Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
+CLIP change.",cs.CV,['cs.CV']
+BEVSpread: Spread Voxel Pooling for Bird’s-Eye-View Representation in Vision-based Roadside 3D Object Detection,Wenjie Wang · Yehao Lu · Guangcong Zheng · Shuigenzhan · Xiaoqing Ye · Zichang Tan · Jingdong Wang · Gaoang Wang · Xi Li,https://github.com/DaTongjie/BEVSpread,https://arxiv.org/abs/2312.00633,,2312.00633.pdf,Towards Efficient 3D Object Detection in Bird's-Eye-View Space for Autonomous Driving: A Convolutional-Only Approach,"3D object detection in Bird's-Eye-View (BEV) space has recently emerged as a
+prevalent approach in the field of autonomous driving. Despite the demonstrated
+improvements in accuracy and velocity estimation compared to perspective view
+methods, the deployment of BEV-based techniques in real-world autonomous
+vehicles remains challenging. This is primarily due to their reliance on
+vision-transformer (ViT) based architectures, which introduce quadratic
+complexity with respect to the input resolution. To address this issue, we
+propose an efficient BEV-based 3D detection framework called BEVENet, which
+leverages a convolutional-only architectural design to circumvent the
+limitations of ViT models while maintaining the effectiveness of BEV-based
+methods. Our experiments show that BEVENet is 3$\times$ faster than
+contemporary state-of-the-art (SOTA) approaches on the NuScenes challenge,
+achieving a mean average precision (mAP) of 0.456 and a nuScenes detection
+score (NDS) of 0.555 on the NuScenes validation dataset, with an inference
+speed of 47.6 frames per second. To the best of our knowledge, this study
+stands as the first to achieve such significant efficiency improvements for
+BEV-based methods, highlighting their enhanced feasibility for real-world
+autonomous driving applications.",cs.CV,"['cs.CV', 'cs.AI']"
+GARField: Group Anything with Radiance Fields,Chung Min Kim · Mingxuan Wu · Justin Kerr · Ken Goldberg · Matthew Tancik · Angjoo Kanazawa, ,https://arxiv.org/abs/2401.09419,,2401.09419.pdf,GARField: Group Anything with Radiance Fields,"Grouping is inherently ambiguous due to the multiple levels of granularity in
+which one can decompose a scene -- should the wheels of an excavator be
+considered separate or part of the whole? We present Group Anything with
+Radiance Fields (GARField), an approach for decomposing 3D scenes into a
+hierarchy of semantically meaningful groups from posed image inputs. To do this
+we embrace group ambiguity through physical scale: by optimizing a
+scale-conditioned 3D affinity feature field, a point in the world can belong to
+different groups of different sizes. We optimize this field from a set of 2D
+masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
+hierarchy, using scale to consistently fuse conflicting masks from different
+viewpoints. From this field we can derive a hierarchy of possible groupings via
+automatic tree construction or user interaction. We evaluate GARField on a
+variety of in-the-wild scenes and find it effectively extracts groups at many
+levels: clusters of objects, objects, and various subparts. GARField inherently
+represents multi-view consistent groupings and produces higher fidelity groups
+than the input SAM masks. GARField's hierarchical grouping could have exciting
+downstream applications such as 3D asset extraction or dynamic scene
+understanding. See the project website at https://www.garfield.studio/",cs.CV,"['cs.CV', 'cs.GR']"
+General Point Model Pretraining with Autoencoding and Autoregressive,Zhe Li · Zhangyang Gao · Cheng Tan · Bocheng Ren · Laurence Yang · Stan Z. Li, ,https://arxiv.org/abs/2310.16861,,2310.16861.pdf,General Point Model with Autoencoding and Autoregressive,"The pre-training architectures of large language models encompass various
+types, including autoencoding models, autoregressive models, and
+encoder-decoder models. We posit that any modality can potentially benefit from
+a large language model, as long as it undergoes vector quantization to become
+discrete tokens. Inspired by GLM, we propose a General Point Model (GPM) which
+seamlessly integrates autoencoding and autoregressive tasks in point cloud
+transformer. This model is versatile, allowing fine-tuning for downstream point
+cloud representation tasks, as well as unconditional and conditional generation
+tasks. GPM enhances masked prediction in autoencoding through various forms of
+mask padding tasks, leading to improved performance in point cloud
+understanding. Additionally, GPM demonstrates highly competitive results in
+unconditional point cloud generation tasks, even exhibiting the potential for
+conditional generation tasks by modifying the input's conditional information.
+Compared to models like Point-BERT, MaskPoint and PointMAE, our GPM achieves
+superior performance in point cloud understanding tasks. Furthermore, the
+integration of autoregressive and autoencoding within the same transformer
+underscores its versatility across different downstream tasks.",cs.LG,"['cs.LG', 'cs.CV']"
+NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation,Minh-Tuan Tran · Trung Le · Xuan-May Le · Mehrtash Harandi · Quan Tran · Dinh Phung,https://github.com/tmtuan1307/NAYER,https://arxiv.org/abs/2310.00258,,2310.00258.pdf,NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation,"Data-Free Knowledge Distillation (DFKD) has made significant recent strides
+by transferring knowledge from a teacher neural network to a student neural
+network without accessing the original data. Nonetheless, existing approaches
+encounter a significant challenge when attempting to generate samples from
+random noise inputs, which inherently lack meaningful information.
+Consequently, these models struggle to effectively map this noise to the
+ground-truth sample distribution, resulting in prolonging training times and
+low-quality outputs. In this paper, we propose a novel Noisy Layer Generation
+method (NAYER) which relocates the random source from the input to a noisy
+layer and utilizes the meaningful constant label-text embedding (LTE) as the
+input. LTE is generated by using the language model once, and then it is stored
+in memory for all subsequent training processes. The significance of LTE lies
+in its ability to contain substantial meaningful inter-class information,
+enabling the generation of high-quality samples with only a few training steps.
+Simultaneously, the noisy layer plays a key role in addressing the issue of
+diversity in sample generation by preventing the model from overemphasizing the
+constrained label information. By reinitializing the noisy layer in each
+iteration, we aim to facilitate the generation of diverse samples while still
+retaining the method's efficiency, thanks to the ease of learning provided by
+LTE. Experiments carried out on multiple datasets demonstrate that our NAYER
+not only outperforms the state-of-the-art methods but also achieves speeds 5 to
+15 times faster than previous approaches. The code is available at
+https://github.com/tmtuan1307/nayer.",cs.CV,['cs.CV']
+MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning,Zhe Li · Laurence Yang · Bocheng Ren · Xin Nie · Zhangyang Gao · Cheng Tan · Stan Z. Li, ,https://arxiv.org/abs/2402.02045,,2402.02045.pdf,MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning,"The scarcity of annotated data has sparked significant interest in
+unsupervised pre-training methods that leverage medical reports as auxiliary
+signals for medical visual representation learning. However, existing research
+overlooks the multi-granularity nature of medical visual representation and
+lacks suitable contrastive learning techniques to improve the models'
+generalizability across different granularities, leading to the
+underutilization of image-text information. To address this, we propose MLIP, a
+novel framework leveraging domain-specific medical knowledge as guiding signals
+to integrate language information into the visual domain through image-text
+contrastive learning. Our model includes global contrastive learning with our
+designed divergence encoder, local token-knowledge-patch alignment contrastive
+learning, and knowledge-guided category-level contrastive learning with expert
+knowledge. Experimental evaluations reveal the efficacy of our model in
+enhancing transfer performance for tasks such as image classification, object
+detection, and semantic segmentation. Notably, MLIP surpasses state-of-the-art
+methods even with limited annotated data, highlighting the potential of
+multimodal pre-training in advancing medical representation learning.",cs.CV,['cs.CV']
+Inversion-Free Image Editing with Language-Guided Diffusion Models,Sihan Xu · Yidong Huang · Jiayi Pan · Ziqiao Ma · Joyce Chai,https://sled-group.github.io/InfEdit/,https://arxiv.org/abs/2312.04965,,2312.04965.pdf,Inversion-Free Image Editing with Natural Language,"Despite recent advances in inversion-based editing, text-guided image
+manipulation remains challenging for diffusion models. The primary bottlenecks
+include 1) the time-consuming nature of the inversion process; 2) the struggle
+to balance consistency with accuracy; 3) the lack of compatibility with
+efficient consistency sampling methods used in consistency models. To address
+the above issues, we start by asking ourselves if the inversion process can be
+eliminated for editing. We show that when the initial sample is known, a
+special variance schedule reduces the denoising step to the same form as the
+multi-step consistency sampling. We name this Denoising Diffusion Consistent
+Model (DDCM), and note that it implies a virtual inversion strategy without
+explicit inversion in sampling. We further unify the attention control
+mechanisms in a tuning-free framework for text-guided editing. Combining them,
+we present inversion-free editing (InfEdit), which allows for consistent and
+faithful editing for both rigid and non-rigid semantic changes, catering to
+intricate modifications without compromising on the image's integrity and
+explicit inversion. Through extensive experiments, InfEdit shows strong
+performance in various editing tasks and also maintains a seamless workflow
+(less than 3 seconds on one single A40), demonstrating the potential for
+real-time applications. Project Page: https://sled-group.github.io/InfEdit/",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+DUDF: Differentiable Unsigned Distance Fields with Hyperbolic Scaling,Miguel Fainstein · Viviana Siless · Emmanuel Iarussi,https://lia-ditella.github.io/DUDF/,https://arxiv.org/abs/2402.08876,,2402.08876.pdf,DUDF: Differentiable Unsigned Distance Fields with Hyperbolic Scaling,"In recent years, there has been a growing interest in training Neural
+Networks to approximate Unsigned Distance Fields (UDFs) for representing open
+surfaces in the context of 3D reconstruction. However, UDFs are
+non-differentiable at the zero level set which leads to significant errors in
+distances and gradients, generally resulting in fragmented and discontinuous
+surfaces. In this paper, we propose to learn a hyperbolic scaling of the
+unsigned distance field, which defines a new Eikonal problem with distinct
+boundary conditions. This allows our formulation to integrate seamlessly with
+state-of-the-art continuously differentiable implicit neural representation
+networks, largely applied in the literature to represent signed distance
+fields. Our approach not only addresses the challenge of open surface
+representation but also demonstrates significant improvement in reconstruction
+quality and training performance. Moreover, the unlocked field's
+differentiability allows the accurate computation of essential topological
+properties such as normal directions and curvatures, pervasive in downstream
+tasks such as rendering. Through extensive experiments, we validate our
+approach across various data sets and against competitive baselines. The
+results demonstrate enhanced accuracy and up to an order of magnitude increase
+in speed compared to previous methods.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'I.2.10; I.4.10; I.3.7']"
+RoMa: Robust Dense Feature Matching,Johan Edstedt · Qiyu Sun · Georg Bökman · Mårten Wadenbäck · Michael Felsberg,https://parskatt.github.io/RoMa/,https://arxiv.org/html/2305.15404v2,,2305.15404v2.pdf,RoMa: Robust Dense Feature Matching,"Feature matching is an important computer vision task that involves
+estimating correspondences between two images of a 3D scene, and dense methods
+estimate all such correspondences. The aim is to learn a robust model, i.e., a
+model able to match under challenging real-world changes. In this work, we
+propose such a model, leveraging frozen pretrained features from the foundation
+model DINOv2. Although these features are significantly more robust than local
+features trained from scratch, they are inherently coarse. We therefore combine
+them with specialized ConvNet fine features, creating a precisely localizable
+feature pyramid. To further improve robustness, we propose a tailored
+transformer match decoder that predicts anchor probabilities, which enables it
+to express multimodality. Finally, we propose an improved loss formulation
+through regression-by-classification with subsequent robust regression. We
+conduct a comprehensive set of experiments that show that our method, RoMa,
+achieves significant gains, setting a new state-of-the-art. In particular, we
+achieve a 36% improvement on the extremely challenging WxBS benchmark. Code is
+provided at https://github.com/Parskatt/RoMa",cs.CV,['cs.CV']
+Harnessing Large Language Models for Training-free Video Anomaly Detection,Luca Zanella · Willi Menapace · Massimiliano Mancini · Yiming Wang · Elisa Ricci, ,,https://paperswithcode.com/paper/harnessing-large-language-models-for-training,,,,,nan
+Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models,Nikita Starodubcev · Dmitry Baranchuk · Artem Fedorov · Artem Babenko, ,https://arxiv.org/abs/2312.10835,,2312.10835.pdf,Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models,"Knowledge distillation methods have recently shown to be a promising
+direction to speedup the synthesis of large-scale diffusion models by requiring
+only a few inference steps. While several powerful distillation methods were
+recently proposed, the overall quality of student samples is typically lower
+compared to the teacher ones, which hinders their practical usage. In this
+work, we investigate the relative quality of samples produced by the teacher
+text-to-image diffusion model and its distilled student version. As our main
+empirical finding, we discover that a noticeable portion of student samples
+exhibit superior fidelity compared to the teacher ones, despite the
+""approximate"" nature of the student. Based on this finding, we propose an
+adaptive collaboration between student and teacher diffusion models for
+effective text-to-image synthesis. Specifically, the distilled model produces
+the initial sample, and then an oracle decides whether it needs further
+improvements with a slow teacher model. Extensive experiments demonstrate that
+the designed pipeline surpasses state-of-the-art text-to-image alternatives for
+various inference budgets in terms of human preference. Furthermore, the
+proposed approach can be naturally used in popular applications such as
+text-guided image editing and controllable generation.",cs.CV,['cs.CV']
+Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering,Zhaohe Liao · Jiangtong Li · Li Niu · Liqing Zhang, ,,https://dl.acm.org/doi/abs/10.1145/3581783.3613909,,,,,nan
+$360+x$: A Panoptic Multi-modal Scene Understanding Dataset,Hao Chen · Yuqi Hou · Chenyuan Qu · Irene Testini · Xiaohan Hong · Jianbo Jiao,https://x360dataset.github.io/,https://arxiv.org/abs/2404.00989,,2404.00989.pdf,360+x: A Panoptic Multi-modal Scene Understanding Dataset,"Human perception of the world is shaped by a multitude of viewpoints and
+modalities. While many existing datasets focus on scene understanding from a
+certain perspective (e.g. egocentric or third-person views), our dataset offers
+a panoptic perspective (i.e. multiple viewpoints with multiple data
+modalities). Specifically, we encapsulate third-person panoramic and front
+views, as well as egocentric monocular/binocular views with rich modalities
+including video, multi-channel audio, directional binaural delay, location data
+and textual scene descriptions within each scene captured, presenting
+comprehensive observation of the world. Figure 1 offers a glimpse of all 28
+scene categories of our 360+x dataset. To the best of our knowledge, this is
+the first database that covers multiple viewpoints with multiple data
+modalities to mimic how daily information is accessed in the real world.
+Through our benchmark analysis, we presented 5 different scene understanding
+tasks on the proposed 360+x dataset to evaluate the impact and benefit of each
+data modality and perspective in panoptic scene understanding. We hope this
+unique dataset could broaden the scope of comprehensive scene understanding and
+encourage the community to approach these problems from more diverse
+perspectives.",cs.CV,"['cs.CV', 'cs.AI', 'cs.MM', 'cs.SD', 'eess.AS']"
+Text-Enhanced Data-free Approach for Federated Class-Incremental Learning,Minh-Tuan Tran · Trung Le · Xuan-May Le · Mehrtash Harandi · Dinh Phung,https://github.com/tmtuan1307/LANDER,https://arxiv.org/abs/2403.14101,,2403.14101.pdf,Text-Enhanced Data-free Approach for Federated Class-Incremental Learning,"Federated Class-Incremental Learning (FCIL) is an underexplored yet pivotal
+issue, involving the dynamic addition of new classes in the context of
+federated learning. In this field, Data-Free Knowledge Transfer (DFKT) plays a
+crucial role in addressing catastrophic forgetting and data privacy problems.
+However, prior approaches lack the crucial synergy between DFKT and the model
+training phases, causing DFKT to encounter difficulties in generating
+high-quality data from a non-anchored latent space of the old task model. In
+this paper, we introduce LANDER (Label Text Centered Data-Free Knowledge
+Transfer) to address this issue by utilizing label text embeddings (LTE)
+produced by pretrained language models. Specifically, during the model training
+phase, our approach treats LTE as anchor points and constrains the feature
+embeddings of corresponding training samples around them, enriching the
+surrounding area with more meaningful information. In the DFKT phase, by using
+these LTE anchors, LANDER can synthesize more meaningful samples, thereby
+effectively addressing the forgetting problem. Additionally, instead of tightly
+constraining embeddings toward the anchor, the Bounding Loss is introduced to
+encourage sample embeddings to remain flexible within a defined radius. This
+approach preserves the natural differences in sample embeddings and mitigates
+the embedding overlap caused by heterogeneous federated settings. Extensive
+experiments conducted on CIFAR100, Tiny-ImageNet, and ImageNet demonstrate that
+LANDER significantly outperforms previous methods and achieves state-of-the-art
+performance in FCIL. The code is available at
+https://github.com/tmtuan1307/lander.",cs.CV,"['cs.CV', 'cs.CL', 'cs.LG']"
+Rethinking Boundary Discontinuity Problem for Oriented Object Detection,Hang Xu · Xinyuan Liu · Haonan Xu · Yike Ma · Zunjie Zhu · Chenggang Yan · Feng Dai,https://github.com/hangxu-cv/cvpr24acm,,https://ieeexplore.ieee.org/abstract/document/10475581,,,,,nan
+GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation,Tong Wu · Guandao Yang · Zhibing Li · Kai Zhang · Ziwei Liu · Leonidas Guibas · Dahua Lin · Gordon Wetzstein, ,https://arxiv.org/abs/2401.04092,,2401.04092.pdf,GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation,"Despite recent advances in text-to-3D generative methods, there is a notable
+absence of reliable evaluation metrics. Existing metrics usually focus on a
+single criterion each, such as how well the asset aligned with the input text.
+These metrics lack the flexibility to generalize to different evaluation
+criteria and might not align well with human preferences. Conducting user
+preference studies is an alternative that offers both adaptability and
+human-aligned results. User studies, however, can be very expensive to scale.
+This paper presents an automatic, versatile, and human-aligned evaluation
+metric for text-to-3D generative models. To this end, we first develop a prompt
+generator using GPT-4V to generate evaluating prompts, which serve as input to
+compare text-to-3D models. We further design a method instructing GPT-4V to
+compare two 3D assets according to user-defined criteria. Finally, we use these
+pairwise comparison results to assign these models Elo ratings. Experimental
+results suggest our metric strongly align with human preference across
+different evaluation criteria.",cs.CV,['cs.CV']
+Adversarial Text to Continuous Image Generation,Kilichbek Haydarov · Aashiq Muhamed · Xiaoqian Shen · Jovana Lazarevic · Ivan Skorokhodov · Chamuditha Jayanga Galappaththige · Mohamed Elhoseiny, ,https://arxiv.org/abs/2312.14440,,2312.14440.pdf,Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks,"The widespread use of Text-to-Image (T2I) models in content generation
+requires careful examination of their safety, including their robustness to
+adversarial attacks. Despite extensive research on adversarial attacks, the
+reasons for their effectiveness remain underexplored. This paper presents an
+empirical study on adversarial attacks against T2I models, focusing on
+analyzing factors associated with attack success rates (ASR). We introduce a
+new attack objective - entity swapping using adversarial suffixes and two
+gradient-based attack algorithms. Human and automatic evaluations reveal the
+asymmetric nature of ASRs on entity swap: for example, it is easier to replace
+""human"" with ""robot"" in the prompt ""a human dancing in the rain."" with an
+adversarial suffix, but the reverse replacement is significantly harder. We
+further propose probing metrics to establish indicative signals from the
+model's beliefs to the adversarial ASR. We identify conditions that result in a
+success probability of 60% for adversarial attacks and others where this
+likelihood drops below 5%.",cs.LG,"['cs.LG', 'cs.CR']"
+Contextrast: Contextual Contrastive Learning for Semantic Segmentation,Changki Sung · Wanhee Kim · Jungho An · WooJu Lee · Hyungtae Lim · Hyun Myung, ,https://arxiv.org/abs/2404.10633,,2404.10633.pdf,Contextrast: Contextual Contrastive Learning for Semantic Segmentation,"Despite great improvements in semantic segmentation, challenges persist
+because of the lack of local/global contexts and the relationship between them.
+In this paper, we propose Contextrast, a contrastive learning-based semantic
+segmentation method that allows to capture local/global contexts and comprehend
+their relationships. Our proposed method comprises two parts: a) contextual
+contrastive learning (CCL) and b) boundary-aware negative (BANE) sampling.
+Contextual contrastive learning obtains local/global context from multi-scale
+feature aggregation and inter/intra-relationship of features for better
+discrimination capabilities. Meanwhile, BANE sampling selects embedding
+features along the boundaries of incorrectly predicted regions to employ them
+as harder negative samples on our contrastive learning, resolving segmentation
+issues along the boundary region by exploiting fine-grained details. We
+demonstrate that our Contextrast substantially enhances the performance of
+semantic segmentation networks, outperforming state-of-the-art contrastive
+learning approaches on diverse public datasets, e.g. Cityscapes, CamVid,
+PASCAL-C, COCO-Stuff, and ADE20K, without an increase in computational cost
+during inference.",cs.CV,['cs.CV']
+DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception,Yibo Wang · Ruiyuan Gao · Kai Chen · Kaiqiang Zhou · Yingjie CAI · Lanqing Hong · Zhenguo Li · Lihui Jiang · Dit-Yan Yeung · Qiang Xu · Kai Zhang, ,https://arxiv.org/abs/2403.13304,,2403.13304.pdf,DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception,"Current perceptive models heavily depend on resource-intensive datasets,
+prompting the need for innovative solutions. Leveraging recent advances in
+diffusion models, synthetic data, by constructing image inputs from various
+annotations, proves beneficial for downstream tasks. While prior methods have
+separately addressed generative and perceptive models, DetDiffusion, for the
+first time, harmonizes both, tackling the challenges in generating effective
+data for perceptive models. To enhance image generation with perceptive models,
+we introduce perception-aware loss (P.A. loss) through segmentation, improving
+both quality and controllability. To boost the performance of specific
+perceptive models, our method customizes data augmentation by extracting and
+utilizing perception-aware attribute (P.A. Attr) during generation.
+Experimental results from the object detection task highlight DetDiffusion's
+superior performance, establishing a new state-of-the-art in layout-guided
+generation. Furthermore, image syntheses from DetDiffusion can effectively
+augment training data, significantly enhancing downstream detection
+performance.",cs.CV,['cs.CV']
+Boosting Image Quality Assessment through Efficient Transformer Adaptation with Local Feature Enhancement,Kangmin Xu · Liang Liao · Jing Xiao · Chaofeng Chen · Haoning Wu · Qiong Yan · Weisi Lin, ,https://arxiv.org/abs/2308.12001,,2308.12001.pdf,Local Distortion Aware Efficient Transformer Adaptation for Image Quality Assessment,"Image Quality Assessment (IQA) constitutes a fundamental task within the
+field of computer vision, yet it remains an unresolved challenge, owing to the
+intricate distortion conditions, diverse image contents, and limited
+availability of data. Recently, the community has witnessed the emergence of
+numerous large-scale pretrained foundation models, which greatly benefit from
+dramatically increased data and parameter capacities. However, it remains an
+open problem whether the scaling law in high-level tasks is also applicable to
+IQA task which is closely related to low-level clues. In this paper, we
+demonstrate that with proper injection of local distortion features, a larger
+pretrained and fixed foundation model performs better in IQA tasks.
+Specifically, for the lack of local distortion structure and inductive bias of
+vision transformer (ViT), alongside the large-scale pretrained ViT, we use
+another pretrained convolution neural network (CNN), which is well known for
+capturing the local structure, to extract multi-scale image features. Further,
+we propose a local distortion extractor to obtain local distortion features
+from the pretrained CNN and a local distortion injector to inject the local
+distortion features into ViT. By only training the extractor and injector, our
+method can benefit from the rich knowledge in the powerful foundation models
+and achieve state-of-the-art performance on popular IQA datasets, indicating
+that IQA is not only a low-level problem but also benefits from stronger
+high-level features drawn from large-scale pretrained models.",cs.CV,['cs.CV']
+Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples,Junhao Dong · Piotr Koniusz · Junxi Chen · Z. Wang · Yew-Soon Ong, ,,https://www.a-star.edu.sg/cfar/news/news/features/10-papers-accepted-at-cvpr-2024,,,,,nan
+Adversarially Robust Few-shot Learning via Parameter Co-distillation of Similarity and Class Concept Learners,Junhao Dong · Piotr Koniusz · Junxi Chen · Xiaohua Xie · Yew-Soon Ong, ,,https://openreview.net/forum?id=h9TTpQdGKJ,,,,,nan
+Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition,Kyle Buettner · Sina Malakouti · Xiang Li · Adriana Kovashka,https://krbuettner.github.io/GeoKnowledgePrompting/,https://arxiv.org/abs/2401.01482,,2401.01482.pdf,Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition,"Existing object recognition models have been shown to lack robustness in
+diverse geographical scenarios due to domain shifts in design and context.
+Class representations need to be adapted to more accurately reflect an object
+concept under these shifts. In the absence of training data from target
+geographies, we hypothesize that geographically diverse descriptive knowledge
+of categories can enhance robustness. For this purpose, we explore the
+feasibility of probing a large language model for geography-based object
+knowledge, and we examine the effects of integrating knowledge into zero-shot
+and learnable soft prompting with CLIP. Within this exploration, we propose
+geography knowledge regularization to ensure that soft prompts trained on a
+source set of geographies generalize to an unseen target set. Accuracy gains
+over prompting baselines on DollarStreet while training only on Europe data are
+up to +2.8/1.2/1.6 on target data from Africa/Asia/Americas, and +4.6 overall
+on the hardest classes. Competitive performance is shown vs. few-shot target
+training, and analysis is provided to direct future study of geographical
+robustness.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG']"
+Clustering for Protein Representation Learning,Ruijie Quan · Wenguan Wang · Fan Ma · Hehe Fan · Yi Yang, ,https://arxiv.org/abs/2404.00254,,2404.00254.pdf,Clustering for Protein Representation Learning,"Protein representation learning is a challenging task that aims to capture
+the structure and function of proteins from their amino acid sequences.
+Previous methods largely ignored the fact that not all amino acids are equally
+important for protein folding and activity. In this article, we propose a
+neural clustering framework that can automatically discover the critical
+components of a protein by considering both its primary and tertiary structure
+information. Our framework treats a protein as a graph, where each node
+represents an amino acid and each edge represents a spatial or sequential
+connection between amino acids. We then apply an iterative clustering strategy
+to group the nodes into clusters based on their 1D and 3D positions and assign
+scores to each cluster. We select the highest-scoring clusters and use their
+medoid nodes for the next iteration of clustering, until we obtain a
+hierarchical and informative representation of the protein. We evaluate on four
+protein-related tasks: protein fold classification, enzyme reaction
+classification, gene ontology term prediction, and enzyme commission number
+prediction. Experimental results demonstrate that our method achieves
+state-of-the-art performance.",cs.LG,"['cs.LG', 'cs.CE', 'q-bio.BM', 'q-bio.QM']"
+Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment,Alvi Md Ishmam · Chris Thomas, ,https://arxiv.org/abs/2402.06659,,2402.06659.pdf,Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models,"Vision-Language Models (VLMs) excel in generating textual responses from
+visual inputs, yet their versatility raises significant security concerns. This
+study takes the first step in exposing VLMs' susceptibility to data poisoning
+attacks that can manipulate responses to innocuous, everyday prompts. We
+introduce Shadowcast, a stealthy data poisoning attack method where poison
+samples are visually indistinguishable from benign images with matching texts.
+Shadowcast demonstrates effectiveness in two attack types. The first is Label
+Attack, tricking VLMs into misidentifying class labels, such as confusing
+Donald Trump for Joe Biden. The second is Persuasion Attack, which leverages
+VLMs' text generation capabilities to craft narratives, such as portraying junk
+food as health food, through persuasive and seemingly rational descriptions. We
+show that Shadowcast are highly effective in achieving attacker's intentions
+using as few as 50 poison samples. Moreover, these poison samples remain
+effective across various prompts and are transferable across different VLM
+architectures in the black-box setting. This work reveals how poisoned VLMs can
+generate convincing yet deceptive misinformation and underscores the importance
+of data quality for responsible deployments of VLMs. Our code is available at:
+https://github.com/umd-huang-lab/VLM-Poisoning.",cs.CR,"['cs.CR', 'cs.AI', 'cs.LG']"
+Structured Model Probing: Empowering Efficient Transfer Learning by Structured Regularization,Zhi-Fan Wu · Chaojie Mao · Xue Wang · Jianwen Jiang · Yiliang Lv · Rong Jin, ,https://arxiv.org/abs/2403.10799,,2403.10799.pdf,Efficient Pruning of Large Language Model with Adaptive Estimation Fusion,"Large language models (LLMs) have become crucial for many generative
+downstream tasks, leading to an inevitable trend and significant challenge to
+deploy them efficiently on resource-constrained devices. Structured pruning is
+a widely used method to address this challenge. However, when dealing with the
+complex structure of the multiple decoder layers, general methods often employ
+common estimation approaches for pruning. These approaches lead to a decline in
+accuracy for specific downstream tasks. In this paper, we introduce a simple
+yet efficient method that adaptively models the importance of each
+substructure. Meanwhile, it can adaptively fuse coarse-grained and finegrained
+estimations based on the results from complex and multilayer structures. All
+aspects of our design seamlessly integrate into the endto-end pruning
+framework. Our experimental results, compared with state-of-the-art methods on
+mainstream datasets, demonstrate average accuracy improvements of 1.1%, 1.02%,
+2.0%, and 1.2% for LLaMa-7B,Vicuna-7B, Baichuan-7B, and Bloom-7b1,
+respectively.",cs.CL,"['cs.CL', 'cs.AI', 'cs.LG']"
+Artist-Friendly Relightable and Animatable Neural Heads,Yingyan Xu · Prashanth Chandran · Sebastian Weiss · Markus Gross · Gaspard Zoss · Derek Bradley,https://studios.disneyresearch.com/2024/06/03/artist-friendly-relightable-and-animatable-neural-heads/,https://arxiv.org/abs/2312.03420,,2312.03420.pdf,Artist-Friendly Relightable and Animatable Neural Heads,"An increasingly common approach for creating photo-realistic digital avatars
+is through the use of volumetric neural fields. The original neural radiance
+field (NeRF) allowed for impressive novel view synthesis of static heads when
+trained on a set of multi-view images, and follow up methods showed that these
+neural representations can be extended to dynamic avatars. Recently, new
+variants also surpassed the usual drawback of baked-in illumination in neural
+representations, showing that static neural avatars can be relit in any
+environment. In this work we simultaneously tackle both the motion and
+illumination problem, proposing a new method for relightable and animatable
+neural heads. Our method builds on a proven dynamic avatar approach based on a
+mixture of volumetric primitives, combined with a recently-proposed lightweight
+hardware setup for relightable neural fields, and includes a novel architecture
+that allows relighting dynamic neural avatars performing unseen expressions in
+any environment, even with nearfield illumination and viewpoints.",cs.CV,"['cs.CV', 'cs.GR']"
+Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity,Ruijie Quan · Wenguan Wang · Zhibo Tian · Fan Ma · Yi Yang, ,https://arxiv.org/abs/2403.20022,,2403.20022.pdf,Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity,"Reconstructing the viewed images from human brain activity bridges human and
+computer vision through the Brain-Computer Interface. The inherent variability
+in brain function between individuals leads existing literature to focus on
+acquiring separate models for each individual using their respective brain
+signal data, ignoring commonalities between these data. In this article, we
+devise Psychometry, an omnifit model for reconstructing images from functional
+Magnetic Resonance Imaging (fMRI) obtained from different subjects. Psychometry
+incorporates an omni mixture-of-experts (Omni MoE) module where all the experts
+work together to capture the inter-subject commonalities, while each expert
+associated with subject-specific parameters copes with the individual
+differences. Moreover, Psychometry is equipped with a retrieval-enhanced
+inference strategy, termed Ecphory, which aims to enhance the learned fMRI
+representation via retrieving from prestored subject-specific memories. These
+designs collectively render Psychometry omnifit and efficient, enabling it to
+capture both inter-subject commonality and individual specificity across
+subjects. As a result, the enhanced fMRI representations serve as conditional
+signals to guide a generation model to reconstruct high-quality and realistic
+images, establishing Psychometry as state-of-the-art in terms of both
+high-level and low-level metrics.",cs.CV,['cs.CV']
+JointSQ: Joint Sparsification-Quantization for Distributed Learning,Weiying Xie · Haowei Li · Ma Jitao · Yunsong Li · Jie Lei · donglai Liu · Leyuan Fang, ,,https://www.semanticscholar.org/paper/Joint-Sparsification-and-Quantization-for-Wireless-Su-Wang/f940a77cd570b121a727d59cd249513930cd830a,,,,,nan
+PAPR in Motion: Seamless Point-level 3D Scene Interpolation,Shichong Peng · Yanshu Zhang · Ke Li, ,https://arxiv.org/abs/2307.11086,,2307.11086.pdf,PAPR: Proximity Attention Point Rendering,"Learning accurate and parsimonious point cloud representations of scene
+surfaces from scratch remains a challenge in 3D representation learning.
+Existing point-based methods often suffer from the vanishing gradient problem
+or require a large number of points to accurately model scene geometry and
+texture. To address these limitations, we propose Proximity Attention Point
+Rendering (PAPR), a novel method that consists of a point-based scene
+representation and a differentiable renderer. Our scene representation uses a
+point cloud where each point is characterized by its spatial position,
+influence score, and view-independent feature vector. The renderer selects the
+relevant points for each ray and produces accurate colours using their
+associated features. PAPR effectively learns point cloud positions to represent
+the correct scene geometry, even when the initialization drastically differs
+from the target geometry. Notably, our method captures fine texture details
+while using only a parsimonious set of points. We also demonstrate four
+practical applications of our method: zero-shot geometry editing, object
+manipulation, texture transfer, and exposure control. More results and code are
+available on our project website at https://zvict.github.io/papr/.",cs.CV,"['cs.CV', 'cs.AI', 'cs.GR', 'cs.LG', 'cs.NE']"
+Anatomically Constrained Implicit Face Models,Prashanth Chandran · Gaspard Zoss, ,https://arxiv.org/abs/2312.07538,,2312.07538.pdf,Anatomically Constrained Implicit Face Models,"Coordinate based implicit neural representations have gained rapid popularity
+in recent years as they have been successfully used in image, geometry and
+scene modeling tasks. In this work, we present a novel use case for such
+implicit representations in the context of learning anatomically constrained
+face models. Actor specific anatomically constrained face models are the state
+of the art in both facial performance capture and performance retargeting.
+Despite their practical success, these anatomical models are slow to evaluate
+and often require extensive data capture to be built. We propose the anatomical
+implicit face model; an ensemble of implicit neural networks that jointly learn
+to model the facial anatomy and the skin surface with high-fidelity, and can
+readily be used as a drop in replacement to conventional blendshape models.
+Given an arbitrary set of skin surface meshes of an actor and only a neutral
+shape with estimated skull and jaw bones, our method can recover a dense
+anatomical substructure which constrains every point on the facial surface. We
+demonstrate the usefulness of our approach in several tasks ranging from shape
+fitting, shape editing, and performance retargeting.",cs.GR,"['cs.GR', 'cs.CV']"
+EscherNet: A Generative Model for Scalable View Synthesis,Xin Kong · Shikun Liu · Xiaoyang Lyu · Marwan Taher · Xiaojuan Qi · Andrew J. Davison,https://kxhit.github.io/EscherNet,https://arxiv.org/abs/2402.03908,,2402.03908.pdf,EscherNet: A Generative Model for Scalable View Synthesis,"We introduce EscherNet, a multi-view conditioned diffusion model for view
+synthesis. EscherNet learns implicit and generative 3D representations coupled
+with a specialised camera positional encoding, allowing precise and continuous
+relative control of the camera transformation between an arbitrary number of
+reference and target views. EscherNet offers exceptional generality,
+flexibility, and scalability in view synthesis -- it can generate more than 100
+consistent target views simultaneously on a single consumer-grade GPU, despite
+being trained with a fixed number of 3 reference views to 3 target views. As a
+result, EscherNet not only addresses zero-shot novel view synthesis, but also
+naturally unifies single- and multi-image 3D reconstruction, combining these
+diverse tasks into a single, cohesive framework. Our extensive experiments
+demonstrate that EscherNet achieves state-of-the-art performance in multiple
+benchmarks, even when compared to methods specifically tailored for each
+individual problem. This remarkable versatility opens up new directions for
+designing scalable neural architectures for 3D vision. Project page:
+https://kxhit.github.io/EscherNet.",cs.CV,['cs.CV']
+Revisiting Adversarial Training under Long-Tailed Distributions,Xinli Yue · Ningping Mou · Qian Wang · Lingchen Zhao,https://github.com/NISPLab/AT-BSL,https://arxiv.org/abs/2403.10073,,2403.10073.pdf,Revisiting Adversarial Training under Long-Tailed Distributions,"Deep neural networks are vulnerable to adversarial attacks, often leading to
+erroneous outputs. Adversarial training has been recognized as one of the most
+effective methods to counter such attacks. However, existing adversarial
+training techniques have predominantly been tested on balanced datasets,
+whereas real-world data often exhibit a long-tailed distribution, casting doubt
+on the efficacy of these methods in practical scenarios.
+  In this paper, we delve into adversarial training under long-tailed
+distributions. Through an analysis of the previous work ""RoBal"", we discover
+that utilizing Balanced Softmax Loss alone can achieve performance comparable
+to the complete RoBal approach while significantly reducing training overheads.
+Additionally, we reveal that, similar to uniform distributions, adversarial
+training under long-tailed distributions also suffers from robust overfitting.
+To address this, we explore data augmentation as a solution and unexpectedly
+discover that, unlike results obtained with balanced data, data augmentation
+not only effectively alleviates robust overfitting but also significantly
+improves robustness. We further investigate the reasons behind the improvement
+of robustness through data augmentation and identify that it is attributable to
+the increased diversity of examples. Extensive experiments further corroborate
+that data augmentation alone can significantly improve robustness. Finally,
+building on these findings, we demonstrate that compared to RoBal, the
+combination of BSL and data augmentation leads to a +6.66% improvement in model
+robustness under AutoAttack on CIFAR-10-LT. Our code is available at
+https://github.com/NISPLab/AT-BSL .",cs.CV,['cs.CV']
+UniGS: Unified Representation for Image Generation and Segmentation,Lu Qi · Lehan Yang · Weidong Guo · Yu Xu · Bo Du · Varun Jampani · Ming-Hsuan Yang, ,https://arxiv.org/abs/2312.01985,,2312.01985.pdf,UniGS: Unified Representation for Image Generation and Segmentation,"This paper introduces a novel unified representation of diffusion models for
+image generation and segmentation. Specifically, we use a colormap to represent
+entity-level masks, addressing the challenge of varying entity numbers while
+aligning the representation closely with the image RGB domain. Two novel
+modules, including the location-aware color palette and progressive dichotomy
+module, are proposed to support our mask representation. On the one hand, a
+location-aware palette guarantees the colors' consistency to entities'
+locations. On the other hand, the progressive dichotomy module can efficiently
+decode the synthesized colormap to high-quality entity-level masks in a
+depth-first binary search without knowing the cluster numbers. To tackle the
+issue of lacking large-scale segmentation training data, we employ an
+inpainting pipeline and then improve the flexibility of diffusion models across
+various tasks, including inpainting, image synthesis, referring segmentation,
+and entity segmentation. Comprehensive experiments validate the efficiency of
+our approach, demonstrating comparable segmentation mask quality to
+state-of-the-art and adaptability to multiple tasks. The code will be released
+at \href{https://github.com/qqlu/Entity}{https://github.com/qqlu/Entity}.",cs.CV,['cs.CV']
+Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation,guo · Tianwei Lin, ,https://arxiv.org/abs/2312.10113,,2312.10113.pdf,Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation,"Recently, diffusion-based methods, like InstructPix2Pix (IP2P), have achieved
+effective instruction-based image editing, requiring only natural language
+instructions from the user. However, these methods often inadvertently alter
+unintended areas and struggle with multi-instruction editing, resulting in
+compromised outcomes. To address these issues, we introduce the Focus on Your
+Instruction (FoI), a method designed to ensure precise and harmonious editing
+across multiple instructions without extra training or test-time optimization.
+In the FoI, we primarily emphasize two aspects: (1) precisely extracting
+regions of interest for each instruction and (2) guiding the denoising process
+to concentrate within these regions of interest. For the first objective, we
+identify the implicit grounding capability of IP2P from the cross-attention
+between instruction and image, then develop an effective mask extraction
+method. For the second objective, we introduce a cross attention modulation
+module for rough isolation of target editing regions and unrelated regions.
+Additionally, we introduce a mask-guided disentangle sampling strategy to
+further ensure clear region isolation. Experimental results demonstrate that
+FoI surpasses existing methods in both quantitative and qualitative
+evaluations, especially excelling in multi-instruction editing task.",cs.CV,['cs.CV']
+MorpheuS: Neural Dynamic 360$^{\circ}$ Surface Reconstruction from Monocular RGB-D Video,Hengyi Wang · Jingwen Wang · Lourdes Agapito,https://hengyiwang.github.io/projects/morpheus.html,https://arxiv.org/abs/2312.00778,,2312.00778.pdf,MorpheuS: Neural Dynamic 360° Surface Reconstruction from Monocular RGB-D Video,"Neural rendering has demonstrated remarkable success in dynamic scene
+reconstruction. Thanks to the expressiveness of neural representations, prior
+works can accurately capture the motion and achieve high-fidelity
+reconstruction of the target object. Despite this, real-world video scenarios
+often feature large unobserved regions where neural representations struggle to
+achieve realistic completion. To tackle this challenge, we introduce MorpheuS,
+a framework for dynamic 360{\deg} surface reconstruction from a casually
+captured RGB-D video. Our approach models the target scene as a canonical field
+that encodes its geometry and appearance, in conjunction with a deformation
+field that warps points from the current frame to the canonical space. We
+leverage a view-dependent diffusion prior and distill knowledge from it to
+achieve realistic completion of unobserved regions. Experimental results on
+various real-world and synthetic datasets show that our method can achieve
+high-fidelity 360{\deg} surface reconstruction of a deformable object from a
+monocular RGB-D video.",cs.CV,['cs.CV']
+DiffusionLight: Light Probes for Free by Painting a Chrome Ball,Pakkapon Phongthawee · Worameth Chinchuthakun · Nontaphat Sinsunthithet · Varun Jampani · Amit Raj · Pramook Khungurn · Supasorn Suwajanakorn,https://diffusionlight.github.io/,https://arxiv.org/abs/2312.09168v2,,2312.09168v2.pdf,DiffusionLight: Light Probes for Free by Painting a Chrome Ball,"We present a simple yet effective technique to estimate lighting in a single
+input image. Current techniques rely heavily on HDR panorama datasets to train
+neural networks to regress an input with limited field-of-view to a full
+environment map. However, these approaches often struggle with real-world,
+uncontrolled settings due to the limited diversity and size of their datasets.
+To address this problem, we leverage diffusion models trained on billions of
+standard images to render a chrome ball into the input image. Despite its
+simplicity, this task remains challenging: the diffusion models often insert
+incorrect or inconsistent objects and cannot readily generate images in HDR
+format. Our research uncovers a surprising relationship between the appearance
+of chrome balls and the initial diffusion noise map, which we utilize to
+consistently generate high-quality chrome balls. We further fine-tune an LDR
+difusion model (Stable Diffusion XL) with LoRA, enabling it to perform exposure
+bracketing for HDR light estimation. Our method produces convincing light
+estimates across diverse settings and demonstrates superior generalization to
+in-the-wild scenarios.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG', 'I.3.3; I.4.8']"
+JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments,Duy Tho Le · Chenhui Gou · Stavya Datta · Hengcan Shi · Ian Reid · Jianfei Cai · Hamid Rezatofighi, ,https://arxiv.org/abs/2404.01686,,2404.01686.pdf,JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments,"Autonomous robot systems have attracted increasing research attention in
+recent years, where environment understanding is a crucial step for robot
+navigation, human-robot interaction, and decision. Real-world robot systems
+usually collect visual data from multiple sensors and are required to recognize
+numerous objects and their movements in complex human-crowded settings.
+Traditional benchmarks, with their reliance on single sensors and limited
+object classes and scenarios, fail to provide the comprehensive environmental
+understanding robots need for accurate navigation, interaction, and
+decision-making. As an extension of JRDB dataset, we unveil JRDB-PanoTrack, a
+novel open-world panoptic segmentation and tracking benchmark, towards more
+comprehensive environmental perception. JRDB-PanoTrack includes (1) various
+data involving indoor and outdoor crowded scenes, as well as comprehensive 2D
+and 3D synchronized data modalities; (2) high-quality 2D spatial panoptic
+segmentation and temporal tracking annotations, with additional 3D label
+projections for further spatial understanding; (3) diverse object classes for
+closed- and open-world recognition benchmarks, with OSPA-based metrics for
+evaluation. Extensive evaluation of leading methods shows significant
+challenges posed by our dataset.",cs.CV,['cs.CV']
+Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning,Youqi Pan · Wugen Zhou · Yingdian Cao · Hongbin Zha, ,https://arxiv.org/html/2308.11228v2,,2308.11228v2.pdf,VIO-DualProNet: Visual-Inertial Odometry with Learning Based Process Noise Covariance,"Visual-inertial odometry (VIO) is a vital technique used in robotics,
+augmented reality, and autonomous vehicles. It combines visual and inertial
+measurements to accurately estimate position and orientation. Existing VIO
+methods assume a fixed noise covariance for the inertial uncertainty. However,
+accurately determining in real-time the noise variance of the inertial sensors
+presents a significant challenge as the uncertainty changes throughout the
+operation leading to suboptimal performance and reduced accuracy. To circumvent
+this, we propose VIO-DualProNet, a novel approach that utilizes deep learning
+methods to dynamically estimate the inertial noise uncertainty in real-time. By
+designing and training a deep neural network to predict inertial noise
+uncertainty using only inertial sensor measurements, and integrating it into
+the VINS-Mono algorithm, we demonstrate a substantial improvement in accuracy
+and robustness, enhancing VIO performance and potentially benefiting other
+VIO-based systems for precise localization and mapping across diverse
+conditions.",cs.RO,"['cs.RO', 'cs.SY', 'eess.SY']"
+ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images,Yiqi Shi · Duo Liu · Liguo Zhang · Ye Tian · Xuezhi Xia · fuxiaojing,https://github.com/Doyle59217/ZeroIG,https://arxiv.org/abs/2311.02995,,2311.02995.pdf,Zero-Shot Enhancement of Low-Light Image Based on Retinex Decomposition,"Two difficulties here make low-light image enhancement a challenging task;
+firstly, it needs to consider not only luminance restoration but also image
+contrast, image denoising and color distortion issues simultaneously. Second,
+the effectiveness of existing low-light enhancement methods depends on paired
+or unpaired training data with poor generalization performance.
+  To solve these difficult problems, we propose in this paper a new
+learning-based Retinex decomposition of zero-shot low-light enhancement method,
+called ZERRINNet. To this end, we first designed the N-Net network, together
+with the noise loss term, to be used for denoising the original low-light image
+by estimating the noise of the low-light image. Moreover, RI-Net is used to
+estimate the reflection component and illumination component, and in order to
+solve the color distortion and contrast, we use the texture loss term and
+segmented smoothing loss to constrain the reflection component and illumination
+component. Finally, our method is a zero-reference enhancement method that is
+not affected by the training data of paired and unpaired datasets, so our
+generalization performance is greatly improved, and in the paper, we have
+effectively validated it with a homemade real-life low-light dataset and
+additionally with advanced vision tasks, such as face detection, target
+recognition, and instance segmentation. We conducted comparative experiments on
+a large number of public datasets and the results show that the performance of
+our method is competitive compared to the current state-of-the-art methods. The
+code is available at:https://github.com/liwenchao0615/ZERRINNet",cs.CV,"['cs.CV', 'cs.GR']"
+Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models,Gianni Franchi · Olivier Laurent · Maxence Leguéry · Andrei Bursuc · Andrea Pilzer · Angela Yao,https://ensta-u2is-ai.github.io/ABNN-Make-me-a-BNN/,https://arxiv.org/abs/2312.15297,,2312.15297.pdf,Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models,"Deep Neural Networks (DNNs) are powerful tools for various computer vision
+tasks, yet they often struggle with reliable uncertainty quantification - a
+critical requirement for real-world applications. Bayesian Neural Networks
+(BNN) are equipped for uncertainty estimation but cannot scale to large DNNs
+that are highly unstable to train. To address this challenge, we introduce the
+Adaptable Bayesian Neural Network (ABNN), a simple and scalable strategy to
+seamlessly transform DNNs into BNNs in a post-hoc manner with minimal
+computational and training overheads. ABNN preserves the main predictive
+properties of DNNs while enhancing their uncertainty quantification abilities
+through simple BNN adaptation layers (attached to normalization layers) and a
+few fine-tuning steps on pre-trained models. We conduct extensive experiments
+across multiple datasets for image classification and semantic segmentation
+tasks, and our results demonstrate that ABNN achieves state-of-the-art
+performance without the computational budget typically associated with ensemble
+methods.",cs.LG,"['cs.LG', 'cs.CV', 'stat.ML']"
+OpenBias: Open-set Bias Detection in Text-to-Image Generative Models,Moreno D&#x27;Incà · Elia Peruzzo · Massimiliano Mancini · Dejia Xu · Vidit Goel · Xingqian Xu · Zhangyang Wang · Humphrey Shi · Nicu Sebe,https://github.com/Picsart-AI-Research/OpenBias,https://arxiv.org/abs/2404.07990v1,,2404.07990v1.pdf,OpenBias: Open-set Bias Detection in Text-to-Image Generative Models,"Text-to-image generative models are becoming increasingly popular and
+accessible to the general public. As these models see large-scale deployments,
+it is necessary to deeply investigate their safety and fairness to not
+disseminate and perpetuate any kind of biases. However, existing works focus on
+detecting closed sets of biases defined a priori, limiting the studies to
+well-known concepts. In this paper, we tackle the challenge of open-set bias
+detection in text-to-image generative models presenting OpenBias, a new
+pipeline that identifies and quantifies the severity of biases agnostically,
+without access to any precompiled set. OpenBias has three stages. In the first
+phase, we leverage a Large Language Model (LLM) to propose biases given a set
+of captions. Secondly, the target generative model produces images using the
+same set of captions. Lastly, a Vision Question Answering model recognizes the
+presence and extent of the previously proposed biases. We study the behavior of
+Stable Diffusion 1.5, 2, and XL emphasizing new biases, never investigated
+before. Via quantitative experiments, we demonstrate that OpenBias agrees with
+current closed-set bias detection methods and human judgement.",cs.CV,"['cs.CV', 'cs.AI']"
+Depth-Aware Concealed Crop Detection in Dense Agricultural Scenes,Liqiong Wang · Jinyu Yang · Yanfu Zhang · Fangyi Wang · Feng Zheng,https://github.com/Kki2Eve/RISNet,,https://www.mdpi.com/1424-8220/24/6/1942,,,,,nan
+GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement,Linfang Zheng · Tze Ho Elden Tse · Chen Wang · Yinghan Sun · Hua Chen · Aleš Leonardis · Wei Zhang · Hyung Jin Chang,https://lynne-zheng-linfang.github.io/georef.github.io/,https://arxiv.org/abs/2404.11139v1,,2404.11139v1.pdf,GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement,"Object pose refinement is essential for robust object pose estimation.
+Previous work has made significant progress towards instance-level object pose
+refinement. Yet, category-level pose refinement is a more challenging problem
+due to large shape variations within a category and the discrepancies between
+the target object and the shape prior. To address these challenges, we
+introduce a novel architecture for category-level object pose refinement. Our
+approach integrates an HS-layer and learnable affine transformations, which
+aims to enhance the extraction and alignment of geometric information.
+Additionally, we introduce a cross-cloud transformation mechanism that
+efficiently merges diverse data sources. Finally, we push the limits of our
+model by incorporating the shape prior information for translation and size
+error prediction. We conducted extensive experiments to demonstrate the
+effectiveness of the proposed framework. Through extensive quantitative
+experiments, we demonstrate significant improvement over the baseline method by
+a large margin across all metrics.",cs.CV,['cs.CV']
+Learning to Control Camera Exposure via Reinforcement Learning,Kyunghyun Lee · Ukcheol Shin · Byeong-Uk Lee,https://sites.google.com/view/drl-ae,https://arxiv.org/abs/2404.01636,,2404.01636.pdf,Learning to Control Camera Exposure via Reinforcement Learning,"Adjusting camera exposure in arbitrary lighting conditions is the first step
+to ensure the functionality of computer vision applications. Poorly adjusted
+camera exposure often leads to critical failure and performance degradation.
+Traditional camera exposure control methods require multiple convergence steps
+and time-consuming processes, making them unsuitable for dynamic lighting
+conditions. In this paper, we propose a new camera exposure control framework
+that rapidly controls camera exposure while performing real-time processing by
+exploiting deep reinforcement learning. The proposed framework consists of four
+contributions: 1) a simplified training ground to simulate real-world's diverse
+and dynamic lighting changes, 2) flickering and image attribute-aware reward
+design, along with lightweight state design for real-time processing, 3) a
+static-to-dynamic lighting curriculum to gradually improve the agent's
+exposure-adjusting capability, and 4) domain randomization techniques to
+alleviate the limitation of the training ground and achieve seamless
+generalization in the wild.As a result, our proposed method rapidly reaches a
+desired exposure level within five steps with real-time processing (1 ms).
+Also, the acquired images are well-exposed and show superiority in various
+computer vision tasks, such as feature extraction and object detection.",cs.CV,"['cs.CV', 'cs.AI', 'cs.LG', 'cs.RO', 'cs.SY', 'eess.SY']"
+Differentiable Neural Surface Refinement for Transparent Objects,Weijian Deng · Dylan Campbell · Chunyi Sun · Shubham Kanitkar · Matthew Shaffer · Stephen Gould,https://weijiandeng.xyz/nsr,,https://dl.acm.org/doi/abs/10.1145/3610548.3618236,,,,,nan
+Discovering and Mitigating Visual Biases through Keyword Explanation,Younghyun Kim · Sangwoo Mo · Minkyu Kim · Kyungmin Lee · Jaeho Lee · Jinwoo Shin, ,,https://effl.postech.ac.kr/docs/research/papers/,,,,,nan
+MiKASA: Multi-Key-Anchor Scene-Aware Transformer for 3D Visual Grounding,Chun-Peng Chang · Shaoxiang Wang · Alain Pagani · Didier Stricker, ,https://arxiv.org/abs/2403.03077,,,MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding,"3D visual grounding involves matching natural language descriptions with
+their corresponding objects in 3D spaces. Existing methods often face
+challenges with accuracy in object recognition and struggle in interpreting
+complex linguistic queries, particularly with descriptions that involve
+multiple anchors or are view-dependent. In response, we present the MiKASA
+(Multi-Key-Anchor Scene-Aware) Transformer. Our novel end-to-end trained model
+integrates a self-attention-based scene-aware object encoder and an original
+multi-key-anchor technique, enhancing object recognition accuracy and the
+understanding of spatial relationships. Furthermore, MiKASA improves the
+explainability of decision-making, facilitating error diagnosis. Our model
+achieves the highest overall accuracy in the Referit3D challenge for both the
+Sr3D and Nr3D datasets, particularly excelling by a large margin in categories
+that require viewpoint-dependent descriptions.",cs.CV,['cs.CV']
+Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3),Tsu-Ching Hsiao · Hao-Wei Chen · Hsuan-Kung Yang · Chun-Yi Lee, ,https://arxiv.org/abs/2401.00029,,,6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation,"Estimating the 6D object pose from a single RGB image often involves noise
+and indeterminacy due to challenges such as occlusions and cluttered
+backgrounds. Meanwhile, diffusion models have shown appealing performance in
+generating high-quality images from random noise with high indeterminacy
+through step-by-step denoising. Inspired by their denoising capability, we
+propose a novel diffusion-based framework (6D-Diff) to handle the noise and
+indeterminacy in object pose estimation for better performance. In our
+framework, to establish accurate 2D-3D correspondence, we formulate 2D
+keypoints detection as a reverse diffusion (denoising) process. To facilitate
+such a denoising process, we design a Mixture-of-Cauchy-based forward diffusion
+process and condition the reverse process on the object features. Extensive
+experiments on the LM-O and YCB-V datasets demonstrate the effectiveness of our
+framework.",cs.CV,['cs.CV']
+Text-guided Explorable Image Super-resolution,Kanchana Vaishnavi Gandikota · Paramanand Chandramouli, ,https://arxiv.org/abs/2403.01124,,2403.01124.pdf,Text-guided Explorable Image Super-resolution,"In this paper, we introduce the problem of zero-shot text-guided exploration
+of the solutions to open-domain image super-resolution. Our goal is to allow
+users to explore diverse, semantically accurate reconstructions that preserve
+data consistency with the low-resolution inputs for different large
+downsampling factors without explicitly training for these specific
+degradations. We propose two approaches for zero-shot text-guided
+super-resolution - i) modifying the generative process of text-to-image
+\textit{T2I} diffusion models to promote consistency with low-resolution
+inputs, and ii) incorporating language guidance into zero-shot diffusion-based
+restoration methods. We show that the proposed approaches result in diverse
+solutions that match the semantic meaning provided by the text prompt while
+preserving data consistency with the degraded inputs. We evaluate the proposed
+baselines for the task of extreme super-resolution and demonstrate advantages
+in terms of restoration quality, diversity, and explorability of solutions.",cs.CV,['cs.CV']
+$CrowdDiff$: Multi-hypothesis Crowd Density Estimation using Diffusion Models,Yasiru Ranasinghe · Nithin Gopalakrishnan Nair · Wele Gedara Chaminda Bandara · Vishal M. Patel, ,,https://jarxiv.com/2024/04/05/crowddiff-multi-hypothesis-crowd-density-estimation-using-diffusion-models/,,,,,nan
+Instruct-Imagen: Image Generation with Multi-modal Instruction,Hexiang Hu · Kelvin C.K. Chan · Yu-Chuan Su · Wenhu Chen · Yandong Li · Kihyuk Sohn · Yang Zhao · Xue Ben · William Cohen · Ming-Wei Chang · Xuhui Jia,https://instruct-imagen.github.io/,https://arxiv.org/abs/2401.01952,,2401.01952.pdf,Instruct-Imagen: Image Generation with Multi-modal Instruction,"This paper presents instruct-imagen, a model that tackles heterogeneous image
+generation tasks and generalizes across unseen tasks. We introduce *multi-modal
+instruction* for image generation, a task representation articulating a range
+of generation intents with precision. It uses natural language to amalgamate
+disparate modalities (e.g., text, edge, style, subject, etc.), such that
+abundant generation intents can be standardized in a uniform format.
+  We then build instruct-imagen by fine-tuning a pre-trained text-to-image
+diffusion model with a two-stage framework. First, we adapt the model using the
+retrieval-augmented training, to enhance model's capabilities to ground its
+generation on external multimodal context. Subsequently, we fine-tune the
+adapted model on diverse image generation tasks that requires vision-language
+understanding (e.g., subject-driven generation, etc.), each paired with a
+multi-modal instruction encapsulating the task's essence. Human evaluation on
+various image generation datasets reveals that instruct-imagen matches or
+surpasses prior task-specific models in-domain and demonstrates promising
+generalization to unseen and more complex tasks.",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL']"
+$MonoDiff$: Monocular 3D Object Detection and Pose Estimation with Diffusion Models,Yasiru Ranasinghe · Deepti Hegde · Vishal M. Patel, ,https://arxiv.org/abs/2403.18791,,,Object Pose Estimation via the Aggregation of Diffusion Features,"Estimating the pose of objects from images is a crucial task of 3D scene
+understanding, and recent approaches have shown promising results on very large
+benchmarks. However, these methods experience a significant performance drop
+when dealing with unseen objects. We believe that it results from the limited
+generalizability of image features. To address this problem, we have an
+in-depth analysis on the features of diffusion models, e.g. Stable Diffusion,
+which hold substantial potential for modeling unseen objects. Based on this
+analysis, we then innovatively introduce these diffusion features for object
+pose estimation. To achieve this, we propose three distinct architectures that
+can effectively capture and aggregate diffusion features of different
+granularity, greatly improving the generalizability of object pose estimation.
+Our approach outperforms the state-of-the-art methods by a considerable margin
+on three popular benchmark datasets, LM, O-LM, and T-LESS. In particular, our
+method achieves higher accuracy than the previous best arts on unseen objects:
+98.2% vs. 93.5% on Unseen LM, 85.9% vs. 76.3% on Unseen O-LM, showing the
+strong generalizability of our method. Our code is released at
+https://github.com/Tianfu18/diff-feats-pose.",cs.CV,['cs.CV']
+Towards More Unified In-context Visual Understanding,Dianmo Sheng · Dongdong Chen · Zhentao Tan · Qiankun Liu · Qi Chu · Jianmin Bao · Tao Gong · Bin Liu · Shengwei Xu · Nenghai Yu, ,https://arxiv.org/abs/2312.02520v2,,2312.02520v2.pdf,Towards More Unified In-context Visual Understanding,"The rapid advancement of large language models (LLMs) has accelerated the
+emergence of in-context learning (ICL) as a cutting-edge approach in the
+natural language processing domain. Recently, ICL has been employed in visual
+understanding tasks, such as semantic segmentation and image captioning,
+yielding promising results. However, existing visual ICL framework can not
+enable producing content across multiple modalities, which limits their
+potential usage scenarios. To address this issue, we present a new ICL
+framework for visual understanding with multi-modal output enabled. First, we
+quantize and embed both text and visual prompt into a unified representational
+space, structured as interleaved in-context sequences. Then a decoder-only
+sparse transformer architecture is employed to perform generative modeling on
+them, facilitating in-context learning. Thanks to this design, the model is
+capable of handling in-context vision understanding tasks with multimodal
+output in a unified pipeline.Experimental results demonstrate that our model
+achieves competitive performance compared with specialized models and previous
+ICL baselines. Overall, our research takes a further step toward unified
+multimodal in-context learning.",cs.CV,['cs.CV']
+Compositional Chain-of-Thought Prompting for Large Multimodal Models,Chancharik Mitra · Brandon Huang · Trevor Darrell · Roei Herzig, ,https://arxiv.org/abs/2311.17076,,2311.17076.pdf,Compositional Chain-of-Thought Prompting for Large Multimodal Models,"The combination of strong visual backbones and Large Language Model (LLM)
+reasoning has led to Large Multimodal Models (LMMs) becoming the current
+standard for a wide range of vision and language (VL) tasks. However, recent
+research has shown that even the most advanced LMMs still struggle to capture
+aspects of compositional visual reasoning, such as attributes and relationships
+between objects. One solution is to utilize scene graphs (SGs)--a formalization
+of objects and their relations and attributes that has been extensively used as
+a bridge between the visual and textual domains. Yet, scene graph data requires
+scene graph annotations, which are expensive to collect and thus not easily
+scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic
+forgetting of the pretraining objective. To overcome this, inspired by
+chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a
+novel zero-shot Chain-of-Thought prompting method that utilizes SG
+representations in order to extract compositional knowledge from an LMM.
+Specifically, we first generate an SG using the LMM, and then use that SG in
+the prompt to produce a response. Through extensive experiments, we find that
+the proposed CCoT approach not only improves LMM performance on several vision
+and language VL compositional benchmarks but also improves the performance of
+several popular LMMs on general multimodal benchmarks, without the need for
+fine-tuning or annotated ground-truth SGs. Code:
+https://github.com/chancharikmitra/CCoT",cs.CV,"['cs.CV', 'cs.AI', 'cs.CL', 'cs.LG']"
+"CAMixerSR: Only Details Need More ""Attention""",Yan Wang · Yi Liu · Shijie Zhao · Junlin Li · Li zhang,https://github.com/icandle/CAMixerSR,https://arxiv.org/abs/2402.19289v2,,2402.19289v2.pdf,"CAMixerSR: Only Details Need More ""Attention""","To satisfy the rapidly increasing demands on the large image (2K-8K)
+super-resolution (SR), prevailing methods follow two independent tracks: 1)
+accelerate existing networks by content-aware routing, and 2) design better
+super-resolution networks via token mixer refining. Despite directness, they
+encounter unavoidable defects (e.g., inflexible route or non-discriminative
+processing) limiting further improvements of quality-complexity trade-off. To
+erase the drawbacks, we integrate these schemes by proposing a content-aware
+mixer (CAMixer), which assigns convolution for simple contexts and additional
+deformable window-attention for sparse textures. Specifically, the CAMixer uses
+a learnable predictor to generate multiple bootstraps, including offsets for
+windows warping, a mask for classifying windows, and convolutional attentions
+for endowing convolution with the dynamic property, which modulates attention
+to include more useful textures self-adaptively and improves the representation
+capability of convolution. We further introduce a global classification loss to
+improve the accuracy of predictors. By simply stacking CAMixers, we obtain
+CAMixerSR which achieves superior performance on large-image SR, lightweight
+SR, and omnidirectional-image SR.",eess.IV,"['eess.IV', 'cs.CV']"
+Geometrically-informed aggregation for zero-shot point cloud understanding,Guofeng Mei · Luigi Riz · Yiming Wang · Fabio Poiesi, ,https://arxiv.org/abs/2312.02244,,2312.02244.pdf,Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding,"Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language
+Models (VLMs). Existing strategies directly map Vision-Language Models from 2D
+pixels of rendered or captured views to 3D points, overlooking the inherent and
+expressible point cloud geometric structure. Geometrically similar or close
+regions can be exploited for bolstering point cloud understanding as they are
+likely to share semantic information. To this end, we introduce the first
+training-free aggregation technique that leverages the point cloud's 3D
+geometric structure to improve the quality of the transferred Vision-Language
+Models. Our approach operates iteratively, performing local-to-global
+aggregation based on geometric and semantic point-level reasoning. We benchmark
+our approach on three downstream tasks, including classification, part
+segmentation, and semantic segmentation, with a variety of datasets
+representing both synthetic/real-world, and indoor/outdoor scenarios. Our
+approach achieves new state-of-the-art results in all benchmarks. Our approach
+operates iteratively, performing local-to-global aggregation based on geometric
+and semantic point-level reasoning. Code and dataset are available at
+https://luigiriz.github.io/geoze-website/",cs.CV,['cs.CV']
+CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection,JiaBao Wang · yuming chen · Zhaohui Zheng · Xiang Li · Ming-Ming Cheng · Qibin Hou,https://github.com/jbwang1997/CrossKD,https://arxiv.org/abs/2306.11369,,2306.11369.pdf,CrossKD: Cross-Head Knowledge Distillation for Object Detection,"Knowledge Distillation (KD) has been validated as an effective model
+compression technique for learning compact object detectors. Existing
+state-of-the-art KD methods for object detection are mostly based on feature
+imitation. In this paper, we present a general and effective prediction
+mimicking distillation scheme, called CrossKD, which delivers the intermediate
+features of the student's detection head to the teacher's detection head. The
+resulting cross-head predictions are then forced to mimic the teacher's
+predictions. This manner relieves the student's head from receiving
+contradictory supervision signals from the annotations and the teacher's
+predictions, greatly improving the student's detection performance. Moreover,
+as mimicking the teacher's predictions is the target of KD, CrossKD offers more
+task-oriented information in contrast with feature imitation. On MS COCO, with
+only prediction mimicking losses applied, our CrossKD boosts the average
+precision of GFL ResNet-50 with 1x training schedule from 40.2 to 43.7,
+outperforming all existing KD methods. In addition, our method also works well
+when distilling detectors with heterogeneous backbones. Code is available at
+https://github.com/jbwang1997/CrossKD.",cs.CV,['cs.CV']
+DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos,Arjun Balasingam · Joseph Chandler · Chenning Li · Zhoutong Zhang · Hari Balakrishnan,https://drivetrack.csail.mit.edu/,https://arxiv.org/abs/2312.09523,,2312.09523.pdf,DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos,"This paper presents DriveTrack, a new benchmark and data generation framework
+for long-range keypoint tracking in real-world videos. DriveTrack is motivated
+by the observation that the accuracy of state-of-the-art trackers depends
+strongly on visual attributes around the selected keypoints, such as texture
+and lighting. The problem is that these artifacts are especially pronounced in
+real-world videos, but these trackers are unable to train on such scenes due to
+a dearth of annotations. DriveTrack bridges this gap by building a framework to
+automatically annotate point tracks on autonomous driving datasets. We release
+a dataset consisting of 1 billion point tracks across 24 hours of video, which
+is seven orders of magnitude greater than prior real-world benchmarks and on
+par with the scale of synthetic benchmarks. DriveTrack unlocks new use cases
+for point tracking in real-world videos. First, we show that fine-tuning
+keypoint trackers on DriveTrack improves accuracy on real-world scenes by up to
+7%. Second, we analyze the sensitivity of trackers to visual artifacts in real
+scenes and motivate the idea of running assistive keypoint selectors alongside
+trackers.",cs.CV,['cs.CV']
+CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model,Jianhao Zeng · Dan Song · Weizhi Nie · Hongshuo Tian · Tongtong Wang · An-An Liu,https://zengjianhao.github.io/CAT-DM,https://arxiv.org/abs/2311.18405,,2311.18405.pdf,CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model,"Generative Adversarial Networks (GANs) dominate the research field in
+image-based virtual try-on, but have not resolved problems such as unnatural
+deformation of garments and the blurry generation quality. While the generative
+quality of diffusion models is impressive, achieving controllability poses a
+significant challenge when applying it to virtual try-on and multiple denoising
+iterations limit its potential for real-time applications. In this paper, we
+propose Controllable Accelerated virtual Try-on with Diffusion Model (CAT-DM).
+To enhance the controllability, a basic diffusion-based virtual try-on network
+is designed, which utilizes ControlNet to introduce additional control
+conditions and improves the feature extraction of garment images. In terms of
+acceleration, CAT-DM initiates a reverse denoising process with an implicit
+distribution generated by a pre-trained GAN-based model. Compared with previous
+try-on methods based on diffusion models, CAT-DM not only retains the pattern
+and texture details of the inshop garment but also reduces the sampling steps
+without compromising generation quality. Extensive experiments demonstrate the
+superiority of CAT-DM against both GANbased and diffusion-based methods in
+producing more realistic images and accurately reproducing garment patterns.",cs.CV,['cs.CV']
+Free3D: Consistent Novel View Synthesis without 3D Representation,Chuanxia Zheng · Andrea Vedaldi,https://chuanxiaz.com/free3d/,https://arxiv.org/abs/2312.04551,,2312.04551.pdf,Free3D: Consistent Novel View Synthesis without 3D Representation,"We introduce Free3D, a simple accurate method for monocular open-set novel
+view synthesis (NVS). Similar to Zero-1-to-3, we start from a pre-trained 2D
+image generator for generalization, and fine-tune it for NVS. Compared to other
+works that took a similar approach, we obtain significant improvements without
+resorting to an explicit 3D representation, which is slow and memory-consuming,
+and without training an additional network for 3D reconstruction. Our key
+contribution is to improve the way the target camera pose is encoded in the
+network, which we do by introducing a new ray conditioning normalization (RCN)
+layer. The latter injects pose information in the underlying 2D image generator
+by telling each pixel its viewing direction. We further improve multi-view
+consistency by using light-weight multi-view attention layers and by sharing
+generation noise between the different views. We train Free3D on the Objaverse
+dataset and demonstrate excellent generalization to new categories in new
+datasets, including OmniObject3D and GSO. The project page is available at
+https://chuanxiaz.com/free3d/.",cs.CV,['cs.CV']
+InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields,Dongqing Wang · Tong Zhang · Alaa Abboud · Sabine Süsstrunk, ,https://arxiv.org/html/2401.05335v1,,2401.05335v1.pdf,InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes,"We introduce InseRF, a novel method for generative object insertion in the
+NeRF reconstructions of 3D scenes. Based on a user-provided textual description
+and a 2D bounding box in a reference viewpoint, InseRF generates new objects in
+3D scenes. Recently, methods for 3D scene editing have been profoundly
+transformed, owing to the use of strong priors of text-to-image diffusion
+models in 3D generative modeling. Existing methods are mostly effective in
+editing 3D scenes via style and appearance changes or removing existing
+objects. Generating new objects, however, remains a challenge for such methods,
+which we address in this study. Specifically, we propose grounding the 3D
+object insertion to a 2D object insertion in a reference view of the scene. The
+2D edit is then lifted to 3D using a single-view object reconstruction method.
+The reconstructed object is then inserted into the scene, guided by the priors
+of monocular depth estimation methods. We evaluate our method on various 3D
+scenes and provide an in-depth analysis of the proposed components. Our
+experiments with generative insertion of objects in several 3D scenes indicate
+the effectiveness of our method compared to the existing methods. InseRF is
+capable of controllable and 3D-consistent object insertion without requiring
+explicit 3D information as input. Please visit our project page at
+https://mohamad-shahbazi.github.io/inserf.",cs.CV,"['cs.CV', 'cs.GR', 'cs.LG']"
+Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation,Fahimeh Hosseini Noohdani · Parsa Hosseini · Aryan Yazdan Parast · Hamidreza Araghi · Mahdieh Baghshah, ,https://arxiv.org/abs/2402.18919,,2402.18919.pdf,Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation,"While standard Empirical Risk Minimization (ERM) training is proven effective
+for image classification on in-distribution data, it fails to perform well on
+out-of-distribution samples. One of the main sources of distribution shift for
+image classification is the compositional nature of images. Specifically, in
+addition to the main object or component(s) determining the label, some other
+image components usually exist, which may lead to the shift of input
+distribution between train and test environments. More importantly, these
+components may have spurious correlations with the label. To address this
+issue, we propose Decompose-and-Compose (DaC), which improves robustness to
+correlation shift by a compositional approach based on combining elements of
+images. Based on our observations, models trained with ERM usually highly
+attend to either the causal components or the components having a high spurious
+correlation with the label (especially in datapoints on which models have a
+high confidence). In fact, according to the amount of spurious correlation and
+the easiness of classification based on the causal or non-causal components,
+the model usually attends to one of these more (on samples with high
+confidence). Following this, we first try to identify the causal components of
+images using class activation maps of models trained with ERM. Afterward, we
+intervene on images by combining them and retraining the model on the augmented
+data, including the counterfactual ones. Along with its high interpretability,
+this work proposes a group-balancing method by intervening on images without
+requiring group labels or information regarding the spurious features during
+training. The method has an overall better worst group accuracy compared to
+previous methods with the same amount of supervision on the group labels in
+correlation shift.",cs.CV,"['cs.CV', 'cs.LG']"