positions

DC5: Task-Agnostic Multimodal Foundation Models on Edge Devices

Task: Task-Agnostic Multimodal Foundation Models on Edge Devices (WP2)

Host institution: UNITN

Country: Italy

Supervisor: Dr. E. Ricci [UNITN]

Co-supervisors: Prof. J. Yang [TU Delft]; Prof. P. Casari [UNITN]; Dr. F. Widdershoven [NXP]

Objectives: 1) To develop approaches for pruning and quantization of state-of-the-art open-source large multimodal foundation models imposing specific requirements for achieving a trade-off between model accuracy and computing complexity; 2) To design novel techniques
for Task-Agnostic Pruning (TAP), aiming to overcome the limitations of existing multimodal pruning techniques which are task-specific that require network pruning from scratch for each new task of interest; 3) To evaluate the proposed techniques on relevant multimodal benchmarks and on visual classification tasks.

Expected Results: 1) Novel approaches for compressing multimodal architectures based on pruning and quantization techniques; 2) An approach for task-agnostic pruning, which will increase the generalization capabilities of existing multimodal models; 3) Proof-of-concept experiments showing reduction of computational complexity.

PhD enrolment: Doctoral School of UNITN

Planned secondments: 

  • TU Delft (4 months, M16-M19): Joint development of a method for model pruning based on structured pruning techniques, with Prof. J. Yang (KPI: joint paper)

  • NXP (3 months, M28-M30): Proof-of-concept of developed compressed multimodal models for visual classification tasks, with Dr. F. Widdershoven (KPI: joint paper)

Candidate profile: computer science, electrical engineering, telecommunication engineering, applied mathematics and related fields

Desirable skills/interests: deep and machine learning, multimodal learning, computer vision, signal processing, statistical filtering, applied optimization (the applicant should be proficient in at least one or two of the skills)

Application Deadline: February 14, 2025, AoE

Descriptions of all the 18 DC Positions

Submit Your Application HERE