Rafi Ibn Sultan

I build vision–language systems that segment, ground, and reason about the world.

Ph.D. Candidate, Wayne State University Detroit, Michigan rafis@wayne.edu

On the academic job market. I'm seeking tenure-track Assistant Professor and Research Scientist positions, starting after I complete my Ph.D. in August 2027. Get in touch — I'd be glad to talk.

I'm a Ph.D. candidate in Computer Science at Wayne State University, working at the intersection of computer vision and multimodal AI. Since joining the Trustworthy AI Lab in 2022, I've worked on vision–language models and segmentation foundation models across medical imaging, mobility infrastructure, and remote sensing.

What I care about most is AI for social good — systems that don't just describe a scene, but reason about it well enough to support real decisions.

Curriculum Vitae Google Scholar GitHub
Research

What I work on

Grounded Multimodal Reasoning for Accessibility

My most recent work, WalkGPT (CVPR 2026), is a grounded vision–language model for pedestrian navigation and accessibility. It combines segmentation, depth estimation, and language reasoning to identify sidewalks, crosswalks, curb ramps, and accessibility barriers from real-world pedestrian-view imagery, then delivers step-by-step, context-aware guidance as a conversational agent.

Earlier in this line, GeoSAM (ECAI 2025) introduced sparse- and dense-prompt fine-tuning of SAM for large-scale mobility-infrastructure segmentation from aerial and street-level imagery. It has since been adopted as a benchmark in subsequent NeurIPS work.

I'm currently extending this toward stronger spatial reasoning in vision–language models at scale. SpatialCORE (under review) targets spatially confident reasoning in large VLMs — pushing past surface-level description toward predictions that are grounded in visual evidence rather than asserted.

Medical Image Segmentation & Multimodal Learning

I develop foundation-model-based segmentation and VLM-driven fusion methods for CT and MRI analysis. My current work in this line, MedPlex, explores deep vision–language co-adaptation for clinically grounded medical segmentation — building on BiPVL-Seg, which introduced bidirectional progressive alignment of visual features with structured medical text to improve organ and tumor segmentation.

collaborated with Henry Ford Health, I work on Left Anterior Descending (LAD) artery segmentation using novel encoder–decoder architectures — important for radiotherapy planning, since the LAD is highly sensitive to radiation injury. That work was presented at AAPM 2025 and is forthcoming in Medical Physics.

Technical Skills

Programming & Systems
  • Python
  • PyTorch
  • Linux
  • Git
  • LaTeX
  • Multi-GPU training
Models & Methods
  • Vision transformers
  • SAM
  • Large vision–language models
  • LoRA / PEFT
  • Contrastive learning
  • Segmentation
  • Grounding
  • Representation learning
Tools & Libraries
  • HuggingFace
  • MONAI
  • OpenCV
  • scikit-learn
  • Weights & Biases
Publications

Selected work

  • CVPR 2026
    WalkGPT: Grounded Vision–Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation
    Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu
    WalkGPT framework: a SAM pixel encoder and decoder feed a large language model with LoRA, producing a segmentation mask and a natural-language accessibility assessment. Side panels detail the Multi-Scale Query Projector and the Calibrated Text Projector with region alignment loss.
  • ECAI 2025
    GeoSAM: Fine-tuning SAM with Multi-Modal Prompts for Mobility Infrastructure Segmentation
    Rafi Ibn Sultan, Chengyin Li, Hui Zhu, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu
    arXiv
    GeoSAM pipeline: aerial imagery plus GPT-4 text prompts drive a frozen image encoder and prompt encoder into a tuned mask decoder that predicts infrastructure masks. A second panel shows geo-point generation, extracting foreground and background points from pseudo labels for road and pedestrian infrastructure.
  • Medical Physics 2026
    NA-UNETR: A Neighborhood Attention Transformer Network for Enhanced 3D Segmentation of the Left Anterior Descending Artery
    Rafi Ibn Sultan, et al. — in collaboration with Henry Ford Hospital
    NA-UNETR architecture: a U-shaped encoder–decoder for 3D CT volumes. The encoder runs four stages of overlapping tokenizer and downsampler blocks paired with Res-conv NAT blocks; skip connections carry features through ResBlocks to a decoder that concatenates and deconvolves back to full resolution, ending in a head that outputs the segmented left anterior descending artery. A side panel details the NAT block as layer norm, neighborhood attention or dilated neighborhood attention, and MLP with residual connections.
  • BMVC 2026
    MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
    Rafi Ibn Sultan, et al.
    MedPlex training framework: a vision encoder and decoder alongside a text encoder, joined by bidirectional fusion blocks, aligning visual and text embedding spaces through class-level and region-level concept-grounded contrastive losses against a segmentation map and ground truth.

All Peer-Reviewed Publications

Preprints & Under Review

Full list on Google Scholar.

Talks & Service

Talks, coverage & recognition

Invited Talk
WalkGPT: Pixel-Grounded Navigation Guidance for Pedestrians
Best of CVPR 2026 Series, Voxel51 · July 2026 (virtual)
Invited Talk
WalkGPT: Grounded Vision–Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation
Computer Vision Group, Cohere Labs Open Science Community · May 2026 (virtual)
Invited Talk
Multimodal AI for Mobility Infrastructure and Pedestrian Guidance
Departmental Seminar, Wayne State University · January 2026
Press
Detroit PBS: Wayne State Creates AI for Mobility Project
Feature on our lab and GeoSAM · April 2024
 Conference Talk
AutoProSAM presentation at WACV 2025
Tucson, Arizona · February 2025

Awards & Honors

  • Outstanding Graduate Research Assistant Award 2025
    Department of Computer Science, Wayne State University · Presented April 2026

Conference Reviewing

  • AAAI 2027
  • NeurIPS 2026
  • IJCAI-ECAI 2026
  • BMVC 2026
  • IJCNN 2025
  • ECCV

Journal Reviewing

News

Recent updates

  • Aug 7, 2026 Our paper "MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation" has been accepted to the British Machine Vision Conference (BMVC 2026), taking place 23–26 November 2026 in Lancaster, UK.
  • Aug 4, 2026 Our paper with Henry Ford Hospital, "A Neighborhood Attention Transformer Network for Enhanced 3D Segmentation of the Left Anterior Descending Artery," has been accepted to Medical Physics.
  • Apr 16, 2026 Received the Outstanding Graduate Research Assistant Award for 2025 from the Department of Computer Science at Wayne State University.
  • Feb 20, 2026 WalkGPT has been accepted to CVPR 2026.
  • Jan 27, 2026 I was invited as a guest graduate speaker in the Computer Science Department at Wayne State University, presenting "From Mobility Infrastructure Segmentation to Pedestrian Guidance: A Multimodal AI Perspective."
  • Jul 11, 2025 GeoSAM has been accepted to the 28th European Conference on Artificial Intelligence (ECAI 2025) in Bologna, Italy.
Experience

Where I've worked

  • Graduate Research Assistant Sep 2023 — Present
    Trustworthy AI Lab, Department of Computer Science, Wayne State University · Detroit, MI
    • Lead independent research on grounded vision–language models for spatial reasoning, pedestrian navigation, and medical image segmentation — from problem formulation through publication, resulting in first-author papers at CVPR and ECAI.
    • Drive an ongoing clinical collaboration with Henry Ford Hospital on coronary CT segmentation, translating clinician requirements into model design and working directly with radiology partners on data, evaluation, and deployment-relevant validation.
    • Mentor junior Ph.D. students on research methodology, experimental design, and paper writing.
  • Graduate Teaching Assistant Aug 2022 — Aug 2023
    Department of Computer Science, Wayne State University · Detroit, MI
    • Led Software Engineering labs, supervising 5–6 project teams of 4–5 students each per term through the full development lifecycle, from requirements to delivery.
  • Lecturer Oct 2019 — Aug 2022
    Department of Computer Science and Engineering, Varendra University · Rajshahi, Bangladesh
    • Sole instructor for undergraduate courses including Microprocessor and Assembly Language, Object-Oriented Programming, and Computer Fundamentals — 30–40 students per course.
    • Supervised roughly 8 undergraduate final-year thesis and capstone projects from proposal to defense.
Education

Education

  • Ph.D. in Computer Science Sep 2022 — Expected Aug 2027
    Wayne State University · Detroit, MI
    Advisor: Prof. Dongxiao Zhu
  • M.Sc. in Computer Science Dec 2025
    Wayne State University · Detroit, MI
  • B.Sc. in Computer Science & Engineering 2014 — 2018
    Rajshahi University of Engineering & Technology (RUET) · Rajshahi, Bangladesh
  • Higher Secondary School Certificate 2013
    Rajshahi College
  • Secondary School Certificate 2011
    Shiroil Government High School
Beyond Research

Outside the lab

  • Football — a lifelong Real Madrid loyalist, and I'll defend Cristiano Ronaldo in any argument
  • Anime — always mid-series, always open to recommendations
  • Camping — weekends under canvas whenever the season allows
  • Hiking — trails are where I do my best thinking
  • Gaming — FIFA since '98, a Killjoy main in Valorant, and new to CS2
  • Movies & series — a reliable way to lose an evening
  • Beginner acoustic guitarist
  • Travel — working toward all 50 states

Conferences & Lab Life

Standing beside the WalkGPT poster at CVPR, which lays out the framework, the PAVE dataset construction, and qualitative navigation results.
WalkGPT at CVPR 2026Denver, Colorado
Members of the Trustworthy AI Lab gathered around a table of Detroit-style square pizza, raising their glasses toward the camera.
Pizza with the labDetroit, Michigan
Holding a Wayne State University research excellence certificate alongside two fellow award recipients and a faculty member.
Outstanding Graduate Research Assistant AwardWayne State University · 2025
Standing beside the AutoProSAM poster at WACV 2025, showing the architecture components and multi-organ segmentation results.
AutoProSAM at WACV 2025Tucson, Arizona
The lab group seated along a long table at a Detroit taproom, raising glasses for a Thanksgiving gathering.
Thanksgiving with the labDetroit, Michigan
The Trustworthy AI Lab gathered for a group photo in a backyard on the Fourth of July.
Fourth of July at Prof. Zhu'sTrustworthy AI Lab