This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments-Rule Comprehension, Rule Compliance, and Rule Extraction-based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models like GPT4 and LLaVA against the benchmark, and our study uncovers the existing gaps in MLLMs’ abilities to interpret complex engineering documentation. Key findings suggest that while MLLMs demonstrate potential in navigating technical documents, substantial limitations exist, particularly in accurately extracting and applying detailed requirements to engineering designs. This benchmark sets a foundation for future advancements in AI-supported engineering design processes.
@article{doris2025designqa,title={Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation},author={Doris, Anna C and Grandi, Daniele and Tomich, Ryan and Alam, Md Ferdous and Ataei, Mohammadmehdi and Cheong, Hyunmin and Ahmed, Faez},journal={Journal of Computing and Information Science in Engineering},volume={25},number={2},pages={021009},year={2025},publisher={American Society of Mechanical Engineers}}
Material selection is a crucial step in conceptual design due to its significant impact on the functionality, aesthetics, manufacturability, and sustainability impact of the final product. This study investigates the use of Large Language Models (LLMs) for material selection in the product design process and compares the performance of LLMs against expert choices for various design scenarios. By collecting a dataset of expert material preferences, the study provides a basis for evaluating how well LLMs can align with expert recommendations through prompt engineering and hyperparameter tuning. The divergence between LLM and expert recommendations is measured across different model configurations, prompt strategies, and temperature settings. This approach allows for a detailed analysis of factors influencing the LLMs’ effectiveness in recommending materials. The results from this study highlight two failure modes, and identify parallel prompting as a useful prompt-engineering method when using LLMs for material selection. The findings further suggest that, while LLMs can provide valuable assistance, their recommendations often vary significantly from those of human experts. This discrepancy underscores the need for further research into how LLMs can be better tailored to replicate expert decision-making in material selection. This work contributes to the growing body of knowledge on how LLMs can be integrated into the design process, offering insights into their current limitations and potential for future improvements.
@article{grandi2025evaluating,title={Evaluating large language models for material selection},author={Grandi, Daniele and Jain, Yash Patawari and Groom, Allin and Cramer, Brandon and McComb, Christopher},journal={Journal of Computing and Information Science in Engineering},volume={25},number={2},pages={021004},year={2025},publisher={American Society of Mechanical Engineers}}
@article{patawari2025mseval,title={MSEval: A Dataset for Material Selection in Conceptual Design to Evaluate Algorithmic Models},author={Patawari Jain, Yash and Grandi, Daniele and Groom, Allin and Cramer, Brandon and McComb, Christopher},journal={Journal of Mechanical Design},volume={147},number={4},year={2025},publisher={American Society of Mechanical Engineers Digital Collection}}
Requirements elicitation, a critical, yet time-consuming and challenging step in product development, often fails to capture the full spectrum of user needs. This may lead to products that fall short of expectations. This paper introduces a novel framework that leverages Large Language Models (LLMs) to automate and enhance the requirements elicitation process. LLMs are used to generate a vast array of simulated users (LLM agents), enabling the exploration of a much broader range of user needs and unforeseen use cases. These agents engage in product experience scenarios, through explaining their actions, observations, and challenges. Subsequent agent interviews and analysis uncover valuable user needs, including latent ones. We validate our framework with three experiments. First, we explore different methodologies for diverse agent generation, discussing their advantages and shortcomings. We measure the diversity of identified user needs and demonstrate that context-aware agent generation leads to greater diversity. Second, we show how our framework effectively mimics empathic lead user interviews, identifying a greater number of latent needs than conventional human interviews. Third, we showcase that LLMs can be used to analyze interviews, capture needs, and classify them as latent or not. Our work highlights the potential of using LLM agents to accelerate early-stage product development, reduce costs, and increase innovation.
@article{ataei2025elicitron,title={Elicitron: A Large Language Model Agent-Based Simulation Framework for Design Requirements Elicitation},author={Ataei, Mohammadmehdi and Cheong, Hyunmin and Grandi, Daniele and Wang, Ye and Morris, Nigel and Tessier, Alexander},journal={Journal of Computing and Information Science in Engineering},volume={25},number={2},year={2025},publisher={American Society of Mechanical Engineers Digital Collection}}
@article{ma2025large,title={Do Large Language Models Produce Diverse Design Concepts? A Comparative Study with Human-Crowdsourced Solutions},author={Ma, Kevin and Grandi, Daniele and McComb, Christopher and Goucher-Lambert, Kosa},journal={Journal of Computing and Information Science in Engineering},volume={25},number={2},year={2025},publisher={American Society of Mechanical Engineers Digital Collection}}
@article{bolanos2025recall,title={RECALL-MM: A Multimodal Dataset of Consumer Product Recalls for Risk Analysis using Computational Methods and Large Language Models},author={Bolanos, Diana and Ataei, Mohammadmehdi and Grandi, Daniele and Goucher-Lambert, Kosa},journal={arXiv preprint arXiv:2503.23213},year={2025}}
@article{bandara2024macrostructure,title={Macrostructure topology generation with disparate physical simulation for Computer Aided Design and Manufacturing},author={Bandara, Konara Mudiyanselage Kosala and Willis, Karl Darcy Daniel and Harris, Andrew John and Banadyha, Andriy and Grandi, Daniele and Butscher, Adrian Adam Thomas and Bastian, Andreas Linas and Shayani, Hooman},year={2024},month=apr,note={US Patent 11,947,334}}
This work investigates the value in the natural language part and document names users provide when they create CAD models. In a first step towards multi-modal text-CAD learning, our results show that Large Language Models are able to leverage the noisy text data to predict part-part and part-whole relationships, with direct applications in automations and recommendations for part re-use, auto-complete, assembly categorizations, smart tool suggestions and library part recommendations.
@article{meltzer2024s,title={What’s in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models Through User-Provided Names in Computer Aided Design Files},author={Meltzer, Peter and Lambourne, Joseph G and Grandi, Daniele},journal={Journal of Computing and Information Science in Engineering},volume={24},number={1},pages={011002},year={2024},publisher={American Society of Mechanical Engineers}}
To support intelligent computer-aided design (CAD), we introduce a machine learning architecture, namely HG-CAD, that recommends assembly body material through joint learning of body and assembly-level features using a hierarchical graph representation. Specifically, we formulate the material prediction and recommendation process as a node-level classification task over a novel hierarchical graph representation of CAD models, with a low-level graph capturing the body geometry, a high-level graph representing the assembly topology, and a batch-level mask randomization enabling contextual awareness. This enables our network to aggregate geometric and topological features from both the body and assembly levels, leading to competitive performance. Qualitative and quantitative evaluation of the proposed architecture on the Fusion 360 Gallery Assembly Dataset demonstrates the feasibility of our approach, outperforming selected computer vision and human baselines while showing promise in application scenarios. The proposed HG-CAD architecture that unifies the processing, encoding, and joint learning of multi-modal CAD features indicates the potential to serve as a recommendation system for design automation and a baseline for future work.
@article{bian2024hg,title={HG-CAD: hierarchical graph learning for material prediction and recommendation in computer-aided design},author={Bian, Shijie and Grandi, Daniele and Liu, Tianyang and Jayaraman, Pradeep Kumar and Willis, Karl and Sadler, Elliot and Borijin, Bodia and Lu, Thomas and Otis, Richard and Ho, Nhut and others},journal={Journal of Computing and Information Science in Engineering},volume={24},number={1},pages={011007},year={2024},publisher={American Society of Mechanical Engineers}}
Requirements elicitation, a critical, yet time-consuming and challenging step in product development, often fails to capture the full spectrum of user needs. This may lead to products that fall short of expectations. This paper introduces a novel framework that leverages Large Language Models (LLMs) to automate and enhance the requirements elicitation process. LLMs are used to generate a vast array of simulated users (LLM agents), enabling the exploration of a much broader range of user needs and unforeseen use cases. These agents engage in product experience scenarios, through explaining their actions, observations, and challenges. Subsequent agent interviews and analysis uncover valuable user needs, including latent ones. We validate our framework with three experiments. First, we explore different methodologies for diverse agent generation, discussing their advantages and shortcomings. We measure the diversity of identified user needs and demonstrate that context-aware agent generation leads to greater diversity. Second, we show how our framework effectively mimics empathic lead user interviews, identifying a greater number of latent needs than conventional human interviews. Third, we showcase that LLMs can be used to analyze interviews, capture needs, and classify them as latent or not. Our work highlights the potential of using LLM agents to accelerate early-stage product development, reduce costs, and increase innovation.
@article{ataei2024elicitron,title={Elicitron: An LLM agent-based simulation framework for design requirements elicitation},author={Ataei, Mohammadmehdi and Cheong, Hyunmin and Grandi, Daniele and Wang, Ye and Morris, Nigel and Tessier, Alexander},journal={arXiv preprint arXiv:2404.16045},year={2024}}
@article{ma2024exploring,title={Exploring the capabilities of large language models for generating diverse design solutions},author={Ma, Kevin and Grandi, Daniele and McComb, Christopher and Goucher-Lambert, Kosa},journal={arXiv preprint arXiv:2405.02345},year={2024}}
@inproceedings{doris2024designqa,title={DesignQA: Benchmarking Multimodal Large Language Models on Questions Grounded in Engineering Documentation},author={Doris, Anna C and Grandi, Daniele and Tomich, Ryan and Alam, Md Ferdous and Cheong, Hyunmin and Ahmed, Faez},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={88360},pages={V03AT03A016},year={2024},organization={American Society of Mechanical Engineers}}
@inproceedings{ataei2024elicitroo,title={Elicitron: A Framework for Simulating Design Requirements Elicitation Using Large Language Model Agents},author={Ataei, Mohammadmehdi and Cheong, Hyunmin and Grandi, Daniele and Wang, Ye and Morris, Nigel and Tessier, Alexander},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={88377},pages={V03BT03A056},year={2024},organization={American Society of Mechanical Engineers}}
@inproceedings{patawari2024material,title={Material Selection Using Large Language Models},author={Patawari Jain, Yash and Grandi, Daniele and Groom, Allin and Cramer, Brandon and McComb, Christopher},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={88377},pages={V03BT03A058},year={2024},organization={American Society of Mechanical Engineers}}
Knowledge collection, extraction, and organization are critical activities in all aspects of the engineering design process. However, it remains challenging to surface and organize design knowledge, which often contains implicit or tacit dimensions that are difficult to capture in a scalable and accessible manner. Knowledge graphs (KGs) have been explored to address this issue, but have been primarily semantic in nature in engineering design contexts, typically focusing on sharing explicit knowledge. Our work seeks to understand knowledge organization during an experiential activity and how it can be transformed into a scalable representation. To explore this, we examine 23 professional designers’ knowledge organization practices as they virtually engage with data collected during a teardown of a consumer product. Using this data, we develop a searchable knowledge graph as a mechanism for representing the experiential knowledge and afford its use in complex queries. We demonstrate the knowledge graph with two extended examples to reveal insights and patterns from design knowledge. These findings provide insight into professional designers’ knowledge organization practices and represent a preliminary step toward design knowledge bases that more accurately reflect designer behavior, ultimately enabling more effective data-driven support tools for design.
@article{wang2023embedding,title={Embedding Experiential Design Knowledge in Interactive Knowledge Graphs},author={Wang, Ye and Goridkov, Nicole and Rao, Vivek and Cui, Dixun and Grandi, Daniele and Goucher-Lambert, Kosa},journal={Journal of Mechanical Design},volume={145},number={4},pages={041412},year={2023},publisher={American Society of Mechanical Engineers}}
@article{meltzer2023s,title={What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files},author={Meltzer, Peter and Lambourne, Joseph G and Grandi, Daniele},journal={arXiv preprint arXiv:2304.14275},year={2023}}
Concept generation is a creative step in the conceptual design phase, where designers often turn to brainstorming, mindmapping, or crowdsourcing design ideas to complement their own knowledge of the domain. Recent advances in natural language processing (NLP) and machine learning (ML) have led to the rise of Large Language Models (LLMs) capable of generating seemingly creative outputs from textual prompts. The success of these models has led to their integration and application across a variety of domains, including art, entertainment, and other creative work. In this paper, we leverage LLMs to generate solutions for a set of 12 design problems and compare them to a baseline of crowdsourced solutions. We evaluate the differences between generated and crowdsourced design solutions through multiple perspectives, including human expert evaluations and computational metrics. Expert evaluations indicate that the LLM-generated solutions have higher average feasibility and usefulness while the crowdsourced solutions have more novelty. We experiment with prompt engineering and find that leveraging few-shot learning can lead to the generation of solutions that are more similar to the crowdsourced solutions. These findings provide insight into the quality of design solutions generated with LLMs and begins to evaluate prompt engineering techniques that could be leveraged by practitioners to generate higher-quality design solutions synergistically with LLMs.
@inproceedings{ma2023conceptual,title={Conceptual design generation using large language models},author={Ma, Kevin and Grandi, Daniele and McComb, Christopher and Goucher-Lambert, Kosa},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={87349},pages={V006T06A021},year={2023},organization={American Society of Mechanical Engineers}}
@misc{willis2023conversion,title={Conversion of geometry to boundary representation with facilitated editing for computer aided design and 2.5-axis subtractive manufacturing},author={Willis, Karl Darcy Daniel and Morris, Nigel Jed Wesley and Bastian, Andreas Linas and Butscher, Adrian Adam Thomas and Grandi, Daniele and Furuta, Suguru and Lambourne, Joseph George and Barback, Tristan Ward and Marinov, Martin Cvetanov and Amagliani, Marco and others},year={2023},month=feb,note={US Patent App. 17/945,008}}
Graph Neural Networks
HG-CAD: Hierarchical Graph Learning for Material Prediction and Recommendation in CAD
Shijie Bian, Daniele Grandi, Tianyang Liu, and 8 more authors
@article{bian2023hg,title={HG-CAD: Hierarchical Graph Learning for Material Prediction and Recommendation in CAD},author={Bian, Shijie and Grandi, Daniele and Liu, Tianyang and Jayaraman, Pradeep Kumar and DD, Karl and Willis, Elliot Sadler and Borijin, Bodia and Lu, Thomas and Otis, Richard and Ho, Nhut and others},year={2023}}
Successful material selection is critical in designing and manufacturing products for design automation. Designers leverage their knowledge and experience to create high-quality designs by selecting the most appropriate materials through performance, manufacturability, and sustainability evaluation. Intelligent tools can help designers with varying expertise by providing recommendations learned from prior designs. To enable this, we introduce a graph representation learning framework that supports the material prediction of bodies in assemblies. We formulate the material selection task as a node-level prediction task over the assembly graph representation of CAD models and tackle it using Graph Neural Networks (GNNs). Evaluations over three experimental protocols performed on the Fusion 360 Gallery dataset indicate the feasibility of our approach, achieving a 0.75 top-3 micro-F1 score. The proposed framework can scale to large datasets and incorporate designers’ knowledge into the learning process. These capabilities allow the framework to serve as a recommendation system for design automation and a baseline for future work, narrowing the gap between human designers and intelligent design agents.
@inproceedings{bian2022material,title={Material prediction for design automation using graph representation learning},author={Bian, Shijie and Grandi, Daniele and Hassani, Kaveh and Sadler, Elliot and Borijin, Bodia and Fernandes, Axel and Wang, Andrew and Lu, Thomas and Otis, Richard and Ho, Nhut and others},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={86229},pages={V03AT03A001},year={2022},organization={American Society of Mechanical Engineers}}
@article{ferrero2022classifying,title={Classifying component function in product assemblies with graph neural networks},author={Ferrero, Vincenzo and DuPont, Bryony and Hassani, Kaveh and Grandi, Daniele},journal={Journal of Mechanical Design},volume={144},number={2},pages={021406},year={2022},publisher={American Society of Mechanical Engineers}}
@inproceedings{wang2022a3d,title={A3d: Studying pretrained representations with programmable datasets},author={Wang, Ye and Mu, Norman and Grandi, Daniele and Savva, Nicolas and Steinhardt, Jacob},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},pages={4878--4889},year={2022}}
Physical products are often complex assemblies combining a multitude of 3D parts modeled in computer-aided design (CAD) software. CAD designers build up these assemblies by aligning individual parts to one another using constraints called joints. In this paper we introduce JoinABLe, a learning-based method that assembles parts together to form joints. JoinABLe uses the weak supervision available in standard parametric CAD files without the help of object class labels or human guidance. Our results show that by making network predictions over a graph representation of solid models we can outperform multiple baseline methods with an accuracy (79.53%) that approaches human performance (80%). Finally, to support future research we release the Fusion 360 Gallery assembly dataset, containing assemblies with rich information on joints, contact surfaces, holes, and the underlying assembly graph structure.
@inproceedings{willis2022joinable,title={Joinable: Learning bottom-up assembly of parametric cad joints},author={Willis, Karl DD and Jayaraman, Pradeep Kumar and Chu, Hang and Tian, Yunsheng and Li, Yifei and Grandi, Daniele and Sanghi, Aditya and Tran, Linh and Lambourne, Joseph G and Solar-Lezama, Armando and others},booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},pages={15849--15860},year={2022}}
@book{willis2022conversion,title={Conversion of geometry to boundary representation with facilitated editing for computer aided design and 2.5-axis subtractive manufacturing},author={Willis, Karl Darcy Daniel and Morris, Nigel Jed Wesley and Bastian, Andreas Linas and Butscher, Adrian Adam Thomas and Grandi, Daniele and Furuta, Suguru and Lambourne, Joseph George and Barback, Tristan Ward and Marinov, Martin Cvetanov and Amagliani, Marco and others},year={2022},month=sep,note={US Patent 11,455,435}}
Knowledge collection, extraction, and organization are critical activities in all aspects of the engineering design process. However, it remains challenging to surface and organize design knowledge in a scalable and accessible manner given it often contains implicit or tacit dimensions that are difficult to capture. Knowledge graphs have been explored to address this issue but have been primarily semantic in nature in engineering design contexts, typically focusing on sharing explicit knowledge. In this work, we explore how knowledge graphs could offer a mechanism to organize experiential design knowledge and afford its use in complex queries. We develop a searchable knowledge graph based on data from a previous virtual product teardown activity with 23 professional designers. Examples of the underlying data within this corpus include descriptions of product components and their purpose as well as participant-determined relationships between these components. To structure the knowledge graph, we develop a schema that uses its constituent nodes and edges to represent design knowledge, relational information, and properties such as the node author’s discipline and the node’s function-behavior-structure classification. We propose and demonstrate two user-driven graph search types – intentional and exploratory – and four data-driven graph search methods, and illustrate through two extended examples their potential to reveal insights and patterns from teardown knowledge. These findings suggest that knowledge graphs can be a valuable approach to organizing and availing experiential design knowledge emerging from complex design activities.
@inproceedings{goridkov2022capturing,title={Capturing designers’ experiential knowledge in scalable representation systems: a case study of knowledge graphs for product teardowns},author={Goridkov, Nicole and Rao, Vivek and Cui, Dixun and Grandi, Daniele and Wang, Ye and Goucher-Lambert, Kosa},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={86267},pages={V006T06A032},year={2022},organization={American Society of Mechanical Engineers}}
@inproceedings{wang2021understanding,title={Understanding professional designers’ knowledge organization behavior: A case study in product teardowns},author={Wang, Ye and Grandi, Daniele and Cui, Dixun and Rao, Vivek and Goucher-Lambert, Kosa},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={85420},pages={V006T06A046},year={2021},organization={American Society of Mechanical Engineers}}
2016
Generative Design
Embedded sensors and feedback loops for iterative improvement in design synthesis for additive manufacturing
Mehdi Nourbakhsh, Nigel Morris, Michael Bergin, and 2 more authors
In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Sep 2016
@inproceedings{nourbakhsh2016embedded,title={Embedded sensors and feedback loops for iterative improvement in design synthesis for additive manufacturing},author={Nourbakhsh, Mehdi and Morris, Nigel and Bergin, Michael and Iorio, Francesco and Grandi, Daniele},booktitle={International Design Engineering Technical Conferences and Computers and Information in Engineering Conference},volume={50077},pages={V01AT02A031},year={2016},organization={American Society of Mechanical Engineers}}
