ChatGPT for Autodesk
An AI assistant built specifically for Autodesk products (AutoCAD, Revit, Fusion 360, Maya, Civil 3D) that understands engineering and design domains, processes multi-modal inputs, executes operations through Autodesk APIs, and complies with industry standards like BIM, ISO, and building codes.
Problem Statement
"Design an AI assistant for Autodesk's product suite that can understand text, images, and 3D model context, retrieve domain-specific engineering documentation, and execute design operations through product APIs. How would you architect a system that serves 50K+ concurrent users across multiple Autodesk products?"
Clarifying Questions to Ask
- Which Autodesk products are in scope for V1? All products at once or a phased rollout starting with one (e.g., Revit)?
- Assistant vs. autonomous designer? Should it suggest changes requiring human approval or autonomously make design decisions?
- Enterprise vs. individual? Is this for enterprise teams with data residency requirements or individual subscribers?
- What multi-modal inputs are expected? Text only, or also screenshots, annotated images, and 3D model context from the viewport?
- Which engineering standards must it know? BIM, ISO, ASHRAE, IBC, local building codes, or all of the above?
- What is the latency expectation? Is sub-3-second first-token acceptable, or does the user expect instant responses?
- How should it handle destructive operations? Always preview and confirm, or allow auto-apply for low-risk changes?
- Collaboration requirements? Should AI suggestions be shareable with team members in real time?
Requirements
Functional Requirements
- Multi-modal input -- accept text queries, annotated screenshots, 2D drawings, and 3D model references
- Domain-specific RAG -- retrieve information from Autodesk documentation, engineering standards (BIM, ISO, ASHRAE), and building codes
- Plugin architecture -- extensible plugins for each Autodesk product with product-specific actions
- Natural language to design operations -- translate user intent into Autodesk API calls (draw, modify, dimension, annotate)
- Version control for designs -- track changes made by the assistant, support undo/redo, maintain change history
- Collaboration -- share AI-generated suggestions with team members, support multi-user sessions
- Rendering preview -- generate visual previews of proposed changes before applying them
- Standards compliance -- check designs against applicable building codes, material specifications, and industry standards
Non-Functional Requirements
| Requirement | Target |
|---|---|
| Response latency (text) | < 3s for first token |
| Response latency (with preview) | < 10s including render |
| API operation accuracy | > 95% of generated operations are valid |
| Standards compliance recall | > 90% of applicable violations detected |
| Concurrent users | 50K+ simultaneous sessions |
| Availability | 99.9% uptime during business hours |
| Data residency | Support regional deployment for enterprise customers |
Out of Scope
- Fully autonomous design generation (V1 is assistant, not autonomous designer)
- Real-time collaborative editing (handled by Autodesk's existing systems)
- Physical simulation (structural, thermal) -- delegated to existing simulation tools
High-Level Architecture
Architecture Walkthrough
Requests flow from product-specific client plugins (AutoCAD, Revit, Fusion 360, Maya) or a web dashboard through an API gateway that handles authentication, rate limiting, and routing. The gateway supports both REST and WebSocket connections for real-time streaming responses.
The Input Processing layer runs three parallel processors -- text, image, and 3D model context -- and fuses their outputs into a unified representation. This fused context passes to the AI Core, where an agent orchestrator delegates to a domain router. The domain router selects the appropriate specialized agent: a design agent for operations, a compliance agent for standards checking, or an explain agent for documentation queries.
All agents access a RAG Pipeline that performs hybrid retrieval (semantic + keyword) over Autodesk documentation, engineering standards, and building codes stored in a vector database.
The Execution Layer translates agent decisions into product-specific API calls through adapter plugins, and a preview renderer generates before/after visuals on a headless GPU farm. All sessions, change logs, model snapshots, and rendered previews are persisted across PostgreSQL, object storage, and a dedicated change log store.
Component Design
Multi-Modal Input Processing
The system must fuse information from text, images, and 3D model context into a unified understanding. The Text Processor performs intent classification and entity extraction from the user's natural language query. The Image Processor uses a vision model to analyze screenshots and annotated drawings, identifying the active Autodesk product, visible elements (walls, beams, components), annotations, and the current view context (plan view, 3D perspective, section). The 3D Model Processor extracts geometric information from the user's active model -- selected elements with their properties, spatial relationships, and a geometry summary.
A Multi-Modal Fuser then combines these three modality-specific representations into a single unified context object that downstream agents can reason over. This late-fusion approach allows each modality to use a specialized model optimized for its input type while still enabling cross-modal reasoning.
Domain Router and Plugin Architecture
Each Autodesk product has its own API surface, terminology, and workflow patterns. The Domain Router detects which product the user is working in, classifies the user's intent (design operation, compliance check, explanation, troubleshooting), and routes to the appropriate handler.
Each product has a dedicated Plugin that exposes product-specific tools. For example, the Revit Plugin exposes tools like create wall, place family instance, modify parameter, run clash detection, and export schedule. When the design agent decides on an operation, the plugin translates it into product-specific API calls and executes them within a transaction that supports atomic undo. If any step fails, the entire transaction rolls back, preventing the model from entering a broken state.
| Approach | Pros | Cons |
|---|---|---|
| Product-specific plugins | Independent versioning, specialized tools, isolated failures | More code to maintain, need shared abstractions |
| Single unified agent | Simpler codebase, shared logic | Cannot capture product-specific nuances, single point of failure |
Domain-Specific RAG Pipeline
Engineering documentation requires specialized chunking and retrieval. Building codes are chunked by section, preserving the full section hierarchy and cross-references so that a section like "IBC 2021 Section 1607.1" remains self-contained. Technical standards are chunked by clause. Autodesk help documents are chunked by topic page.
Retrieval uses a hybrid strategy: semantic search over a vector database for conceptual queries, plus keyword search for exact code references (users often search for specific section numbers like "ASHRAE 90.1-2019"). Results are merged and re-ranked using a cross-encoder reranker to produce the top-10 most relevant chunks.
Rendering Preview Generation
Before applying changes, the system generates a visual preview. The renderer clones the current model state, applies the proposed operations to the clone, and renders before/after views from the most relevant camera angle. It also generates a diff overlay highlighting exactly what changed. The user sees all three views and can approve, reject, or modify the proposed changes. The clone is discarded after preview generation to avoid storage waste.
Data Flow
The sequence diagram above shows the full lifecycle of a design modification. Every change follows the pattern: parse intent, validate, checkpoint, preview, confirm, execute, commit. This ensures that every AI-initiated modification is reversible and auditable. The checkpoint-based version control allows users to undo to any point in the session history, not just the most recent change.
Scaling Considerations
| Component | Strategy |
|---|---|
| Input processing | GPU-backed vision model pool; pre-process images at edge |
| RAG retrieval | Partitioned vector DB by product; cached frequent queries |
| Plugin execution | One plugin instance per active session; pooled connections to Autodesk APIs |
| Preview rendering | Headless GPU render farm; pre-warmed containers |
| Model storage | Tiered: hot (SSD) for active sessions, warm (object store) for recent, cold (archive) |
Cost Analysis
| Component | Cost/Session | Notes |
|---|---|---|
| LLM inference | $0.05-0.20 | Depends on query complexity |
| Vision processing | $0.01-0.05 | Only when images provided |
| RAG retrieval | $0.001 | Cached embeddings |
| Preview rendering | $0.02-0.10 | GPU render time |
| Infrastructure | $0.01 | Amortized |
| Total | $0.09-0.37 | Per session (avg 5-10 interactions) |
Autodesk products have high per-seat costs ($2K-$5K/year). An AI assistant adding $50-100/year to the cost is easily justified if it saves even 30 minutes per week of engineering time.
Trade-offs & Alternatives
| Decision | Rationale | Alternative | Why Not |
|---|---|---|---|
| Product-specific plugins | Each product has unique APIs and workflows; plugins enable independent versioning and specialized tooling | Single unified agent for all products | Cannot capture product-specific nuances; a Revit BIM workflow differs fundamentally from a Maya animation workflow |
| Server-side headless rendering | Consistent rendering quality; client may lack GPU; enables web dashboard previews | Client-side rendering within the Autodesk product | Not all clients have GPUs; web dashboard would not work; inconsistent results across machines |
| Separate RAG pipeline for standards | Standards are large and version-specific; RAG enables updates without redeployment of the AI model | Embedding standards directly in system prompts | Context window limits; cannot version or update standards independently; poor recall on large corpora |
| Transaction-wrapped API execution | Must support atomic undo; partial changes leave models in broken states that are hard to recover from | Direct API calls without transactions | A failed multi-step operation could leave the model in an inconsistent state with no way to roll back |
| Late fusion for multi-modal inputs | Allows specialized models per modality; easier to debug, test, and improve each modality independently | Early fusion with a single multi-modal model | Harder to debug; single model cannot be best-in-class for all modalities; tighter coupling |
Interview Tips
:::tip How to Present This (35 minutes)
- Minutes 1-5: Clarify requirements -- which Autodesk products, assistant vs. autonomous, enterprise vs. individual, multi-modal scope
- Minutes 5-15: Draw the architecture diagram -- client plugins, API gateway, input processing with multi-modal fusion, AI core with domain routing, RAG pipeline, execution layer with product adapters
- Minutes 15-25: Deep dive into plugin architecture (why per-product plugins), domain-specific RAG (specialized chunking for building codes), and the preview-confirm-execute workflow
- Minutes 25-30: Scaling (GPU render farm, partitioned vector DB, tiered storage) and cost analysis ($0.09-0.37 per session, easily justified against $2K-5K seat costs)
- Minutes 30-35: Trade-offs (late vs. early fusion, server vs. client rendering, transaction-wrapped execution) and security considerations (IP protection for proprietary designs, data residency, access control) :::