Talks / Proposal

Intelligence Design Studio

A proposal for an extensible IDE that fuses coding with the media-rich aspects of machine learning workflows — covering environment authoring, dataset curation, experiment provisioning, workload assignment, and extensibility via a unified platform.

Presenter Patrick Hebron
Date
Context Proposal for AI IDE
1/1

Slide Transcript

  1. Intelligence Design Studio Patrick Hebron patrick@patrickhebron.com
  2. Mission Create a tool that supports the entire research pipeline... An extensible IDE that fuses coding with the more media-rich aspects of machine learning workflows. Intelligence Design Studio | Patrick Hebron
  3. Motivations ML brings software further into the visual, auditory and physical world. Existing dev tools relegate simulation environments, multimedia assets, visualization tools, etc to auxiliary libraries and interfaces. In the ML age, they must be brought to the very center of our tools. Intelligence Design Studio | Patrick Hebron
  4. env = deepmind_lab.Lab(level_script, config, num_episodes) env.reset() observation_spec = env.observation_spec() action_spec = env.action_spec() obs = env.observations() rgb_i = obs['RGB_INTERLEAVED'] action = np.zeros([7], dtype=np.intc) action[3] = 1 • Bridging independent tools reduces workflow efficiency and creates technical bottlenecks. • Repurposing tools (e.g. from game dev to simulation building) limits research opportunities and explorations. Intelligence Design Studio | Patrick Hebron
  5. Unified Tool Goals Coding Environment Authoring Dataset Curation & Augmentation Experiment Provisioning Monitoring Workload Assignment Versioning Extensions & Personalization • Bring all aspects of R&D process into one dev environment. • Focus functionality around ML-specific workflows and use cases. • Close the gap between programming, ML and media authoring tools. Intelligence Design Studio | Patrick Hebron
  6. Multimedia Input Multimedia Intermediate Multimedia Output Multimedia Output-as-Input • Bring code editor together with a powerful design tool that offers: • Dataset curation and augmentation interfaces. • Visualization tools for monitoring and analysis. • Prototyping tools for building ML-enhanced apps and experiences. • Scenograph, GUI and simulation authoring tools with extensive I/O support for agent interactions. Intelligence Design Studio | Patrick Hebron
  7. Environment & Synthetic Dataset Authoring Simulation Environments Synthetic Data • Provide visual and programmatic interfaces for creating and controlling 3D environments. • Focus design tool functionality on authoring use cases for simulation (RL) and synthetic data (SL). Intelligence Design Studio | Patrick Hebron
  8. Primitives Mesh Loaders Scene Layout Physics Rigging Scenegraph I/O • Provide basic scene modeling and layout functionality (and extend over time). • Provide extensive physics, rigging and scenegraph I/O functionality. Intelligence Design Studio | Patrick Hebron
  9. Structural & Behavioral Variation Structures Behaviors Scenarios Rendering Modes Basic Photorealistic Medical Engineering • Provide mechanisms for conditional and parametric variation of scene structure, behaviors and scenarios. • Provide a wide range of rendering styles and controls for color, grain, camera properties and artifacts. • Enable users to share 3D models, rigs, environments and scenarios via extensibility platform. Intelligence Design Studio | Patrick Hebron
  10. Real Dataset Curation Curriculum Curator 875 images Daisy blooming wild 943 images Dandelion bright 984 images Orchid wild 779 images Rose garden red 802 images Sunflower blooming garden 861 images Tulip bulb • Provide visual and programmatic interfaces for dataset curation. • Leverage clustering and embedding models to aid user-directed curation process. Intelligence Design Studio | Patrick Hebron
  11. Real Dataset Augmentation Time-of-Day Variation -5 hrs +5 hrs Camera Artifacts Camera Move Time-of-Day Season • Provide visual and programmatic interfaces for dataset augmentation. • Leverage GAN and style transfer models as well as conventional crop & distort to aid user-directed augmentation process. Intelligence Design Studio | Patrick Hebron
  12. Experiment Provisioning Parallel Experimenter Learning Rate Min: 0.010 Max: 0.100 Inc: 0.001 Momentum Min: 0.500 Max: 0.900 Inc: 0.010 Dropout Rate Min: 0.200 Max: 0.500 Inc: 0.010 Rain Min: 0.000 Max: 1.000 Inc: 0.100 Snow Min: 0.000 Max: 0.250 Inc: 0.100 Fog Min: 0.000 Max: 0.750 Inc: 0.100 Ray Bounces Min: 2.000 Max: 9.000 Inc: 1.000 Anti-Aliasing Level Min: 0.000 Max: 5.000 Inc: 1.000 Optimization Technique: Random Grid Bayesian Evolutionary Curriculum Arc Creator • Provide visual and programmatic interfaces for parallel experiment provisioning, population based training, curriculum arcs and versioning. Intelligence Design Studio | Patrick Hebron
  13. Workload Assignment Interactive Development Studio Local Training Runtime Cloud Training Runtime Deploy to Embedded System Deploy to Native Application Deploy to Web App / Service • Provide visual and programmatic interfaces for workload assignment. • Provide seamless abstraction of local and remote compute resources. Intelligence Design Studio | Patrick Hebron
  14. Extensibility biggan-deep-128 By DeepMind i3d-kinetics-600 By DeepMind biggan-512 By DeepMind - Create centralized home for libraries, models, environments, scenarios and datasets. - Provide drag-and-drop code blocks in IDE for pre-built model loaders, architectural and data wrangling components. Intelligence Design Studio | Patrick Hebron
  15. Directives Configure 'Driving Simulation Environment' Time-of-Day Variation: 12 AM 12 PM 12 AM Rain Variation: None Moderate Heavy Snow Variation: None Moderate Heavy Fog Variation: None Moderate Heavy Cancel Create Simulation • Provide visual and programmatic interfaces for authoring directives - custom widgets that: • Guide users through a set of decision points (e.g. a curation or configuration process). • Serve as custom dashboards for monitoring, live events, etc. • Provide simple mechanism for sharing directives via extensibility platform. Intelligence Design Studio | Patrick Hebron
  16. Prior Work Foil FoilBuck Foil Intelligence Demo What's next? var imagePath = askForImageDirective(); imageView = Object("ImageView"); imageView->setFile(imagePath); imageView->setDimensionFromInternal(); var output = classifyImageFile(imagePath); print(imageView); print("Your image contains: " + output[2] + " (confidence: " + String(output[1]) * 100.0 + "%)"); Submit clear output • Foil is a combined ML and creative coding platform, toolkit and IDE. • It attempts to bring ML development, multimedia asset wrangling and ML-enhanced-application development into a single tool. • It is highly customizable. The IDE is built on top of Foil's own user-facing GUI and scenegraph toolkits. Intelligence Design Studio | Patrick Hebron
  17. Foil is a native application written in C++ by a single author (me). It is built on top of: CINDER boost BULLET PHYSICS LIBRARY TensorFlow Eigen3 It runs on: macOS Linux Windows Intelligence Design Studio | Patrick Hebron
  18. Foil Features • Multi-threaded runtime controller. • Dynamically-typed, JIT compiled programming language. • Extensive GUI, 2D/3D scenegraph and data visualization toolkit. • Extensive linear algebra and machine learning toolkit. • Native IDE built with Foil's own GUI toolkit. • Novel language feature aimed at facilitating interactive ML development. Intelligence Design Studio | Patrick Hebron
  19. GUI Scenegraph Data Visualization var i = 0; var count = 100; var dataX = zeros(count, 1); var dataY = zeros(count, 2); while ((i < count)) { dataX[i, 0] = (i * ((2 * pi()) / (count - 1))); dataY[i, 0] = sin(dataX[i, 0]); dataY[i, 1] = cos(dataX[i, 0]); i = (i + 1); } plot(dataY, dataX); Intelligence Design Studio | Patrick Hebron
  20. Proposal Create an extensible IDE that fuses coding with the more media-rich aspects of machine learning workflows... • Build on top of Foil's runtime, GUI and scenegraph libraries. • Provide deep integration with DeepMind's tools and workflows. • Focus on building an end-to-end solution for ongoing research work. • Extend functionality to support emerging research directions. Intelligence Design Studio | Patrick Hebron
  21. Process Observe Observe current practices across numerous teams, projects and roles. Interview Interview researchers and other stakeholders about workflows and needs. Plan Create roadmap for tool functionality in coordination with relevant teams. Design Design interface mock-ups and perform preliminary user testing. Build Implement tool, perform user testing and iterate. Intelligence Design Studio | Patrick Hebron
  22. Thanks! Intelligence Design Studio | Patrick Hebron