From 595c5404caced7a2714f8d67c1d1802930fd4c8f Mon Sep 17 00:00:00 2001 From: LiCHO Date: Tue, 2 Apr 2024 15:59:04 -0400 Subject: [PATCH] test video --- README.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 7fedbdc..85f95d3 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,13 @@ In the realm of object manipulation, human engagement typically manifests throug We introduce **ActAIM2**, which given an RGBD image of an articulated object and a robot, identifies meaningful interaction modes like opening drawer and closing drawer. ActAIM2 represents the interaction modes as discrete clusters of embedding. ActAIM2 then trains a policy that takes cluster embedding as input and produces control actions for the corresponding interactions. -teaser_3 - +![Alt text for the image](Pictures/teaser.png) ## Problem Formulation Our model training aims to uncover the policy's distribution $\mathbb{P}(a|o)$, with $o$ representing the observation and $a=(\mathbf{p}, \mathbf{R}, \mathbf{q})$ the action, through a decomposition strategy that reconfigures the action distribution as: -![problem_formulation](https://github.com/pairlab/actaim2-eccv24/assets/30140814/1f01df77-8c06-449f-91ba-7bddcb034c98) +![Alt text for the image](Pictures/problem_formulation.png) ## Data Generation @@ -21,11 +20,11 @@ Our dataset was constructed through a combination of random sampling, heuristic The figure below indicates how we achieve diverse interaction mode sampling by using GMM-based adaptive sampling. -![data_collection_gmm](https://github.com/pairlab/actaim2-eccv24/assets/30140814/976e31a2-1986-4506-a50a-bd764eb430a8) +![Alt text for the image](Pictures/data_generation.png) We also formulate our data collection algorithm here. -![data_collection_algo](https://github.com/pairlab/actaim2-eccv24/assets/30140814/3032710a-ac79-400e-99c9-8d23ea881806) +![Alt text for the image](Pictures/data_generation_algo.png) ## Unsupervised Mode Selector Learning @@ -33,21 +32,20 @@ In this part, we show how we train and infer from the mode selector to extract t ### Mode Selector Training Process -mode_selector_training +![Alt text for the image](Pictures/mode_train.png) This figure illustrates the training procedure of the mode selector, mirroring the approach of a conditional generative model. It highlights the contrastive analysis between the initial and final observations—the latter serving as the ground truth for task embedding—to delineate generated data against the backdrop of encoded initial images as the conditional variable. The process involves inputting both the generated task embedding data and the conditional variable into a 4-layer Residual network-based mode encoder, which then predicts the categorical variable $c$. Following the Gaussian Mixture Variational Autoencoder (GMVAE) methodology, the Gaussian Mixture Model (GMM) variable $x$ is computed and introduced alongside the conditional variable to the task embedding transformer decoder. This model is tasked with predicting the reconstructed task embedding, sampled from the Gaussian distribution as outlined in the architecture of the mode selector decoder and calculating the reconstruction loss against the input ground truth data. ### Mode Selector Inference Process -mode_selector_inference +![Alt text for the image](Pictures/mode_infer.png) In the inference phase, the agent discretely samples a cluster from the trained Gaussian Mixture Variational Autoencoder (GMVAE) model to calculate the Mixture of Gaussian variable $x$. This variable $x$, in conjunction with the conditional variable (initial image observation), is then inputted into the mode selector transformer decoder. The objective is to reconstruct the task embedding for inference, effectively translating the conditional information and sampled cluster into actionable embeddings. ### Mode Selector Qualitative Results -vis_mode_selector - +![Alt text for the image](Pictures/mode_qual.png) This disentanglement visualization with CGMVAE illustrates the efficacy of the Conditional Gaussian Mixture Variational Autoencoder (CGMVAE) in disentangling interaction modes for the "single drawer" object (ID: 20411), using a t-SNE plot for visualization. Task embeddings $\epsilon_j$, defined by the variance between initial and final object states, are visualized in distinct colors to denote various interaction modes and clusters. The sequence of figures demonstrates the CGMVAE's precision in clustering and aligning data points with their respective interaction modes: (1) Generated clusters from the CGMVAE mode selector reveal distinct groupings. (2) Ground truth task embeddings confirm the model's capacity for accurate interaction mode classification. (3) A combined visualization underscores the alignment between generated clusters and ground truth, showcasing the model's ability to consistently categorize tasks within identical interaction modes. @@ -55,7 +53,7 @@ This disentanglement visualization with CGMVAE illustrates the efficacy of the C Our objective is to infer a sequence of low-level actions $a=(\mathbf{p}, \mathbf{R}, \mathbf{q})$ from the current observation $O$ and task representation $\epsilon$, ensuring the action sequence effectively accomplishes the articulated object manipulation task while aligning with the constraints imposed by $\epsilon$. -![actaim2_rvt](https://github.com/pairlab/actaim2-eccv24/assets/30140814/1f98e07b-a38a-4833-8b86-9247159f3e2d) +![Alt text for the image](Pictures/action.png) Interaction mode $\epsilon$ is sampled from latent space embedding from the model selector. Multiview RGBD observations are back-projected and fused into a color point cloud. Novel views are rendered by projecting the point cloud onto orthogonal image planes. Rendered image tokens and interaction mode tokens are contacted and fed through the multiview transformer. This output consists of global feature for rotation $\mathbf{R}$ and gripper state $\mathbf{q}$ estimation and 2D per-view heatmap for position $\mathbf{p}$ prediction. @@ -65,10 +63,15 @@ Here, we provide more qualitative results about how our agent interacts with art We also show the correspondent video of how our robot interacts with the object by extracting the gripper pose from the predicted heatmap. ### Interacting with Faucet in 3 different interaction modes -qual_2 +![Alt text for the image](Pictures/qual1.png) Click the video here to see how the robot is interacting with the faucet above in 3 different interaction modes. + + + https://github.com/pairlab/actaim2-eccv24/assets/30140814/e895a768-315f-4141-a16c-89fbaf7c9911 https://github.com/pairlab/actaim2-eccv24/assets/30140814/adfe14b1-cc89-45b1-8b38-7bba6dd9001d @@ -76,7 +79,7 @@ https://github.com/pairlab/actaim2-eccv24/assets/30140814/adfe14b1-cc89-45b1-8b3 https://github.com/pairlab/actaim2-eccv24/assets/30140814/581f9083-2200-4322-a441-e3a885ab96e7 ### Interacting with a Table with multiple drawers in 3 different interaction modes -qual_3 +![Alt text for the image](Pictures/qual2.png) Click the video here to see how the robot is interacting with the table with multiple drawers above in 3 different interaction modes. @@ -88,7 +91,7 @@ https://github.com/pairlab/actaim2-eccv24/assets/30140814/27109fa4-56df-45f4-b27 ### Interacting with a Table with multiple drawers in 3 different interaction modes -qual_4 +![Alt text for the image](Pictures/qual3.png) Click the video here to see how the robot is interacting with the table with multiple drawers above in 3 different interaction modes. @@ -115,7 +118,6 @@ Interacting with a single drawer table and performing opening and closing the dr https://github.com/pairlab/actaim2-eccv24/assets/30140814/92e03793-5e60-4e2f-9af3-f729ed02977a - https://github.com/pairlab/actaim2-eccv24/assets/30140814/b2b5bf73-5b92-4682-8d71-220bc5806f62 Interacting with a door and performing opening and closing on either side of the door