Skip to content

AI_Usage.md

Latest commit

 

History

History
76 lines (41 loc) · 4.42 KB

AI_Usage.md

File metadata and controls

76 lines (41 loc) · 4.42 KB
Raw

AI Usage Report

Group ID: 1

Members: Muhammad Hussain Habib (27100016), Ayaan Ahmed (27100155)

Week 1: Feature Matching & Visualization

1. Image Display Boilerplate

Generated Python code to display a set of 10 images in a grid format.

2. Implementation of cv2.drawMatches

Consulted AI to verify the correct parameter parameters for cv2.drawMatches and to generate the code for displaying the final matched output because the drawMatches function requires specific input types (keypoints vs. raw points) and flag configurations. AI helped clarify these requirements to ensure the lines connecting matches were drawn correctly

Week 2: Point Cloud Generation & Processing

1. Filtering Inliers with Masks

We used AI-suggested syntax for filtering point arrays using the mask generated by RANSAC:

pts1_inliers = pts1[mask.ravel() == 1]
pts2_inliers = pts2[mask.ravel() == 1]

Used ravel() to flatten the mask array into a 1D shape. And == 1 because in OpenCV masks, a value of 1 indicates an "inlier" while 0 is an outlier so it allowed us to discard noise and keep only the valid geometric points

2. Color Extraction

Generated code to extract RGB color values from the source images corresponding to specific 2D keypoint coordinates so as to colorize 3D reconstruction

3. PLY File Saving & Visualization

  1. PLY Format: Generated the function to write the 3D points and colors into a .ply file format, including the correct header structure (element vertex, property float x, etc.) because the .ply file header requires a strict format to be readable by software like CloudCompare
  2. Visualization: Generated code snippets to visualize the resulting point cloud

Week 3: Multi-View Reconstruction & Refinement

1. Reconstruction Data Structure

Used AI to design the structure of the ReconstructionMap class. This helper class was needed to modularize the data management, keeping track of growing lists of 3D points, camera poses, and the complex associations (correspondences) between 2D image keypoints and 3D map points.

2. Bundle Adjustment Implementation

Utilized AI to generate the boilerplate code for the scipy.optimize.least_squares implementation. Specifically, AI assisted in constructing the residual function (calculating the difference between projected 3D points and observed 2D points) and setting up the Jacobian sparsity matrix. This was crucial for optimizing memory usage and runtime when refining thousands of points.

3. PnP (Perspective-n-Point) Logic

Generated the solve_pnp function logic to bridge the feature matching and pose estimation steps. AI provided the workflow for:

  1. Identifying matches between descriptors in a new image and descriptors already associated with 3D points

  2. Formatting these 2D-3D correspondences correctly for input into cv2.solvePnPRansac

  3. Filtering the matches based on the resulting RANSAC inliers

4. JSON Export & Coordinate Transformation

Generated the export_final_fixed function to handle the data export for the Week 4 web viewer. We consulted AI to correctly implement the coordinate system transformation matrix needed to convert our OpenCV-based reconstruction (Y-down, Z-forward) into a WebGL/Three.js compatible format (Y-up, Z-backward), and to serialize the camera matrices into the required JSON structure.

Week 4: Interactive Web Viewer

1. Three.js Integration & Scene Setup

Utilized AI to scaffold the web viewer implementation using Three.js. While we understood the conceptual logic (loading the point cloud, positioning camera waypoints, and enabling user navigation) we used AI to translate these requirements into the correct Three.js API calls, including setting up the PLYLoader, OrbitControls, and TWEEN animation library for smooth camera transitions.

2. Waypoint Visualization & Interaction

AI assisted in implementing the interactive cone-based waypoints that represent camera positions. This included correctly decomposing the 4x4 transformation matrices to extract position and orientation, and setting up the raycasting logic to detect user clicks on waypoints for photo overlay display.

3. Navigation Controls

Generated the keyboard-based walking controls (WASD/Arrow keys) and double-click teleportation system. AI helped structure the Plane.intersectLine logic for ground-plane raycasting and the smooth camera animation using TWEEN to interpolate between positions.