CGGC and GIP lab project
CS faculty, Technion
Live Matrix effect
Application of "digital rain" shader in real-time on iOS using triplanar shading and LiDAR sensor
GOALS & STORY
Considering existing matrix effect shader that was built as one of the previous projects of the lab, the main goal of this project is to apply the digital rain effect on the mesh that was created in real time using some kind of sensor, meaning - to move the matrix effect to AR.
As we decided to use Unity as a base engine for the shader itself, we tried few options on different platforms, including Android and iOS. Using on of the recent iPhone models with LiDAR sensor we got the best results in terms of mesh building and rendering time and the quality of the scanned mesh itself, but we faced one serious problem. The mesh generated by LiDAR does not contain UV coordinates - and that means that usual shader building techniques wouldn't work. The solution that worked both towards avoiding UV coordinates and improving performance - triplanar shader.
OUR MODEL
After attempting multiple different approaches we managed to make the triplanar shader work with LiDAR. The application uses AR Foundation which uses updated ARKit on iPhones. This allows us to access and use the sensor API and receive a mesh. The mesh receives a shader and the shader itself is being managed through a script (we need to update the raindrops falling).
Other options were not as successful. On Android we can’t use the same component and could only identify very basic planes using the basic AR Foundation and ARCore components. This is why we consider the obtained result the best we could achieve.
Final result video
iPhone with LiDAR (big letters)
Android
Report
Project files (GitHub)
