A cultural heritage digital experience powered by Unreal Engine and the Epic ecosystem
Jinyoung Choi
April 3, 2024
Technology Research Institute for Culture & Heritage (TRIC) is a company that digitally preserves global heritage artifacts (including cultural heritage, intangible cultural heritage, natural heritage, and future heritage) and uses them in interactive content. TRIC contributes to the preservation of authentic heritage: increasing accessibility to it, enabling its rediscovery, and promoting its value.
Cultural heritage is an important way for a country to understand its past. Heritage is how history and cultural identity are maintained through the present and into the future. In South Korea, around 230,000 cultural assets were either legally or controversially exported, but various efforts to repatriate them back to the country are underway.
Among these efforts is the idea of digital sharing, a South Korean government-led initiative that leverages digital technology and equipment to replace the complicated and difficult physical repatriation process.
With digital sharing, Korean artwork stored in museums abroad can be exhibited in its home country as a precise digital replica of the original artifact. Interactive 3D content is developed based on the digital replica, providing a realistic and engaging experience for viewers.
The first project under the banner of this idea was the Digital Homecoming Project for the Korean Cultural Assets at the Cleveland Museum of Art (CMA). To get a deeper look at what it’s all about, we interviewed Kim Jikio, CEO of the Technology Research Institute for Culture & Heritage (TRIC), who directed this project to digitally bring Korean cultural assets from the CMA home.
Among these efforts is the idea of digital sharing, a South Korean government-led initiative that leverages digital technology and equipment to replace the complicated and difficult physical repatriation process.
With digital sharing, Korean artwork stored in museums abroad can be exhibited in its home country as a precise digital replica of the original artifact. Interactive 3D content is developed based on the digital replica, providing a realistic and engaging experience for viewers.
The first project under the banner of this idea was the Digital Homecoming Project for the Korean Cultural Assets at the Cleveland Museum of Art (CMA). To get a deeper look at what it’s all about, we interviewed Kim Jikio, CEO of the Technology Research Institute for Culture & Heritage (TRIC), who directed this project to digitally bring Korean cultural assets from the CMA home.
Could you tell us about TRIC and the Digital Homecoming Project for the Korean Cultural Assets at CMA?
TRIC is a company that breathes new life into culture and heritage through digital technology, R&D, and related enterprises . Over the past ten years, TRIC has been developing digital archives and content for major global heritage curators in places like Indonesia, Laos, Turkey, Kazakhstan, Japan, and Egypt to make them available to everyone. Recently, thanks to the development of real-time content workflows, we've been working on the Cleveland Museum of Art project with a pipeline built upon Unreal Engine.
Various projects carried out by TRIC
The Digital Homecoming Project for the Korean Cultural Assets at the CMA is an international collaboration project that will be exhibited simultaneously in Korea and the US from March.
We're capturing digital data from 3D scans of the Korean cultural assets held by the CMA and producing realistic digital content based upon that data. The project has been ongoing for three years. In the first year of the project back in 2022, we digitally archived 13 major Korean artifacts at the CMA. In 2023, we selected the 19th-century folding screen ‘Seven Jeweled Mountain’ from those artifacts based on its suitability for large-scale exhibition display and began to produce immersive, realistic content based on the artwork. Now the project is in its third year, the results of our hard work will be shown in an exhibition that will take place in both Korea and the US.
We're capturing digital data from 3D scans of the Korean cultural assets held by the CMA and producing realistic digital content based upon that data. The project has been ongoing for three years. In the first year of the project back in 2022, we digitally archived 13 major Korean artifacts at the CMA. In 2023, we selected the 19th-century folding screen ‘Seven Jeweled Mountain’ from those artifacts based on its suitability for large-scale exhibition display and began to produce immersive, realistic content based on the artwork. Now the project is in its third year, the results of our hard work will be shown in an exhibition that will take place in both Korea and the US.
Digital Homecoming Project for the Korean Cultural Assets at CMA: Seven Jeweled Mountain
Why did you bring Unreal Engine into TRIC’s pipeline?
A critical factor in the success of a content production project based around cultural artifact data is the ability to achieve parity between the fidelity of the original artwork and the digital version.
However, as the mesh and texture get more precise, the data inevitably gets heavier—especially when trying to create something like a building structure rather than a single artwork. Therefore, we used to put a lot of resources into lightening and optimizing the data.
This optimization process was necessary because the real-time interactive content will primarily be viewed in VR or AR, as well as in physical installations. TRIC has always considered seamless interaction between the audience and art through media technologies to be crucial. That's why we adopted Unreal Engine, which can produce real-time content while still preserving the quality as is, with minimal work.
High-precision digital artifacts created with Unreal Engine
Using Lumen and Nanite in Unreal Engine 5, the details of the digitized artifact can be depicted at a high visual quality. These tools also reduce the burden of the long optimization process that used to be necessary for real-time production, and enable a quicker delivery of the desired quality with a smaller team.
In addition, Unreal Engine has remarkably shortened the pipelines for all TRIC projects, including those for immersive multiplanar imaging that require complicated calculation and simulation from the base videos, anamorphic imaging, and content development for new output formats.
In addition, Unreal Engine has remarkably shortened the pipelines for all TRIC projects, including those for immersive multiplanar imaging that require complicated calculation and simulation from the base videos, anamorphic imaging, and content development for new output formats.
One of TRIC’s anamorphic project
Please tell us about the pipeline used for the project.
The project process is divided into two parts: scanning and digitizing the 13 artifacts; and producing the realistic ‘Seven Jeweled Mountain’ content. We mainly used RealityCapture for the scanning stage, and the Unreal Engine-based pipeline was built to create the realistic ‘Seven Jeweled Mountain’ content.
In the first year of the project, we captured the ultra-precise digital data for the 13 Korean artifacts kept in the CMA, including the ‘Seven Jeweled Mountain’. Precise 3D scanning ordinarily requires the use of professional equipment such as LiDAR or structured light scanners.
However, the artifacts were overseas in this case, making it difficult to bring in the equipment or create an optimal scanning environment. As an alternative, we leveraged a photogrammetry method to capture the data using RealityCapture, which we frequently rely on to produce 3D datasets for spaces and objects in the buildings housing the artworks. We took thousands of pictures for each artifact and created high-fidelity assets, both in mesh and texture, using RealityCapture. On a side note, RealityCapture was also crucial for completing the Charlottenburg Palace Ceramic Room project—despite the fact that the German border was closed due to COVID-19—in collaboration with an architectural photographer in Berlin.
Comparison of 13 scanned artifacts: (Left) Real Assets / (Right) Digital Assets
In the production of the realistic content that rolled out in the second year, we aimed to translate the original ‘Seven Jeweled Mountain’ artwork, which is painted on a wide folding screen, to three immersive digital screens while preserving the traditional art style and adding the sense of three-dimensional space.
To achieve this, we first built a vast gigapixel data network connecting the thousands of images that made up each side of the folding screen using a camera rotator, then created separate layers for each object, color gamut, and level. We used Unreal Engine to place all the elements in a three-dimensional space and filmed them to recreate the traditional 2D artwork of the ‘Seven Jeweled Mountain’ as spatial, dynamic 3D content.
To achieve this, we first built a vast gigapixel data network connecting the thousands of images that made up each side of the folding screen using a camera rotator, then created separate layers for each object, color gamut, and level. We used Unreal Engine to place all the elements in a three-dimensional space and filmed them to recreate the traditional 2D artwork of the ‘Seven Jeweled Mountain’ as spatial, dynamic 3D content.
Layering a 2D folding picture and rearrange it in 3D space in Unreal Engine
One of the most important parts in creating the three-dimensional effect is expressing the brushstroke of the character outline. We created a shader using Unreal Engine's Material Editor. The paper-like, high-quality texture was downloaded from Megascans, modified, and applied to the character's post-processing material.
The fog was added using Unreal Engine's Niagara Fluids plugin to bring a dynamic feel to the artwork. Niagara Fluid enabled us to produce a simulation that reacts to other objects in real time or control the desired timing via Sequencer.
Niagara Fluids is lighter than the previous method of using OpenVDB and any required edits could be made instantly. To display these well-made characters and effects on a triple-planar screen, the animation was effectively processed using a camera rig that binds three cameras into one. Each video was rendered simultaneously using Movie Render Queue which enables rendering from three cameras placed in Sequencer at the same time.
The fog was added using Unreal Engine's Niagara Fluids plugin to bring a dynamic feel to the artwork. Niagara Fluid enabled us to produce a simulation that reacts to other objects in real time or control the desired timing via Sequencer.
Niagara Fluids is lighter than the previous method of using OpenVDB and any required edits could be made instantly. To display these well-made characters and effects on a triple-planar screen, the animation was effectively processed using a camera rig that binds three cameras into one. Each video was rendered simultaneously using Movie Render Queue which enables rendering from three cameras placed in Sequencer at the same time.
Scene rendered simultaneously with three cameras
How were Unreal Engine and the Epic Games ecosystem helpful in the making of the realistic ‘Seven Jeweled Mountain’ content?
Content that uses traditional Korean art resources like the ‘Seven Jeweled Mountain’ requires a consistency in aesthetic sense and should accurately represent the style of Oriental painting throughout all elements and scenes.
That's why a production tool that enables precise control in all stages is essential. Unreal Engine provides an environment where we can control the composition assets in detailed precision via Niagara, Sequencer, and other systems, while preserving both visual quality and performance.
Moreover, the Unreal Engine Marketplace and Quixel Megascans offer an array of resources from their vast libraries for various natural phenomena such as clouds, the moon, fog, and wind. These libraries also help us to reduce the time it takes to complete tasks by providing the essential materials for realizing the three-dimensional yet traditional style of the rigged animation characters, the high-quality shaders, and textures that are suitable for the original texture of the characters.
Comparison of brush line shader: (Left) Opacity 0.2 / (Right) Opacity 0.8
Most of all, real-time rendering in Unreal Engine was the foundational technology of this project. We only had two months to develop the content and we were running short on time. Meanwhile, we had to film with three virtual cameras and render in an extremely high resolution to bring the content to an immersive screen with three screens placed in a trapezoidal shape.
Since the location of the audience and the angle of view would hugely impact the sense of immersion, repetitive and detailed adjustment was required. Because the project required professionals from Korea and the US to work together, we had to communicate detailed feedback through live conference calls.
Using Unreal Engine, we were able to reduce that communication time remarkably, making adjustments in real time together from opposite ends of the globe. What’s more, we were able to review and quality check the final product in the production stages, minimizing the chances of communication errors or the risk of having to redo work.
If we were to produce this in a traditional way, most of our precious time would’ve been spent on conference calls. Even if we’d have arrived at a conclusion, it would have been impossible to finish the project within the time allotted, not to mention at the quality required.
Since the location of the audience and the angle of view would hugely impact the sense of immersion, repetitive and detailed adjustment was required. Because the project required professionals from Korea and the US to work together, we had to communicate detailed feedback through live conference calls.
Using Unreal Engine, we were able to reduce that communication time remarkably, making adjustments in real time together from opposite ends of the globe. What’s more, we were able to review and quality check the final product in the production stages, minimizing the chances of communication errors or the risk of having to redo work.
If we were to produce this in a traditional way, most of our precious time would’ve been spent on conference calls. Even if we’d have arrived at a conclusion, it would have been impossible to finish the project within the time allotted, not to mention at the quality required.
TRIC holding a conference call with overseas experts
Please tell us about the future goals and direction of TRIC.
In terms of short-term goals, we’re focusing on successfully finishing the Digital Homecoming Project for the Korean Cultural Assets at CMA and the Egyptian Cultural Heritage Official Development Assistance (ODA) projects.
In the long term, we aim to build a real-time world for each historical era, like a metaverse, that connects the numerous datasets and real-time content we've amassed from these other projects. We would like to restore cities with a high historical preservation value in a three-dimensional way using Unreal Engine and the Epic Games ecosystem.
As a part of that, we're working on a project to construct the entirety of eighth century Seorabeol (present-day Gyeongju) using Unreal Engine. Creating a digital replica of Seorabeol, the capital of Silla, including its terrain and vegetation will produce a vast amount of data for architecture, roads, people, historical incidents, and more in one platform. We may also be able to update the virtual city as archeological excavation and research continues. It will be possible to use this interactive environment to instantly create a scene from a movie or documentary at photorealistic quality.
Seorabeol Real-time World: (Left) Lit View Mode / (Right) Nanite Triangle Visualization
Get Unreal Engine today!
Get the world’s most open and advanced creation tool.
With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.
With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.