Pipeline
FOMENO breaks the work of bringing intangible heritage into XR into a series of clear, repeatable steps. Each step has a defined purpose, and for each one there are different tools you can use depending on your material and your target platform.
The pipeline moves through three phases: building the character, giving it movement, and bringing it into XR. For each step below, we explain what we did, list the tools we used, and note alternatives where they exist.
This list isn't fixed. If you know of a tool that fits a step, you can suggest it through the Contribute page.
Character
Use references to create characters that can be animated.
-
Step 1 Generate the character
- In
- References
- Out
- Generated character images
This step produces the character as 2D images, which later steps turn into a rigged 3D model. The starting point can be scientific papers describing how people looked in a given place and period, combined with stories, written records, or artefacts carrying a depiction. Those sources are used to generate the images with AI.
What we did
Our sources were mostly scientific papers on how the figures looked at the site and period (source documents (link to be added)). We gathered them into Google NotebookLM and asked it to pull the relevant detail and write a prompt for an image generator, rather than describing the character ourselves. We also told it what the images were for: because the character had to be rigged later, we needed a neutral T-pose, and the prompt had to account for that. We then generated the images from that prompt in ChatGPT (resulting image (link to be added)).
Image to be added
The generated character in the neutral T-pose, beside the documented figures it was drawn from. Watch out for
AI image generation drifts towards stereotypes and game-like figures instead of what the sources actually describe. Two things keep it in check. First, state in the prompt that the result must rely only on the documented detail, with no stereotypes and no stylised game characters. Second, review both the references and the generated images with the scientists as you go.
Tools
This area moves fast and new tools appear constantly, so it is worth checking what is available before you start. The tools below are grouped by the two jobs this step involves. A single tool may eventually handle both at once.
Information gathering
-
Google NotebookLM We used this
Reads the source documents and writes the image-generation prompt.
-
Another option for reading the sources and drafting the prompt.
Image generation
-
ChatGPT We used this
Generates the character images from the prompt.
-
Alternative image generator.
-
Seedream (ByteDance)
Alternative image generator.
Know another tool for this step? Suggest a tool for step 1: Generate the character
-
Step 2 Build an animatable 3D model
- In
- Generated character images
- Out
- A rigged 3D model
This step turns the generated T-pose images into a rigged 3D model, ready to be animated.
What we did
Rather than modelling each character by hand from the reference sheet, we used an AI tool called Meshy.ai to build a rigged 3D character directly from the 2D T-pose images.
Image to be added
The 3D model produced from the T-pose images, with the rig Meshy generated for it. Watch out for
The models coming from the AI tools are not always cleanly rigged, and they are rarely optimised for a specific XR medium out of the box. Depending on the medium you are targeting, it is possible a 3D artist needs to fine-tune them.
Tools
This area moves fast and new tools appear constantly, so it is worth checking what is available before you start.
-
Meshy AI We used this
Generates a rigged 3D character directly from 2D T-pose images.
Know another tool for this step? Suggest a tool for step 2: Build an animatable 3D model
Movement
Movement is taken out of video and transferred onto the character.
-
Step 3 Source the movement
- In
- Reference information for the movement
- Out
- Source video of the movement
We find or generate the source videos that contain the motion we want the character to perform.
What we did
We did not generate the video. We were fortunate to find a dance group (link to be added) already performing dances similar to the ones we wanted to capture, and with their permission we used some of their footage. That meant we did not need to generate the movement or stage a shoot ourselves. The performers are credited by name on the Prototype page.
Video to be added
An excerpt of the dance group's footage, as it went into the capture step. Watch out for
Copyright
Whenever the source material comes from someone else, whether that is the people performing or existing footage of them, get their permission before using it.
Video details
The next step pulls the movement out of this video with AI and transfers it onto the 3D character, so the footage has to be readable by a machine, not just by a person:
- Show the full body.
- Make sure the movement you want to capture is clearly visible.
- Keep the colour of clothing and background distinguishable so the motion stands out. This matters most for generated video, but helps in every case.
- If the heritage involves several people, film them one at a time where you can. A crowded scene makes it hard for the tool to isolate the movement you are after.
Tools and techniques
Use existing footage
-
Existing performance footage We used this
Find a group or archive already performing the practice and use their footage, with permission.
Generate the video with AI
-
AI video generation
Generates reference video of the movement from a description or references.
Film your own shoot
-
Staged shoot with actors
Use the reference documents, stories or depictions to direct actors performing the practice, and film it.
Know another tool for this step? Suggest a tool for step 3: Source the movement
-
Step 4 Capture the motion
- In
- Source video of the movement
- Out
- Captured animation data
AI motion-capture tools extract the movement from those source videos as animation data.
What we did
We fed the source video from the previous step into an AI tool that reads the movement and outputs it as motion data.
Tool name to be added.
Video to be added
The captured motion data played back on a plain skeleton, next to the source video. Watch out for
If your source video is good and you followed the points in the previous step, it translates almost directly into motion data. The quality here is really set upstream, in how the video was captured or generated.
Tools
This area moves fast and new tools appear constantly, so it is worth checking what is available before you start.
-
AI motion capture We used this
Reads the movement from the source video and outputs it as motion data.
Know another tool for this step? Suggest a tool for step 4: Capture the motion
-
Step 5 Apply the motion to the model
- In
- A rigged 3D model and captured animation data
- Out
- An animated character
The captured animation is applied to the 3D model, bringing the character to life.
The right approach depends on the platform you will publish to, which is decided in the following steps. It helps to know your target medium before you start, since it determines whether you merge inside the platform or prepare a finished animated model to import.
This step is where the pipeline branches by platform. The material from the earlier steps, the rigged model and the animation data, stays reusable. If you decide to target a different platform later, you come back to this step and produce it for that platform, without redoing any of the earlier work.
What we did
We used Unity as our distribution platform, so we combined the animation data and the rigged 3D model directly in Unity.
Video to be added
The character performing the captured movement. Tools and techniques
There are two ways to bring the animation data and the rigged model together.
Merge in the XR platform
Most 3D platforms for XR production can combine the animation data and the rigged model directly. Use this route when you build in the same platform you will publish to.
-
Unity We used this
We combined the animation data and the rigged model here, since it was our distribution platform.
-
Web-based AR, now a free open-source toolset.
-
Web-based AR, another browser option.
Merge in 3D software, then import
Combine the model and the animation in 3D software to produce a finished animated character, then import it into the platform you have chosen.
-
Produces an animated character to import into your platform.
Know another tool for this step? Suggest a tool for step 5: Apply the motion to the model
XR
The character is placed in a scene built for the platform your audience will use.
-
Step 6 Add supporting elements
- In
- One or more animated characters
- Out
- A composed experience with all elements added
This step is where a single animated character becomes a full experience. If the practice involves more than one performer, you compose the group here so their arrangement reflects the source material. And if it carries other modalities, such as music or a surrounding environment, you add those here too. This work happens in the platform you are building in, and it can merge with the steps around it, but we keep it separate because assembling the full experience is its own job.
What we did
What we did for the prototype, to be added.
Video to be added
The composed scene, with the performers arranged and the audio in place. Tools
The composition and any extra modalities are built in the platform you are working in. Which additional tools you need depends on what you are adding, such as audio software for music or asset tools for an environment.
Tools to be added.
We haven't documented the tools for this step yet. Suggest a tool for step 6: Add supporting elements
-
Step 7 Compose for the medium
- In
- A composed experience
- Out
- A finished XR experience
This is the final step: preparing the assembled experience for publishing to its target XR medium. Which platform you use depends on the medium you are publishing to. It can overlap with the earlier steps, and they can merge, but we keep publishing-readiness separate because it is its own task.
What we did
We prepared the prototype for publishing in Unity, the platform we had been building in.
Target medium and specifics to be added.
Video to be added
The finished experience running on its target device. Platforms
These are the platforms we see used most often. They are not the only options, and new ones appear regularly. For web-based delivery, use 8th Wall or ZapWorks. For app-based delivery, Unity is usually the better choice.
Which XR media each platform can publish to
Platform Web AR Mobile AR AR headset VR headset Installation Unity We used this No Yes Yes Yes Yes Unreal Engine No Yes Yes Yes Yes 8th Wall Yes Yes Yes Yes Yes ZapWorks (Zappar) Yes Yes Yes Yes No Know another tool for this step? Suggest a tool for step 7: Compose for the medium