Getting Started with RealityKit: Models
Max Cobb
Posted on February 1, 2021
There are two main ways of adding models to your RealityKit scene, importing from a USD (usd, usda, usdc, or usdz) or a Reality file, or creating a mesh using the MeshResource generators provided with Xcode. There are a few shapes that can be generated via the MeshResource generators, other shape must be imported via a USDZ model or experience file from Reality Composer.
This post also includes examples of how to import USD files from remote URLs, and customise imported models.
Importing from USDZ
One of the more popular and exciting methods of importing a model to RealityKit would be importing it from a USD file. There are several ways to do this, and not knowing the subtle differences can lead you to miss parts of the imported models such as animations.
All the import methods either require a name and an optional bundle (omit the bundle to use the app's main bundle), or require a url and a unique resource name. To see how to use each see this link for passing a name and bundle, and this link for the url and resource name methods. The rest of this post will only discuss the name + bundle examples for simplicity.
Loading with Entity Hierarchy
To maintain information bundled in the USD file such as animations, the best import methods are Entity.load and Entity.loadAsync. The difference between these two is that load will block the thread it is called on until the file has been imported, while loadAsync will use a LoadRequest (where the Output is an Entity) to get the model imported while allowing you to continue on the current thread.
Another difference between the two, is that load can throw an error, while the error found in loadAsync can be caught in the LoadRequest slightly differently.
Examples:
Simple loading of a USDZ file named toy_robot_vintage.usdz synchronously:
guard let entity = try? Entity.load(
named: "toy_robot_vintage"
) else {
// failed to load entity
return
}
// do something with entity
Loading a USDZ file named toy_robot_vintage.usdz asynchronously:
_ = Entity.loadAsync(named: "toy_robot_vintage").sink(
receiveCompletion: { loadCompletion in
// Added this switch just as an example
switch loadCompletion {
case .failure(let loadErr):
print(loadErr)
case .finished:
print("entity loaded without errors")
}
}, receiveValue: { entity in
// do something with entity
}
)
Loading multiple USDZ files asynchronously, where the parameter in receiveValue is of type [Entity], and will only reach there once all models have been imported:
_ = Entity.loadAsync(named: "toy_robot_vintage")
.append(Entity.loadAsync(named: "gramophone"))
.append(Entity.loadAsync(named: "fender_stratocaster"))
.collect()
.sink(receiveCompletion: { loadCompletion in
// Add switch case for loadCompletion
}, receiveValue: { entities in
// Do something with returned entities array.
}
)
Note: You may notice that there is a small freeze and when you add the entity to your scene, even when using loadAsync. The freeze is due to large meshes or images stored in the materials being loaded into memory, which must be executed on the main thread.
Loading an AnchorEntity
In a very similar way to load and loadAsync, loadAnchor can load your USD file in as an AnchorEntity. This lets you place it directly in your scene without needing to make a separate anchor for it. Any animations in your USDZ file will also be available, as with load and loadAsync.
Example:
guard let entity = try? Entity.loadAnchor(
named: "toy_robot_vintage"
) else {
// failed to load entity
return
}
entity.anchoring = AnchoringComponent(
.plane(
.horizontal, classification: .floor, minimumBounds: [1, 1]
)
)
arview.scene.addAnchor(entity)
In this example, the AnchoringComponent is set to place the contents of the USDZ file to a horizontal plane categorised as the floor with a size of at least 1m x 1m. Only loadAnchor is shown here, but loadAnchorAsync works the same way as loadAsync, it will just return an AnchorEntity instead of an Entity.
Load a BodyTrackedEntity
When you want to import a model that is rigged up to be animated by a real person, you instead want to use loadBodyTracked or loadBodyTrackedAsync to import a BodyTrackedEntity.
guard let entity = try? Entity.loadBodyTracked(
named: "robot"
) else {
// failed to load entity
return
}
// do something with body tracked entity
For more information on tracking human bodies in AR, this session from WWDC 2019 is a great resource:
https://developer.apple.com/videos/play/wwdc2019/607
Loading a Flattened Model
When you just want to import a Model from a USDZ file, removing entity hierarchy as well as animations, you can import using loadModel or loadModelAsync. This will turn your 3d model of what would be various levels of nodes and children into a single Entity containing meshes and an array of all the materials.
In my opinion, this difference can be confusing to someone who is new to RealityKit and wanting to import their model that has an animation from a USDZ file; as seen by various questions on Stack Overflow asking what happened to their USDZ animations when importing a file using these methods. However, once aware of this difference, it is very useful to not have all the overhead of a large hierarchy of entities that you may not need.
Loading a Remote USDZ File
Currently there is no straightforward or builtin way to import USDZ files from remote URLs, but thankfully there are a few solutions, here's one from StackOverflow:
This problem is solved let entity call on the main thread.
let url = URL(string: "download usdz url")
let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let destinationUrl = documentsUrl.appendingPathComponent(url!.lastPathComponent)
let session = URLSession(configuration: URLSessionConfiguration.default, delegate: nil, delegateQueue: nil)
var request = URLRequest(url: url!)
request.httpMethod = "GET"
let downloadTask = session.downloadTask(with:I've also altered the solution into a synchronous function with an example usage at the bottom in this gist:
Generating Shapes
In its current state, the provided generators include Box (cuboid), Plane, Sphere or Text. Ironically, out of the three shapes in the RealityKit logo, only one of those (a sphere) can be produced without importing USDZ files.
All the mesh initialisers can be found on this page:
https://developer.apple.com/documentation/realitykit/meshresource
The easiest way to add one of these shapes to your RealityKit scene would be to use the ModelEntity initialiser:
let newModel = ModelEntity(mesh: .generateBox(size: 1))
Notice here, no material was specified in the ModelEntity initialiser, which means this broken shader sort of look is applied to the cube shape. This is the case for all the other MeshResource initialisers.
For in-depth information on how to create materials in RealityKit and a few tricks with them, see my previous post:
maxxfrazer.Medium
Customising Models
Once you bring a USDZ file in your scene there may be some customisation you want to do to it, such as changing a material, or actually moving around one part of the model.
If you wanted to change the material of a model imported from a USDZ file, there are a couple of ways you can do so.
With Hierarchy
If you import your model with anything other than loadModel or loadModelAsync you can replace the material of your choosing with ease.
Take this file, gramophone.usdz:
In the above image I've selected the record in the middle, and on the right we can see its name is GramophoneRecord1, and if we select the materials tab we can see that it only has one material named pxrUsdPreviewSurface1SG:
There is actually only one material in this whole USDZ file, however different parts of it are used on different meshes in the file.
Next all we need to do is find the Entity named GramophoneRecord1 and replace its material with our own green one.
if let gramRecord1 = gramophoneModel.findEntity(
named: "GramophoneRecord1"
) as! HasModel {
gramRecord1.materials = SimpleMaterial(
color: .green, isMetallic: true
)
}
Original on the left, altered material on the right.
Without Hierarchy
If you imported your model using loadModel or loadModelAsync, then as discussed earlier there is no hierarchy of entities, just the one ModelEntity consisting of a ModelComponent with a single mesh and a collection of materials. Take this file, fender_stratocaster.usdz:
The USDZ file itself consists of two meshes one for the stand and one for the guitar. Each of those meshes contain materials, the stand has one, and the guitar actually has 19, so once imported with loadModel the ModelEntity will have one mesh and a list of 20 materials.
Let’s replace the guitar body with a generated SimpleMaterial. It could take a bit of guess work to figure out which one in those 20 is that material, but by clicking on the part of the model that has the material you want to change Xcode highlights it in the side bar, see that it is the very first material in the guitar’s list of materials. It isn’t incredibly clear, but the first material when imported with loadModel is actually that of the guitar stand, leaving the guitar body material the second in the list, index 1.
After importing the model, simply running this I was able to change the guitar body to an electric blue:
modelAdjusted.model?.materials[1] = SimpleMaterial(
color: .blue, isMetallic: true
)
Original on the left, altered material on the right
The version without hierarchy leads to a bit less code, but doesn't allow for embedded animations, and can lead to issues if our USDZ file ever changes such that the material order differs. Whereas the version with hierarchy lets us find a specific part of the model, which would be preferred with a more complex model.
I hope you found something useful in this post, give it a like if you did! 👏👏👏
Send me a message on twitter or leave a comment if there’s something I’ve missed out or if there’s something that isn’t working for you in RealityKit.
Posted on February 1, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related