Compiling .reality: The Technical Reality

TLDR: Xcode runs a verified two-pass process to compile .reality files, and understanding this unlocks the possibility of custom build pipelines, enables automation, and reveals Apple's USD-based spatial computing strategy.

There was a time in Apple's AR space when there was a format called .reality. It existed just for the purpose of using features that were still not possible on the USD format (and for other proprietary reasons, of course).

We used to see them on WWDC announcements and in demos for things
like For All Mankind's brilliant AR app, and pretty much everywhere there was AR, but over time, we began seeing just simple USDZ files instead. And for a while we feared it was the end of .reality... until 2023... when the introduction of visionOS, made the need for creating reality a thing again. There was a decision to keep the format (although with significant internal changes) and make the upgrade path as invisible as possible. Invisible unless you were using Reality Composer (not pro) and importing the .rcproject into Xcode, and compiling at build time. If you did that, you would be aware that it stopped working without notice (and your clients probably were upset about that).

Earlier versions of Xcode, including Xcode 11 (2019), used a command-line tool called rctool for 3D asset compilation. This was the original tool used to compile the .rcproject format created by (the forever in beta) Reality Composer. Designed for iOS AR experiences.

Key Features of rctool:

• Compiled .rcproject files from Reality Composer
• Simpler model focused on iOS/iPad AR scenes
• Less sophisticated schema and component generation.
• multi-piece and dependent beast.

Directory Structure
RCTool/
├── rctool                                   # Main executable
│   (Universal binary: x86_64 + arm64)
│
├── Protein.framework                        # 3D processing
│   (Asset optimization algorithms)
│
└── RCShared.framework                       # Shared resources
    ├── UI templates & localizations         # 6 languages
    ├── Audio assets                         # 8 sound effects
    ├── 3D models                            # 4 USDZ files
    └── Configuration files                  # JSON settings

Dependencies
rctool (main tool)
    ├── RCFoundation.framework
    ├── RCShared.framework   ◄─── Contains templates, audio, models
    ├── RealityKit.framework
    ├── DVTFoundation.framework
    └── Protein.framework    ◄─── 3D processing logic

But right now, Reality Composer Pro and family are powered by a completely different and unified tool, realitytool. It is the new, universal way of doing these things that came out with Xcode 15 and the release of the visionOS ecosystem. It puts together the required tasks into a single binary with some subcommands. 

USAGE: realitytool <subcommand>

OPTIONS:
  -h, --help              Show help information.

SUBCOMMANDS:
  create-schema           Command to parse custom component swift files to create a .usda schema with initial values
  compile                 Compiles a Reality Composer Pro asset into a runtime file
  image                   Process textures into Reality Assets

Key Features of realitytool:

• Unified CLI: Single binary with subcommands like create-schema and compile
• Schema generation: The create-schema subcommand generates USD schema files (.usda) from Swift component declarations and JSON metadata, enabling custom RealityKit components.
• Advanced compilation: The compile subcommand processes .rkassets bundles (the Reality Composer Pro format) into optimized .reality files
• Platform targeting: Explicit --platform and --deployment-target flags for cross-platform builds (visionOS, iOS, macOS)
• Enhanced sandboxing: Runs with strict sandbox profiles via Xcode
• Reality Composer Pro integration: Designed to work in tandem .rkassets (no support for .rcproject)

For what reason am I telling you all this? Because the change in architecture from rctool to realitytool shows an important part of Apple's strategy for spatial computing: they went from a simple compilation tool that only worked on iOS to a complex USD-based workflow that connects Swift's type system with universal scene description.

Now, let's look at how realitytool works behind the scenes. On the surface, you have three commands: create-schema, compile, and image.

PASS 1: SCHEMA GENERATION

The first pass converts your Swift components into a USD schema file.

• Input: Swift source files containing Component protocol definitions
• Output: .usda file with USD schema definitions for those components
• Build Phase: RealityAssetsSchemaGen (Xcode automatically adds this).

realitytool create-schema \
  --output-schema Build/Intermediates.noindex/.../CustomComponentUSDInitializers.usda \
  Build/Intermediates.noindex/.../ModuleWithDependencies.json

Simple Example: MyVisionApp

Let's say you have a project with two custom components:

MyComponent.swift:

import RealityKit

public struct MyComponent: Component {
    public var speed: Float = 0.13
    public var enabled: Bool = true
}

PortalComponent.swift:

import RealityKit

public struct PortalComponent: Component {
    public var targetScene: String = ""
    public var isActive: Bool = false
}

Xcode creates ModuleWithDependencies.json listing these Swift files, then runs:

realitytool create-schema \
  --output-schema CustomComponentUSDInitializers.usda \
  ModuleWithDependencies.json

This generates CustomComponentUSDInitializers.usda containing the USD schema definitions for both components, with their default values and types.

#usda 1.0
(
    prepend apiSchemas = ["PhysicsRigidBodyAPI"]
    upAxis = "Y"
    metersPerUnit = 1.0
    doc = """Generated USD schema for custom RealityKit components"""
)

def "MyApp" (
    prepend apiSchemas = [
        "PhysicsRigidBodyAPI:PhysicsAPI",
        "Custom:MyComponent",
        "Custom:PortalComponent"
    ]
    custom Schemas:moduleName = "MyApp"
)
{
    # MyComponent schema implementation
    custom Schemas:MyComponent.speed = 1.0 (
        doc = "Custom component field"
        variability = SdfVariabilityUniform
        customData = {
            bool __isCustomField__ = 1
            token __typeName__ = "float"
        }
    )
    custom Schemas:MyComponent.enabled = true (
        doc = "Custom component field"
        variability = SdfVariabilityUniform
        customData = {
            bool __isCustomField__ = 1
            token __typeName__ = "bool"
        }
    )

    # PortalComponent schema implementation
    custom Schemas:PortalComponent.targetScene = "" (
        doc = "Custom component field"
        variability = SdfVariabilityUniform
        customData = {
            bool __isCustomField__ = 1
            token __typeName__ = "string"
        }
    )
    custom Schemas:PortalComponent.isActive = false (
        doc = "Custom component field"
        variability = SdfVariabilityUniform
        customData = {
            bool __isCustomField__ = 1
            token __typeName__ = "bool"
        }
    )
}

Key Observations:

• Each Component struct becomes a USD schema class with a custom prefix
• Property types map: Float → float, Bool → bool, String → string
• Default values from Swift are preserved in USD schema.
• Custom metadata tracks type information and documentation.

Why This Matters

This pass extracts the (Swift) component definitions (including your custom ones) and converts them into explicit USD(A) schemas, which can then be used during the compilation step. 

This is significant because it means the process is capable of integrating Swift's type system into an entirely different format, like this USD's schema definition.

PASS 2: ASSET COMPILATION

What It Does

The second pass compiles your .rkassets bundle using the generated schema.

• Input: .rkassets file + .usda schema file
• Output: Optimized .reality runtime bundle
• Build Phase: RealityAssetsCompile (Xcode automatically adds this)

realitytool compile \
  --platform xrsimulator \
  --deployment-target 26.0 \
  --schema-file Build/Intermediates.noindex/.../CustomComponentUSDInitializers.usda \
  -o Build/Products/Debug-xrsimulator/YourApp/RealityKitContent.reality \
  YourApp/Packages/RealityKitContent/Sources/RealityKitContent/RealityKitContent.rkassets

The Same Example, Continued

With the schema generated in Pass 1, Xcode now compiles your assets:

realitytool compile \
  --platform xrsimulator \
  --deployment-target 26.0 \
  --schema-file CustomComponentUSDInitializers.usda \
  -o YourAssets.reality \
  YourAssets.rkassets

Platform Variations

The same command works for different platforms (deployment targets vary by platform):

visionOS Device:

realitytool compile \
  --platform xros \
  --deployment-target 26.0 \
  --schema-file CustomComponentUSDInitializers.usda \
  -o YourAssets.reality \
  YourAssets.rkassets

iOS:

realitytool compile \
  --platform iphoneos \
  --deployment-target 17.0 \
  --schema-file CustomComponentUSDInitializers.usda \
  -o YourAssets.reality \
  YourAssets.rkassets

macOS:

realitytool compile \
  --platform macosx \
  --deployment-target 15.0 \
  --schema-file YourAssets.usda \
  -o YourAssets.reality \
  YourAssets.rkassets

What Happens Here

1. Xcode loads your .rkassets bundle (Reality Composer Pro format).
2. References the generated schema for custom components
3. Compiles everything into a platform-optimized.reality bundle
4. Generates Swift APIs for type-safe access

Why This Matters

This last compilation step transforms your assets into runtime-ready elements. It is also somewhat flexible by providing platform targeting. The schema integration guarantees that your custom components work across different platforms, while the compilation stage has a significant impact on performance. The separation of schema generation and compilation allows for a clear division of responsibilities and is a fascinating architecture.

In practice, the conversion of your Swift-defined custom components to first-class USD allows true cross-platform compatibility (outside the walled garden) and third-party solutions, and in fact, without this phase, your components would be useless and unable to be accessed in Xcode, for example. In contrast, the compilation phase limits back the output to Apple platforms exclusively.

WHAT (STILL) THIS UNLOCKS

Understanding the two-pass process reveals possibilities that aren't obvious when it's all hidden inside Xcode.

Practical: Direct CLI Automation

You can run this process outside Xcode for automation:

Manual Build Process:

# Step 1: Generate schema
/Applications/Xcode.app/Contents/Developer/usr/bin/realitytool create-schema \
  --output-schema CustomComponentUSDInitializers.usda \
  ModuleWithDependencies.json

# Step 2: Compile assets
/Applications/Xcode.app/Contents/Developer/usr/bin/realitytool compile \
  --platform xrsimulator \
  --deployment-target 26.0 \
  --schema-file CustomComponentUSDInitializers.usda \
  -o YourAssets.reality \
  YourAssets.rkassets

Use cases:

• CI/CD pipelines—Build without Xcode UI
• Batch processing—multiple asset bundles
• Integration—Non-Apple tools
• Automation—Custom-built workflows
• Debugging—Run commands directly
• Custom solutions—Third-party tools

What Xcode does: Xcode's build phases coordinate the execution of these commands:

1. RealityAssetsSchemaGen ...CustomComponentUSDInitializers.usda
   └─ Calls: realitytool create-schema

2. RealityAssetsCompile ...YourAssets.reality
   └─ Calls: realitytool compile --platform xrsimulator

✅ As seen on GitHub Actions

 realitytool works on GitHub Actions CI with macOS runners. This could provide automated Reality Composer Pro builds in your pipeline (specially given that Xcode Cloud is unable to do that as of today).

Key Findings from CI Testing:

• ✅ realitytool is available on GitHub Actions macOS runners
• ✅ Both subcommands work: create-schema and compile
• ✅ No special setup required—part of standard Xcode installation
• ✅ Platform targeting works—supports visionOS, iOS, macOS, and tvOS.

CI Workflow Example:

- name: Build Reality Composer Pro assets
  run: |
    # Get Xcode path dynamically (supports both Xcode.app and Xcode-16.x.app)
    XCODE_PATH=$(xcode-select -p)

    # Pass 1: Generate schema
    "$XCODE_PATH/usr/bin/realitytool" create-schema \
      --output-schema CustomComponentUSDInitializers.usda \
      ModuleWithDependencies.json

    # Pass 2: Compile for visionOS
    "$XCODE_PATH/usr/bin/realitytool" compile \
      --platform xros \
      --deployment-target 2.5 \
      --schema-file CustomComponentUSDInitializers.usda \
      -o YourAssets.reality \
      YourAssets.rkassets

CI Compatibility Notes (as of today):

• Platform support: visionOS 2.5+ (GitHub Actions maximum), iOS 13+, macOS 10.15+, tvOS 13+
• Xcode version: 15.0+ (realitytool introduced with Xcode 15)
• Runner type: macOS (required - realitytool is macOS-only)
• Path handling: Use xcode-select -p for dynamic Xcode path detection.

Why This Matters for CI/CD: This confirms that Reality Composer Pro is CI/CD-ready. Your spatial computing assets can be

• Built automatically on every commit
• Tested across multiple platform targets
• Integrated with existing Apple build pipelines
• Deployed without manual Xcode intervention

Unlike the defunct rctool (Xcode 14 and prior), which had limited CI support, realitytool was clearly designed with automation in mind. At least this is what I've read from it, and IMO it shows Apple's efforts toward more developer-friendly spatial computing toolchains.

🔗 Repository, Proof and Two Smoking Barrels

GitHub - elkraneo/CompilingRealitySample

Contribute to elkraneo/CompilingRealitySample development by creating an account on GitHub.

GitHubelkraneo

What's Inside:

• ✅ Complete visionOS project with Reality Composer Pro
• ✅ GitHub Actions workflow: .github/workflows/visionos-build-test.yml
• ✅ MyCustomComponent.swift (generated by Reality Composer Pro)
• ✅ Build logs proving realitytool execution
• ✅ Working CI configuration

GitHub Actions Workflow: The workflow tests:

1. ✅ realitytool existence on macOS runners
2. ✅ realitytool --help execution
3. ✅ realitytool create-schema functionality
4. ✅ realitytool compile functionality
5. ✅ Full visionOS build with Reality Composer Pro assets

How to Verify:

1. Visit: https://github.com/elkraneo/CompilingRealitySample/actions
2. Click on any workflow run. 
3. the "Build the project" step 
4. "RealityAssetsSchemaGen" and "RealityAssetsCompile" phases
5. You'll see exact realitytool commands being executed.

Every commit triggers an automated build that proves realitytool to work on CI. See commit d54ba3b for the working version with deployment target fixes.

APPENDIX: BUILD SYSTEM INTEGRATION

How Xcode Orchestrates This

Xcode automatically adds custom build phases for Reality Composer Pro projects:

# In your project.pbxproj
buildPhases = (
    "Sources",                      # Compile Swift
    "Frameworks",                   # Link packages
    "Resources",                    # Copy resources
    "RealityAssetsSchemaGen",       # ← Custom: Pass 1
    "RealityAssetsCompile",         # ← Custom: Pass 2
)

What Gets Generated

Intermediate Files (Build/Intermediates.noindex/):

RealityKitContent.build/Debug-xrsimulator/
└── RealityKitContent_RealityKitContent.build/
    └── DerivedSources/RealityAssetsGenerated/
        ├── ModuleWithDependencies.json       # Input to Pass 1
        └── CustomComponentUSDInitializers.usda  # Output of Pass 1, input to Pass 2

Output Files (Build/Products/):

Debug-xrsimulator/
├── RealityKitContent_RealityKitContent.bundle/
│   ├── Info.plist
│   └── RealityKitContent.reality              # Final output
└── RealityKitContent.swiftmodule              # Generated Swift APIs

Build Log Analysis

To see the two-pass process in action:

1. Build the project in Xcode.
2. Report Navigator (Cmd+9)
3. Select the build.
4. Expand the "Build" section.

Look for:

• RealityAssetsSchemaGen (Pass 1)
• realitytool create-schema command
• RealityAssetsCompile (Pass 2)
• realitytool compile command

Platform Support (from Package.swift)

platforms: [
    .visionOS(.v26),    // 26.0+
    .macOS(.v26),       // 26.0+
    .iOS(.v26),         // 26.0+
    .tvOS(.v26)         // 26.0+
]

APPENDIX: PRACTICAL SCRIPTS

#!/bin/bash
# Build Reality Assets: Automated Pipeline
REALITYTOOL="/Applications/Xcode.app/Contents/Developer/usr/bin/realitytool"
PLATFORM="xrsimulator"
DEPLOYMENT_TARGET="26.0"

echo "=== Building Reality Assets ==="

# Pass 1: Generate schema
echo "Pass 1: Generating schema..."
$REALITYTOOL create-schema \
  --output-schema Build/Intermediate/CustomComponentUSDInitializers.usda \
  Build/Intermediate/ModuleWithDependencies.json

# Pass 2: Compile assets
echo "Pass 2: Compiling assets..."
$REALITYTOOL compile \
  --platform $PLATFORM \
  --deployment-target $DEPLOYMENT_TARGET \
  --schema-file Build/Intermediate/CustomComponentUSDInitializers.usda \
  -o Build/Output/MyAssets.reality \
  Assets/MyAssets.rkassets

echo "✓ Complete"

#!/bin/bash
# Build for all platforms
REALITYTOOL="/Applications/Xcode.app/Contents/Developer/usr/bin/realitytool"
SCHEMA_FILE="Build/Intermediate/CustomComponentUSDInitializers.usda"

# Step 1: Generate schema (once)
$REALITYTOOL create-schema \
  --output-schema "$SCHEMA_FILE" \
  Build/Intermediate/ModuleWithDependencies.json

# Step 2: Compile for all platforms
for platform in xros xrsimulator iphoneos iphonesimulator macosx; do
  echo "Compiling for ${platform}..."
  $REALITYTOOL compile \
    --platform "$platform" \
    --deployment-target "26.0" \
    --schema-file "$SCHEMA_FILE" \
    -o "Build/Output/MyAssets-${platform}.reality" \
    Assets/MyAssets.rkassets
done

- name: Build Reality Assets
  run: |
    # Pass 1: Generate schema
    /Applications/Xcode.app/Contents/Developer/usr/bin/realitytool create-schema \
      --output-schema CustomComponentUSDInitializers.usda \
      ModuleWithDependencies.json

    # Pass 2: Compile for visionOS
    /Applications/Xcode.app/Contents/Developer/usr/bin/realitytool compile \
      --platform xrsimulator \
      --deployment-target 26.0 \
      --schema-file CustomComponentUSDInitializers.usda \
      -o MyAssets.reality \
      MyAssets.rkassets

CONCLUSION

The two-pass realitytool process—hidden behind Xcode's build process—represents a sophisticated approach to spatial computing asset compilation:

1. Schema generation from Swift components
2. Platform-specific compilation with schema integration

Understanding this process unlocks:

• Build system visibility—you now know what Xcode is doing.
• Deeper insight into RealityKit's architecture/integration
• Debugging effectiveness—speeds up your iteration.
• Visibility into Apple's USD-based future

The next time Xcode builds your .rkassets files, you'll know it's not (all) magic; it's a carefully orchestrated process designed to make visionOS development future-proof.

And this isn't just about compilation; it's about Apple's deliberate movement to USD as the foundation for spatial computing, transforming reality files into part of a bigger ecosystem in which custom components, platform targeting, and cross-platform compatibility are more significant than ever.

Composing interactive 3D content with RealityKit and Reality Composer Pro | Apple Developer Documentation

Build an interactive scene using an animation timeline.

Apple Developer Documentation

Object tracking with Reality Composer Pro experiences | Apple Developer Documentation

Use object tracking in visionOS to attach digital content to real objects to create engaging experiences.

Apple Developer Documentation

Reality Composer | Apple Developer Documentation

A visual editor for RealityKit AR scenes.

Apple Developer Documentation

GitHub Actions documentation - GitHub Docs

Automate, customize, and execute your software development workflows right in your repository with GitHub Actions. You can discover, create, and share actions to perform any job you’d like, including CI/CD, and combine actions in a completely customized workflow.

GitHub Docs

Apple Augmented Reality by Tutorials

Learn Augmented Reality for the Apple Platform! This book is the easiest and fastest way to get hands-on experience using Apple frameworks and technologies like Reality Composer, RealityKit, and ARKit — all the available rendering technologies Apple has to offer, as well as a collection of fun projects for creating various real-world AR experiences. After reading this book, you’ll have a deep understanding of the technologies and frameworks used to create powerful, immersive AR experiences for the Apple platform. Take a deep dive into: AR Quick Look: Discover how to integrate AR Quick Look into your apps to give them some cool AR superpowers. Reality Composer & Reality Files: Find out how to leverage the power of Reality Composer to create interactive AR-based experiences. Reality Converter & PBR Materials: Discover how PBR materials can add a level of realism to your AR objects, and how you can use Reality Converter to convert, view and customize USDZ content. RealityKit: Find out how to set up and use RealityKit to build a face-based augmented reality app. Facial Blend Shapes: Build a fully interactive augmented reality face mask that reacts to your facial expressions using blend shapes. ARKit: Get a complete introduction to ARKit, Apple’s framework for creating fully interactive augmented reality, and learn about the different types of rendering options available with ARKit. Raycasting & Physics: Learn about raycasting, 2D hit-testing and the SpriteKit physics engine as you add more features and functionality to your game. ECS & Collaborative Experiences: Build a collaborative AR experience and learn how to create and manage a multipeer connection.

kodeco.comchrislanguage