Serverless AR Game with Bedrock AgentCore

AWS Hong Kong Summit 2026 · Developer Lounge Recap · Article 4 of 6

AWS Magazine · Field Report from Victoria Harbour

The Scene: Central-District Swagger Meets Serverless Sorcery

In my six decades of tracking technology shifts, I have stood on hundreds of conference floors, but nothing quite matches the electric pulse of the AWS Hong Kong Summit 2026 Developer Lounge.

Perched against the sweeping backdrop of Victoria Harbour, the Developer Lounge was a striking collision of Hong Kong luxury and AWS high-tech craft. Picture this: venture capitalists and startup founders closing seven-figure deals over artisanal drip coffee, international speakers trading architecture diagrams, and private developer meetups humming with high-stakes venture talk. Money was moving in and out, collaborations were forming on the fly, and amidst the neon shimmer and Central-district energy, builders were gathered around screens to witness pure serverless sorcery.

The standout attraction of the lounge? A demo by Cyrus Wong that brought together luxury entertainment and cloud engineering: The Jujutsu Kaisen Domain Expansion AR Game.

「唔錯喎！反應好似我去搶購減價化妝品噉快！當堂順眼好多！」 (Not bad! Your reaction is almost as fast as me grabbing a limited-edition bag in Shibuya!)

Strike Gojo Satoru’s Unlimited Void (無量空處) or Ryomen Sukuna’s Malevolent Shrine (伏魔御廚子) hand sign in front of a camera, and the screen erupts in 3D particle effects while a real (or simulated) humanoid robot adopts a kung-fu stance—all narrated live by an AI commentator speaking sassy, fashion-obsessed local Cantonese.

Here is the deep, hands-on architectural breakdown of how this fan-made application was constructed using AWS Bedrock AgentCore, AWS CDK, and serverless orchestrations.

The Star of the Show: AI Commentator Nobara

At the heart of the experience is an AI commentator modeled after Kugisaki Nobara (釘崎野薔薇). Powered by multimodal AI vision, she roasts players' hand gestures, clothes, postures, and messy rooms in real-time.

Multilingual Support & Vision Intelligence

• Languages: Local Cantonese, Taiwanese Mandarin, Japanese, and English.

• Personality: Central-district high fashion meets anime cursed techniques.

• Model Stack: moonshotai.kimi-k2.5 running inside a custom ECR container on Bedrock, paired with AWS Polly for high-fidelity neural speech synthesis.

System Architecture Summary

Step-by-Step Hands-On Execution Flow

+-----------------------------------------------------------------------------------+
END-TO-END GESTURE TO SPEECH FLOW

1. [ Browser Client ] -- MediaPipe 60FPS --> [ API Gateway ]
2. [ API Gateway ] -- Cognito JWT --> [ Monolithic Lambda ]
3. [ Monolithic Lambda ] -- XML multimodal payload --> [ AgentCore Runtime ]
4. [ AgentCore Runtime ] --> [ AWS Polly ] neural speech
5. [ AWS Polly ] --> [ S3 Audio Bucket ] --> browser playback
+-----------------------------------------------------------------------------------+

Step A: Client-Side Gesture Detection & API Signalling

The browser runs hand_tracker.js using Google’s MediaPipe Hands to analyze 21 3D landmarks at 60 FPS. When a gesture (e.g., Malevolent Shrine) is verified, battle.js sends a POST request with a Cognito Bearer JWT to Amazon API Gateway:

// battle.js: Dispatching recognized gesture to API Gateway[cite: 1]
async function triggerCursedTechnique(techniqueName) {
    const token = await getCognitoAccessToken(); // High-security Bearer Token[cite: 1]
    try {
        const response = await fetch("/api/trigger-technique", {
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                "Authorization": `Bearer ${token}`
            },
            body: JSON.stringify({
                technique: techniqueName, // e.g. "malevolent_shrine"[cite: 1]
                robotId: "all",
                sessionId: "mcpserver"
            })
        });
        const result = await response.json();
        playCommentaryAudio(result.audioUrl); // Play resulting neural voice track[cite: 1]
    } catch (err) {
        console.error("Failed to execute server-side technique orchestration:", err); //[cite: 1]
    }
}

Step B: Cognito Authorization & XML Base64 Bypass

When API Gateway validates the token, it forwards the request to a monolithic Flask Lambda (lambda_function.py).

To prevent enterprise network proxies from stripping binary multipart image data during transit, commentary.py embeds webcam snapshots inside custom XML tags in the text payload:

# commentary.py: Packaging snapshots inside XML text tags[cite: 1]
def generate_ai_commentary(content_block, session_id="mcpserver", image_base64_p1=""):
    if image_base64_p1:
        content_block += f"\n<p1_webcam_base64_jpeg>{image_base64_p1}</p1_webcam_base64_jpeg>" #[cite: 1]

    agent_client = boto3.client("bedrock-agentcore", region_name="us-east-1") #[cite: 1]

    payload_dict = {
        "prompt": content_block,
        "session_id": session_id
    }

    response = agent_client.invoke_agent_runtime(
        agentRuntimeArn=os.environ.get("AGENTCORE_RUNTIME_ARN"),
        runtimeSessionId=session_id,
        payload=json.dumps(payload_dict).encode("utf-8")
    )
    return response.get("response").read().decode("utf-8") #[cite: 1]

Step C: Serverless AgentCore Runtime Execution

The AWS Bedrock AgentCore Runtime routes the call via IAM SigV4 to the custom container (domain_commentator_agentcore). The container’s FastAPI service (commentator_agent.py) uses regex to extract the base64 XML block and reconstruct binary multimodal payloads for Bedrock:

# commentator_agent.py: Unwrapping XML image parts inside container[cite: 1]
@app.post("/invocations")
async def invoke_agent(request: Request):
    body = await request.json()
    prompt_text = body.get("prompt", "")

    p1_pattern = re.compile(r"<p1_webcam_base64_jpeg>(.*?)</p1_webcam_base64_jpeg>", re.DOTALL) #[cite: 1]
    p1_match = p1_pattern.search(prompt_text)

    image_b64_p1 = ""
    if p1_match:
        image_b64_p1 = p1_match.group(1).strip()
        prompt_text = p1_pattern.sub("", prompt_text).strip() # Clean text prompt[cite: 1]

    img_bytes_p1 = base64.b64decode(image_b64_p1)
    message_content = [
        {"text": "Player 1 webcam snapshot:"},
        {"image": {"format": "jpeg", "source": {"bytes": img_bytes_p1}}},
        {"text": prompt_text}
    ]

    response = await strands_agent.invoke_async(message_content) #[cite: 1]
    return JSONResponse(content={"response": str(response)}, status_code=200) #[cite: 1]

Step D: Hardware Dispatch via AgentCore Tool Gateway

When the model triggers physical actions, the runtime calls the AgentCore Tool Gateway (bedrockagentcore.Gateway), which invokes robot_tool_lambda.py via IAM SigV4 to publish MQTT commands to AWS IoT Core:

# robot_tool_lambda.py: Stripping tool prefix and invoking IoT Core[cite: 1]
def lambda_handler(event, context):
    full_tool_name = event.get("tool_name", "") # e.g., "robot-only-mcp-lambda___robot_wave"[cite: 1]
    local_tool_name = full_tool_name.split("___")[-1] if "___" in full_tool_name else full_tool_name #[cite: 1]

    if local_tool_name == "robot_wave":
        iot_client = boto3.client("iot-data")
        iot_client.publish(
            topic="arn:aws:iot:us-east-1:123456789012:topic/robot_1/topic",
            qos=1,
            payload=json.dumps({"action": "wave_hand", "timestamp": int(time.time())})
        )
        return {"status": "SUCCESS", "message": "Robot hand wave triggered."} #[cite: 1]

Infrastructure as Code: CDK Core Patterns

To keep infrastructure repeatable, modern, and clean, the entire system is modeled in AWS CDK.

1. Centralized Tool Gateway Construct (robot-tool-gateway.ts)

// Constructing the Tool Gateway with IAM SigV4 security[cite: 1]
export class RobotToolGatewayConstruct extends Construct {
  public readonly robotToolFunction: PythonFunction;
  public readonly gateway: bedrockagentcore.Gateway;

  constructor(scope: Construct, id: string, props: RobotToolGatewayConstructProps) {
    super(scope, id);

    this.robotToolFunction = new PythonFunction(this, "RobotToolFunction", {
      entry: path.join(__dirname, "../../../mcp_server"),
      runtime: SHARED_PYTHON_RUNTIME,
      index: "robot_tool_lambda.py",
      handler: "lambda_handler",
      timeout: Duration.seconds(30),
    });

    this.gateway = new bedrockagentcore.Gateway(this, "RobotToolGateway", {
      description: "AgentCore gateway fronting the robot Lambda tools",
      authorizerConfiguration: bedrockagentcore.GatewayAuthorizer.usingAwsIam(), // SigV4 Auth[cite: 1]
    });

    this.gateway.addLambdaTarget("RobotToolLambdaTarget", {
      gatewayTargetName: "robot-only-mcp-lambda",
      lambdaFunction: this.robotToolFunction,
      toolSchema: bedrockagentcore.ToolSchema.fromLocalAsset(materializeRobotToolSchemaAsset()),
    });
  }
}

2. Serverless AgentCore Commentator Runtime (domain-expansion-serverless.ts)

// Deploying commentator runtime with cost-aware lifecycles[cite: 1]
const runtime = new agentcore.Runtime(this, "Runtime", {
  runtimeName: "domain_commentator_agentcore",
  agentRuntimeArtifact: agentcore.AgentRuntimeArtifact.fromAsset(
    path.join(__dirname, "../../../domain-expansion-commentator-agentcore"),
    { platform: Platform.LINUX_ARM64 }
  ),
  authorizerConfiguration: agentcore.RuntimeAuthorizerConfiguration.usingIAM(),
  lifecycleConfiguration: {
    idleRuntimeSessionTimeout: Duration.seconds(120), // 2 min idle timeout[cite: 1]
    maxLifetime: Duration.seconds(900),              // 15 min max session[cite: 1]
  },
  tracingEnabled: true,
  environmentVariables: {
    BEDROCK_MODEL_ID: "moonshotai.kimi-k2.5",
  },
});

Split-Route API Gateway Security Model

The team implemented a split-route REST model to resolve a classic web conflict: standard browser elements (<img src="...">) cannot pass Bearer headers, but write operations cost LLM money.

// Public vs Protected Endpoint Configuration[cite: 1]
const restApi = new apigateway.RestApi(this, "DomainExpansionRestApi", {
  restApiName: "Domain Expansion Serverless REST API",
});

const restAuthorizer = new apigateway.CognitoUserPoolsAuthorizer(this, "DomainExpansionRestApiAuthorizer", {
  cognitoUserPools: [props.userPool],
});

// 🔓 PUBLIC READ-ONLY ENDPOINTS (No Cognito Token required)[cite: 1]
apiResource.addResource("get-snapshot").addMethod("GET", lambdaIntegration, {
  authorizationType: apigateway.AuthorizationType.NONE, //[cite: 1]
});

// 🔒 PROTECTED WRITE ENDPOINTS (Strict Cognito Validation)[cite: 1]
apiResource.addResource("trigger-technique").addMethod("POST", lambdaIntegration, {
  authorizationType: apigateway.AuthorizationType.COGNITO, // Edge validation[cite: 1]
  authorizer: restAuthorizer,
});

Observability & Enterprise Tracing

Observability is handled via CloudWatch Vended Logs namespaces and AWS X-Ray:

// Configuring Vended Logs namespace for AgentCore[cite: 1]
const applicationLogGroup = new logs.LogGroup(scope, `${id}ApplicationLogGroup`, {
  logGroupName: `/aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/${gateway.gatewayId}`, //[cite: 1]
  retention: logs.RetentionDays.THREE_DAYS,
});

This ensures end-to-end distributed tracing on a single X-Ray map:

Browser Gesture → API Gateway → Lambda Router → AgentCore Container → Strands LLM → AWS Polly TTS.

Smart Client Guards & Character Engineering

1. Cost-Guarding Session Logic (auth-check.js)

To avoid billing leaks from idle browser tabs keeping AgentCore sockets open, the client checks the Cognito JWT every 30 seconds on-device:

Session Expired = currentTime >= jwt.exp

If expired, it closes active WebSockets immediately:

function checkSessionGuard() {
    const token = getCognitoToken();
    if (isTokenExpired(token)) {
        if (webSocketConnection) webSocketConnection.close(); // Prevent billing leaks[cite: 1]
        triggerLocalLogout();
    }
}
setInterval(checkSessionGuard, 30000); // 0 cloud compute overhead[cite: 1]

2. Speech Text Sanitization (commentary_tts.py)

Before passing model outputs to AWS Polly, Markdown symbols (**bold**, #) are converted to clean plain text using BeautifulSoup to prevent Polly from reading syntax aloud:

import markdown
from bs4 import BeautifulSoup

def sanitize_text_for_tts(raw_text):
    html_formatted = markdown.markdown(raw_text)
    return BeautifulSoup(html_formatted, "html.parser").get_text() # Clean plain text[cite: 1]

3. Soul as Code Configuration

Character identity is fully decoupled into standard Markdown files loaded at runtime:

• IDENTITY.md: Directives on sassy Cantonese styling and demeanor.

• SOUL.md: Combat lore rules (e.g., how to roast Gojo Satoru users).

Architectural Economics & Pay-As-You-Go

Closing Reflection: The Developer Lounge Vibe

What Cyrus Wong and the team demonstrated at the AWS Hong Kong Summit 2026 Developer Lounge was more than a fun fan game. It was a masterclass in modern serverless design: high fashion, high technology, low latency, and zero idle cost.

As the sun set over Central, illuminating Victoria Harbour, developers were still clustered around the booth, scanning CDK repositories and testing hand signs. That is the real spirit of the AWS Developer Lounge—bringing world-class engineering, vibrant local culture, and serverless power together under one roof.