perf(core): UTF-8-first content blocks + lower-allocation stream/HTTP transports

This change set focuses on reducing allocations and unnecessary transcoding across the MCP wire path (JSON-RPC + MCP content), especially for line-delimited stream transports (stdio / raw streams) and for text content blocks that frequently originate as UTF-8 already.

Key protocol / model updates
- ContentBlock JSON converter now reads the "text" property directly into UTF-8 bytes (including unescaping) without first materializing a UTF-16 string.
- Added Utf8TextContentBlock ("type":"text") to allow hot paths to keep text payloads as UTF-8; TextContentBlock now supports a cached UTF-16 string backed by a Utf8Text buffer.
- Added opt-in deserialization behavior to materialize Utf8TextContentBlock instead of TextContentBlock via McpJsonUtilities.CreateOptions(materializeUtf8TextContentBlocks: true); DefaultOptions remains compatible and continues to materialize TextContentBlock.
- ImageContentBlock and AudioContentBlock still have Data as a base64 string; added DataUtf8 and DecodedData helpers to avoid repeated conversions.
- BlobResourceContents updated to cache and interoperate between (1) base64 string (Blob), (2) base64 UTF-8 bytes (BlobUtf8), and (3) decoded bytes (DecodedData)

Serialization / JSON-RPC improvements
- JsonRpcMessage converter now determines concrete message type (request/notification/response/error) by scanning the top-level payload with Utf8JsonReader.ValueTextEquals and skipping values in-place, avoiding JsonElement.GetRawText() / UTF-16 round-trips.
- McpJsonUtilities now exposes CreateOptions(...) to produce an options instance that can override the ContentBlock converter (materializeUtf8TextContentBlocks) while still chaining MEAI type info resolver support.
- Added McpTextUtilities helpers (UTF-8 decode, base64 encode on older TFMs, and common whitespace checks used by transports).

Transport / streaming changes (stdio + streams)
- StreamClientSessionTransport and StreamServerTransport reading loops were refactored to a newline-delimited (LF) byte scanner:
  - Parses messages directly from pooled byte buffers + a reusable MemoryStream buffer.
  - Handles both LF and CRLF by trimming a trailing '\r' after splitting on '\n'.
  - Avoids UTF-16 materialization for parsing and (optionally) for logging by only decoding to string when trace logging is enabled.
- Added TextStreamClientSessionTransport to optimize the "text writer/reader" client transport:
  - Prefers writing UTF-8 directly when the writer is a StreamWriter.
  - Can read via the underlying stream when the reader is a UTF-8 StreamReader, falling back to ReadLineAsync for arbitrary TextReader implementations.

HTTP transport / client plumbing
- McpHttpClient now uses JsonContent.Create on NET TFMs and a new JsonTypeInfoHttpContent<T> on non-NET TFMs to serialize via JsonTypeInfo without buffering to compute Content-Length.
- StreamableHttpClientSessionTransport and SseClientSessionTransport disposal paths now consistently cancel and "defuse" CTS instances to reduce races/leaks during teardown.
- StreamableHttpSession updates activity tracking and shutdown ordering (dispose transport first, then cancel, then await server run) for cleaner termination.

Cancellation / lifecycle utilities
- Added CanceledTokenSource: a singleton already-canceled CancellationTokenSource plus a Defuse(ref cts, ...) helper to safely swap out mutable CTS fields during disposal.

URI template parsing
- UriTemplate parsing/formatting updated with a more explicit template-expression regex and improved query expression handling; uses GeneratedRegex and (on NET) a non-backtracking regex option for performance.

Tests / samples
- Added ProcessStartInfoUtilities to robustly locate executables on PATH and to handle Windows .cmd/.bat invocation semantics when UseShellExecute=false; updated integration tests accordingly.
- Added TextMaterializationTestHelpers to let tests run with either TextContentBlock or Utf8TextContentBlock materialization.
- Updated tests and samples to align with the new base64-string Data representation for image/audio blocks and to avoid unnecessary allocations in transport tests.

Behavioral notes / compatibility
- Wire format remains MCP/JSON-RPC compatible: messages are still newline-delimited JSON; CRLF continues to work.
- The Utf8TextContentBlock materialization is opt-in via McpJsonUtilities.CreateOptions(materializeUtf8TextContentBlocks: true); default behavior preserves prior materialized types for text blocks.

Author: Tyler-IN

samples/EverythingServer/Resources/SimpleResourceType.cs

@@ -31,7 +31,7 @@ public static ResourceContents TemplateResource(RequestContext<ReadResourceReque
             } :
             new BlobResourceContents
             {
-                Blob = System.Text.Encoding.UTF8.GetBytes(resource.Description!),
+                Blob = resource.Description!,
                 MimeType = resource.MimeType,
                 Uri = resource.Uri,
             };

samples/EverythingServer/Tools/AnnotatedMessageTool.cs

@@ -41,7 +41,7 @@ public static IEnumerable<ContentBlock> AnnotatedMessage(MessageType messageType
         {
             contents.Add(new ImageContentBlock
             {
-                Data = System.Text.Encoding.UTF8.GetBytes(TinyImageTool.MCP_TINY_IMAGE.Split(",").Last()),
+                Data = TinyImageTool.MCP_TINY_IMAGE.Split(",").Last(),
                 MimeType = "image/png",
                 Annotations = new() { Audience = [Role.User], Priority = 0.5f }
             });

src/ModelContextProtocol.AspNetCore/StreamableHttpSession.cs

@@ -1,4 +1,5 @@
-using ModelContextProtocol.Server;
+using ModelContextProtocol.Core;
+using ModelContextProtocol.Server;
 using System.Diagnostics;
 using System.Security.Claims;
 
@@ -16,7 +17,7 @@ internal sealed class StreamableHttpSession(
     private readonly object _stateLock = new();
 
     private int _getRequestStarted;
-    private readonly CancellationTokenSource _disposeCts = new();
+    private CancellationTokenSource _disposeCts = new();
 
     public string Id => sessionId;
     public StreamableHttpServerTransport Transport => transport;
@@ -124,7 +125,8 @@ public async ValueTask DisposeAsync()
             {
                 sessionManager.DecrementIdleSessionCount();
             }
-            _disposeCts.Dispose();
+
+            CanceledTokenSource.Defuse(ref _disposeCts);
         }
     }
 

src/ModelContextProtocol.Core/AIContentExtensions.cs

@@ -1,9 +1,8 @@
 using Microsoft.Extensions.AI;
 using ModelContextProtocol.Client;
 using ModelContextProtocol.Protocol;
-#if !NET
+using System.Buffers.Text;
 using System.Runtime.InteropServices;
-#endif
 using System.Text.Json;
 using System.Text.Json.Nodes;
 
@@ -263,6 +262,8 @@ public static IList<PromptMessage> ToPromptMessages(this ChatMessage chatMessage
         AIContent? ac = content switch
         {
             TextContentBlock textContent => new TextContent(textContent.Text),
+
+            Utf8TextContentBlock utf8TextContent => new TextContent(utf8TextContent.Text),
 
             ImageContentBlock imageContent => new DataContent(imageContent.DecodedData, imageContent.MimeType),
 
@@ -275,7 +276,9 @@ public static IList<PromptMessage> ToPromptMessages(this ChatMessage chatMessage
 
             ToolResultContentBlock toolResult => new FunctionResultContent(
                 toolResult.ToolUseId,
-                toolResult.Content.Count == 1 ? toolResult.Content[0].ToAIContent() : toolResult.Content.Select(c => c.ToAIContent()).OfType<AIContent>().ToList())
+                toolResult.StructuredContent is JsonElement structured ? structured :
+                toolResult.Content.Count == 1 ? toolResult.Content[0].ToAIContent() :
+                toolResult.Content.Select(c => c.ToAIContent()).OfType<AIContent>().ToList())
             {
                 Exception = toolResult.IsError is true ? new() : null,
             },
@@ -307,7 +310,7 @@ public static AIContent ToAIContent(this ResourceContents content)
 
         AIContent ac = content switch
         {
-            BlobResourceContents blobResource => new DataContent(blobResource.Data, blobResource.MimeType ?? "application/octet-stream"),
+            BlobResourceContents blobResource => new DataContent(blobResource.DecodedData, blobResource.MimeType ?? "application/octet-stream"),
             TextResourceContents textResource => new TextContent(textResource.Text),
             _ => throw new NotSupportedException($"Resource type '{content.GetType().Name}' is not supported.")
         };
@@ -380,21 +383,25 @@ public static ContentBlock ToContentBlock(this AIContent content)
 
             DataContent dataContent when dataContent.HasTopLevelMediaType("image") => new ImageContentBlock
             {
-                Data = System.Text.Encoding.UTF8.GetBytes(dataContent.Base64Data.ToString()),
+                Data = MemoryMarshal.TryGetArray(dataContent.Base64Data, out ArraySegment<char> segment)
+                    ? new string(segment.Array!, segment.Offset, segment.Count)
+                    : new string(dataContent.Base64Data.ToArray()),
                 MimeType = dataContent.MediaType,
             },
 
             DataContent dataContent when dataContent.HasTopLevelMediaType("audio") => new AudioContentBlock
             {
-                Data = System.Text.Encoding.UTF8.GetBytes(dataContent.Base64Data.ToString()),
+                Data = MemoryMarshal.TryGetArray(dataContent.Base64Data, out ArraySegment<char> segment)
+                    ? new string(segment.Array!, segment.Offset, segment.Count)
+                    : new string(dataContent.Base64Data.ToArray()),
                 MimeType = dataContent.MediaType,
             },
 
             DataContent dataContent => new EmbeddedResourceBlock
             {
                 Resource = new BlobResourceContents
                 {
-                    Blob = System.Text.Encoding.UTF8.GetBytes(dataContent.Base64Data.ToString()),
+                    DecodedData = dataContent.Data,
                     MimeType = dataContent.MediaType,
                     Uri = string.Empty,
                 }
@@ -414,21 +421,51 @@ public static ContentBlock ToContentBlock(this AIContent content)
                 Content =
                     resultContent.Result is AIContent c ? [c.ToContentBlock()] :
                     resultContent.Result is IEnumerable<AIContent> ec ? [.. ec.Select(c => c.ToContentBlock())] :
-                    [new TextContentBlock { Text = JsonSerializer.Serialize(content, McpJsonUtilities.DefaultOptions.GetTypeInfo<object>()) }],
-                StructuredContent = resultContent.Result is JsonElement je ? je : null,
+                    [new TextContentBlock { Text = "" }],
+                StructuredContent =
+                    resultContent.Result is JsonElement je ? je :
+                    resultContent.Result is null ? null :
+                    JsonSerializer.SerializeToElement(resultContent.Result, McpJsonUtilities.DefaultOptions.GetTypeInfo<object>()),
             },
 
-            _ => new TextContentBlock
+            _ => CreateJsonResourceContentBlock(content)
+        };
+
+        static ContentBlock CreateJsonResourceContentBlock(AIContent content)
+        {
+            byte[] jsonUtf8 = JsonSerializer.SerializeToUtf8Bytes(content, McpJsonUtilities.DefaultOptions.GetTypeInfo(typeof(object)));
+
+#if NET
+            int maxLength = Base64.GetMaxEncodedToUtf8Length(jsonUtf8.Length);
+#else
+            int maxLength = ((jsonUtf8.Length + 2) / 3) * 4;
+#endif
+
+            byte[] base64 = new byte[maxLength];
+            if (Base64.EncodeToUtf8(jsonUtf8, base64, out _, out int bytesWritten) != System.Buffers.OperationStatus.Done)
             {
-                Text = JsonSerializer.Serialize(content, McpJsonUtilities.DefaultOptions.GetTypeInfo(typeof(object))),
+                throw new InvalidOperationException("Failed to base64-encode JSON payload.");
             }
-        };
+
+            ReadOnlyMemory<byte> blob = base64.AsMemory(0, bytesWritten);
+
+            return new EmbeddedResourceBlock
+            {
+                Resource = new BlobResourceContents
+                {
+                    Uri = string.Empty,
+                    MimeType = "application/json",
+                    BlobUtf8 = blob,
+                },
+            };
+        }
 
         contentBlock.Meta = content.AdditionalProperties?.ToJsonObject();
 
         return contentBlock;
     }
 
+
     private sealed class ToolAIFunctionDeclaration(Tool tool) : AIFunctionDeclaration
     {
         public override string Name => tool.Name;

src/ModelContextProtocol.Core/Authentication/ClientOAuthProvider.cs

@@ -6,6 +6,7 @@
 using System.Buffers.Text;
 #endif
 using System.Diagnostics.CodeAnalysis;
+using ModelContextProtocol.Internal;
 using System.Net.Http.Headers;
 using System.Security.Cryptography;
 using System.Text;
@@ -581,8 +582,9 @@ private async Task PerformDynamicClientRegistrationAsync(
             Scope = GetScopeParameter(protectedResourceMetadata),
         };
 
-        var requestJson = JsonSerializer.Serialize(registrationRequest, McpJsonUtilities.JsonContext.Default.DynamicClientRegistrationRequest);
-        using var requestContent = new StringContent(requestJson, Encoding.UTF8, "application/json");
+        using var requestContent = new JsonTypeInfoHttpContent<DynamicClientRegistrationRequest>(
+            registrationRequest,
+            McpJsonUtilities.JsonContext.Default.DynamicClientRegistrationRequest);
 
         using var request = new HttpRequestMessage(HttpMethod.Post, authServerMetadata.RegistrationEndpoint)
         {

src/ModelContextProtocol.Core/CanceledTokenSource.cs

@@ -0,0 +1,44 @@
+using System.Diagnostics.CodeAnalysis;
+
+namespace ModelContextProtocol.Core;
+
+/// <summary>
+/// A <see cref="CancellationTokenSource"/> that is already canceled.
+/// Disposal is a no-op.
+/// </summary>
+public sealed class CanceledTokenSource : CancellationTokenSource
+{
+    /// <summary>
+    /// Gets a singleton instance of a canceled token source.
+    /// </summary>
+    public static readonly CanceledTokenSource Instance = new();
+
+    private CanceledTokenSource()
+        => Cancel();
+
+    /// <inheritdoc/>
+    protected override void Dispose(bool disposing)
+    {
+        // No-op
+    }
+
+    /// <summary>
+    /// Defuses the given <see cref="CancellationTokenSource"/> by optionally canceling it
+    /// and replacing it with the singleton canceled instance.
+    /// The original token source is left for garbage collection and finalization provided
+    /// there are no other references to it outstanding if <paramref name="dispose"/> is false.
+    /// </summary>
+    /// <param name="cts"> The token source to pseudo-dispose. May be null.</param>
+    /// <param name="cancel"> Whether to cancel the token source before pseudo-disposing it.</param>
+    /// <param name="dispose"> Whether to call Dispose on the token source.</param>
+    [SuppressMessage("Design", "CA1062:Validate arguments of public methods")]
+    public static void Defuse(ref CancellationTokenSource cts, bool cancel = true, bool dispose = false)
+    {
+        // don't null check; allow replacing null, allow throw on attempt to call Cancel
+        var orig = cts;
+        if (cancel) orig.Cancel();
+        Interlocked.Exchange(ref cts, Instance);
+        // presume the GC will finalize and dispose the original CTS as needed
+        if (dispose) orig.Dispose();
+    }
+}

src/ModelContextProtocol.Core/Client/McpClientTool.cs

@@ -142,8 +142,27 @@ result.StructuredContent is null &&
                 case 1 when result.Content[0].ToAIContent() is { } aiContent:
                     return aiContent;
 
-                case > 1 when result.Content.Select(c => c.ToAIContent()).ToArray() is { } aiContents && aiContents.All(static c => c is not null):
-                    return aiContents;
+                case > 1:
+                    AIContent[] aiContents = new AIContent[result.Content.Count];
+                    bool allConverted = true;
+
+                    for (int i = 0; i < aiContents.Length; i++)
+                    {
+                        if (result.Content[i].ToAIContent() is not { } c)
+                        {
+                            allConverted = false;
+                            break;
+                        }
+
+                        aiContents[i] = c;
+                    }
+
+                    if (allConverted)
+                    {
+                        return aiContents;
+                    }
+
+                    break;
             }
         }
 

src/ModelContextProtocol.Core/Client/McpHttpClient.cs

@@ -32,10 +32,9 @@ internal virtual async Task<HttpResponseMessage> SendAsync(HttpRequestMessage re
 #if NET
         return JsonContent.Create(message, McpJsonUtilities.JsonContext.Default.JsonRpcMessage);
 #else
-        return new StringContent(
-            JsonSerializer.Serialize(message, McpJsonUtilities.JsonContext.Default.JsonRpcMessage),
-            Encoding.UTF8,
-            "application/json"
+        return new ModelContextProtocol.Internal.JsonTypeInfoHttpContent<JsonRpcMessage>(
+            message,
+            McpJsonUtilities.JsonContext.Default.JsonRpcMessage
         );
 #endif
     }

src/ModelContextProtocol.Core/Client/SseClientSessionTransport.cs

@@ -1,5 +1,6 @@
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Logging.Abstractions;
+using ModelContextProtocol.Core;
 using ModelContextProtocol.Protocol;
 using System.Diagnostics;
 using System.Net.Http.Headers;
@@ -18,7 +19,7 @@ internal sealed partial class SseClientSessionTransport : TransportBase
     private readonly HttpClientTransportOptions _options;
     private readonly Uri _sseEndpoint;
     private Uri? _messageEndpoint;
-    private readonly CancellationTokenSource _connectionCts;
+    private CancellationTokenSource _connectionCts;
     private Task? _receiveTask;
     private readonly ILogger _logger;
     private readonly TaskCompletionSource<bool> _connectionEstablished;
@@ -114,7 +115,7 @@ private async Task CloseAsync()
             }
             finally
             {
-                _connectionCts.Dispose();
+                CanceledTokenSource.Defuse(ref _connectionCts, dispose: true);
             }
         }
         finally

src/ModelContextProtocol.Core/Client/StdioClientSessionTransport.cs

@@ -7,7 +7,7 @@ namespace ModelContextProtocol.Client;
 /// <summary>Provides the client side of a stdio-based session transport.</summary>
 internal sealed class StdioClientSessionTransport(
     StdioClientTransportOptions options, Process process, string endpointName, Queue<string> stderrRollingLog, ILoggerFactory? loggerFactory) :
-    StreamClientSessionTransport(process.StandardInput.BaseStream, process.StandardOutput.BaseStream, encoding: null, endpointName, loggerFactory)
+    StreamClientSessionTransport(process.StandardInput.BaseStream, process.StandardOutput.BaseStream, endpointName, loggerFactory)
 {
     private readonly StdioClientTransportOptions _options = options;
     private readonly Process _process = process;