GitHub Copilot Plan Mode Telemetry: From “Misc” Bucket to Actionable Metrics

GitHub Copilot Plan Mode Telemetry is a new observability layer that decouples strategic planning interactions from generic "Custom" chat metrics in the API and dashboard. It allows engineering leaders to specifically track how developers are using the tool for architectural reasoning and multi-step tasks rather than just code generation.

The "Misc" Bucket Problem

Back in 2009, when I was cutting my teeth at a boutique consulting firm in the Valley, I spent weeks parsing terabytes of server logs for a Fortune 100 client. We had this one persistent headache: a metadata field labeled "Other." It accounted for 40% of their traffic, but nobody knew what it actually was. It turned out to be a mix of three critical, distinct user behaviors, but because the engineering team had lazy tagging, the data was useless for decision-making.

I see the same pattern in how we track AI adoption today. Until recently, GitHub Copilot bundled a lot of distinct activities—specifically "Plan Mode"—into a generic "Custom" bucket. If you were trying to figure out if your team was actually using the tool for high-level planning or just asking it to write boilerplate, you were flying blind. That changed with the latest update to the usage metrics API.

I have spent the last decade building data pipelines, and honestly, granularity is the difference between a dashboard that looks pretty and one that actually saves you money. Here is what the new telemetry breakdown looks like and how to ingest it.

De-orphaning the "Plan Mode" Metric

The core of this update is a schema change in the API response. Previously, if a developer used Copilot to plan out a refactor, that activity was lumped under "Custom" usage. Now, it has been broken out into its own explicit category.

If you are pulling data via the API, you will see a new key: chat_panel_plan_mode. This appears under three specific parent keys in the JSON response:

• totals_by_feature
• totals_by_language_feature
• totals_by_model_feature

This matters because "Plan Mode" behaves differently than standard chat. It is slower, more token-intensive, and represents a different cognitive load. Separating these metrics lets you perform better observability evals on how your team interacts with the model during the design phase versus the implementation phase.

The "Dip" in Your Dashboards

A heads-up for anyone maintaining internal dashboards: you are going to see a sudden drop in your "Custom" metrics. This is not a bug. It is a correction.

When I built the initial analytics engine for SocketStore, we had to warn users constantly about schema changes skewing historical data. In this case, the telemetry that used to inflate the "Custom" category is now properly attributed to Plan Mode. Expect charts like "Requests per chat mode" and "Model usage per chat mode" to shift significantly. You might want to annotate your graphs to explain that this is a reclassification, not a drop in usage.

IDE Support Matrix

Not all environments are sending this telemetry yet. In my experience, disparate client versions are the biggest cause of data drift. If you are auditing pull request throughput against AI usage, you need to know which IDEs are actually reporting the data.

It is worth noting that while VS Code Insiders has it, the general release is trailing slightly. If your organization enforces a strict "stable builds only" policy, your data might be incomplete for a few weeks.

Enterprise Controls and Agent Activity

Beyond just Plan Mode, the scope of what we can track is expanding to include the agent control plane. GitHub has rolled out Enterprise AI Controls which give you a lot more governance over how agents interact with your codebase.

The updated metrics now include data on:

• CLI Activity: Terminal-based interactions are no longer a black box.
• Agent Controls: You can see if restrictions on public code suggestions are triggering frequent blocks.
• Pull Request Throughput: The org-level metrics API now has parity here, meaning you can finally correlate Copilot usage directly with merge frequency.

If you are running a large team, you should be piping this data into your own warehouse. Relying on the default UI is fine for a quick check, but for real analysis—like calculating the cost per line of code or API rate limit consumption—you need raw access.

Integrating with Your Data Stack

Getting this data out of GitHub and into a usable format is usually where teams get stuck. The API is straightforward, but handling the rate limits and normalization takes some work.

When we designed the Socket-Store Blog API, we prioritized unified interfaces because nobody wants to write five different connectors. Similarly, you should treat these Copilot metrics as just another stream. I recommend setting up a nightly job to pull the JSON, flatten the totals_by_model_feature array, and store it in a time-series database. This allows you to overlay AI usage against your deployment frequency.

For those interested in the raw implementation, check out our API docs for examples of how to structure similar data pipelines.

Need Better Data Visibility?

At SocketStore, we specialize in unified analytics APIs. While our bread and butter is social media data, the principles of high-uptime (99.9%) data ingestion apply everywhere.

If you are struggling to merge your engineering metrics with your business KPIs, or if you just need a robust way to handle high-volume API streams without hitting rate limits, take a look at our platform. We strip away the complexity of connector maintenance so you can focus on the insights.

Check out our pricing to see how we can help your team, or visit the main site to learn more.

Frequently Asked Questions