I spent a year building a tool that assesses Microsoft 365 tenants automatically. The most useful thing I can pass on isn’t what worked. It’s the list of things that can’t be read at all.
Everyone who automates this runs into the same wall eventually. I couldn’t find the list written down anywhere, so here it is.
If you’re building tooling, checking what a vendor claims to cover, or just wondering why your script keeps returning nothing for a setting you can plainly see in the portal — this is probably why.
1. Entra Password Protection
The custom banned password list and the on-premises agent status aren’t in Graph. You can see them in the Entra portal under Authentication methods, and that’s the only place you can see them.
So if a tool tells you it’s checked whether your organisation blocks weak passwords, it’s reading something adjacent or it’s guessing.
2. Defender for Cloud Apps policy configuration
The alerts come through Graph fine. The policies that generate them don’t, and neither do Cloud Discovery settings.
This matters more than it first appears. A tenant with an MDCA licence and nothing configured looks exactly the same through the API as one that’s fully set up. You can see the licence. You can’t see whether anyone used it.
3. Entra diagnostic settings
Whether your sign-in and audit logs are being exported to a Log Analytics workspace, a storage account or an event hub — not readable through Graph.
That one’s awkward, because a lot of frameworks ask about log retention. PCI DSS 10.1, SOC 2 CC7.2, ISO 27001 A.8.15, CERT-In Direction (iv). You can be asked to show the export is configured and have no clean way to prove it.
Worth adding: someone on Reddit pointed out a better approach than I’d been taking. Rather than looking for the switch, probe the sink. Query the workspace for recent rows — AuditLogs | where TimeGenerated > ago(6h) | summarize count(). Rows flowing is stronger evidence than a config flag, and it catches the failures a flag would miss: retention set too short, a stale workspace ID, a storage account filling up. Verify the sink, not the switch. Good advice, and I’ve taken it.
4. Tenant Allow/Block List
The Defender for Office 365 Tenant Allow/Block List isn’t available app-only through the documented endpoint.
Stale allow entries are a genuine risk — an address allowed for one investigation two years ago is still allowed today — and there’s no tidy way to list them automatically.
5. Information barriers
Get-InformationBarrierPolicy needs the OrganizationManagement role group, and that role group can write. There’s no read-only path to the same information.
Which means a read-only tool can’t check information barriers without asking for more privilege than it should have. That’s a worse trade than leaving the control unassessed. If a vendor tells you they assess information barriers read-only, it’s worth asking how.
6. And the one that cost me three days
This one isn’t a missing API. It’s a documented one that doesn’t behave the way the documentation says.
The Exchange Online admin API docs point ISVs at Exchange.ManageAsAppV2. Grant it, consent it, mint your token — the role shows up in the JWT exactly as you’d expect.
Then every call fails. HTTP 403, empty body, no error code, no clue what’s missing.
The endpoint everyone actually uses is this one:
POST https://outlook.office365.com/adminapi/beta/{tenantId}/InvokeCommand
It’s the same endpoint the official ExchangeOnlineManagement PowerShell module calls. And it authorises on V1 — Exchange.ManageAsApp — not V2.
With V2 alone you get nothing. Not a permissions error naming the role. Not a scope mismatch. Just a 403 with an empty body.
What made it hard was that the official PowerShell module fails in exactly the same way. So I ruled out my own code early and spent the next two days looking at token audiences, tenant consent, regional endpoints — everything except the role version.
Two more things, if you go down this road:
Admin consent in Entra isn’t enough on its own. The app role has to be assigned to your service principal in each customer tenant, through Graph appRoleAssignments or PowerShell. Consenting the app registration gets you a token that quietly doesn’t carry the role.
The role id, if you need it: dc50a0fb-09a3-484d-be87-e023b12c6440
Why any of this matters if you’re buying rather than building
These six make a reasonable test. Ask a vendor what their tool does with them.
There are three honest answers. It reads them some other way. It doesn’t assess them. Or it assesses them and tells you it couldn’t.
The dishonest answer is scoring an unreadable control as a pass. You get a better-looking number and a report that’s quietly wrong — and you act on it, which is worse than having no report at all.
An auditor can work with “we couldn’t read this, and here’s why”. Nobody can work with a pass that nothing actually checked.
I ran into these building Kyvoi, a Microsoft 365 security and compliance assessment tool. If you’ve found a way round any of them, I’d genuinely like to hear it.