Add support for googlebot's indexifembedded tag when using App Router's Metadata Robots object

[FranCarstens]

Goals

1. Allow publishers to signal indexability of iframe, or similar HTML tag, content to googlebot

Non-Goals

1. Does not aim to address any other metaData related functionality

Background

The indexifembedded tag addresses a common issue that especially affects media publishers: while they may want their content indexed when it's embedded on third-party pages, they don't necessarily want their media pages indexed on their own. Because they don't want the media pages indexed, they currently use a noindex tag in such pages. However, the noindex tag also prevents embedding the content in other pages during indexing.

Next's Metadata already supports googlebot tags, but this specific tag is stripped when included in the robots configuration object. This update would add support for this specific tag. It should:

• Allow indexifembedded in the configuration object under the googlebot key
• Enforce the noindex and indexifembedded relationship i.e. noindex is required if indexifembedded is present

https://nextjs.org/docs/app/api-reference/functions/generate-metadata#robots
https://developers.google.com/search/blog/2022/01/robots-meta-tag-indexifembedded

Proposal

I would be happy to contribute a PR for this.

[FranCarstens]

Looks like this is supported and something's wrong on our side.