Unable to send messages after 5.5.2 update

[eanthonyq]

We updated to 5.5.2 on October 15th and haven't been able to send messages from the Application since.

I've reviewed Issues #1061, #1551, #1608 and #1593 and tried the steps taken in them as the errors are similar to what was referenced in them.

1. Validated all the Environment variables from the installation guide in the App Service Configuration as well as the 3 Function Apps.
2. App Registration IDs and associated Secrets Validated
   GraphAppId -- GraphAppPassword
   AzureAd:ClientId -- AzureAd:ClientSecret
   UserAppId -- UserAppPassword
   AuthorAppId -- AuthorAppPassword
3. Reviewed App Registration Manifests
4. Redeployed 5.5.2 Client and Author Apps
5. Resyncs done to GIT repository
6. Restarts of web applications/functions

The errors I'm seeing in the Failures reports are 401 errors to a few API calls:
GET /api/draftnotifications/scheduledDrafts
GET /api/draftnotifications/
GET /api/sentnotifications

As well as OrchestrationFailures in the SyncRecipientsOrchestrator Function.

Example OrchestrationFailures Error:
Exception while executing function: SyncRecipientsOrchestrator Orchestrator function 'SyncRecipientsOrchestrator' failed: The activity function 'BatchRecipientsActivity' failed: "Object reference not set to an instance of an object.". See the function execution logs for additional details.

From the resolved tickets this seems to be a secret related issue, but we've checked a few times that the active secrets in the KeyVault for the App Registrations are valid.

Any guidance would be greatly appreciated.

[v-jaygupta]

Hello @eanthonyq ,
Thanks for reaching out to us!
Could you please go thorugh the v5 migration process as mentioned here.

Also, please share deployment status screenshots of all the Function apps and App service from the deployment center. Here is the reference for App service status. Please do the same for 3 function apps(Data Function, Send Function, Prep Function)

[eanthonyq]

Hi @v-jaygupta,

I've gone back through 5.x upgrade steps.

1. Confirmed the .NET Version
   

2. Confirmed Runtime Versions for functions
   
   
   

3. Confirmed Organization specific configuration settings for Web Application and Website Default Node setting
   
   

4. Resynced Web Apps and 3 Function Apps
   
   
   
   

It looks like the Function Apps were not at the same version as the Web Application. They all look to be at the same versions now.

5. Restarted Web Application and Function Apps

After their restarts I confirmed in the Author Application in Teams that all organization specific configurations were displaying and configured I was able to delete messages from the Application.

Resending a message to users now produces the same Orchestrator Error but there is additional Warning Information. That was not displayed before. The actual being displayed in the Warning is the App ID for the UserAppID App Setting in the Web Application and 3 Function App Environmental Variables.

Errors
Failed to prepare the message for sending:The orchestrator function 'SyncRecipientsOrchestrator' failed: "The activity function 'BatchRecipientsActivity' failed: "Object reference not set to an instance of an object.". See the function execution logs for additional details.". See the function execution logs for additional details.
Warnings
Failed to get members for team @thread.tacv2: Failed to acquire token for client credentials. (A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app ''. Trace ID: Correlation ID: Timestamp: 2024-11-11 14:24:27Z)

[tiwariShiv7]

Hi @eanthonyq, From the screenshot it seems like you are using a custom repository here. Can you please recheck and send screenshot of Deployment Center of any of the function app / Web app ?

Please note that if you are not using Official version of the Company Communicator, it will be out of our Support Scope as per our Support Policy

[eanthonyq]

Hi @tiwariShiv7, It seems like there are 2 repository URLs being used across the Web App and 3 Function Apps.

Web App and the -data-fuction are using:
https://github.com/OfficeDev/microsoft-teams-company-communicator-app.git

The -prep-function and the -function are using:
https://github.com/OfficeDev/microsoft-teams-apps-company-communicator

When I follow the link to the Branch from the Deployment Center on any of those I am taken to the same URL:
https://github.com/OfficeDev/microsoft-teams-apps-company-communicator

If this is not the official version could you help me with the steps to link it to the correct one?

[tiwariShiv7]

The correct URL is https://github.com/OfficeDev/microsoft-teams-company-communicator-app.git
Okay I can see in the history of Code, there are commits from Derek. Please allow me little time ~36 hours. I'll get back to you within this time

[tiwariShiv7]

Hi @eanthonyq, can you please provide your ID to connect over Teams along with your time zone?

[eanthonyq]

Hi @tiwariShiv7, I can be reached at [email protected] and I'm on EST.

[tiwariShiv7]

Hi @eanthonyq We will soon start the conversation over Teams. Meanwhile request you to please update all the Function apps and web app repository URL to "https://github.com/OfficeDev/microsoft-teams-company-communicator-app.git" and perform a one time sync, Wait for 24 hrs tp reflect the changes. If it still doesn't work please let me know, I'll reach to you on Teams after this.

[eanthonyq]

Hi @tiwariShiv7 I've changed the GIT URLs for the functions to all point to the provided URL and performed resyncs. I retried sending a message through communicator after 24 hours and am still receiving the same error:

Errors
Failed to prepare the message for sending:The orchestrator function 'SyncRecipientsOrchestrator' failed: "The activity function 'BatchRecipientsActivity' failed: "Object reference not set to an instance of an object.". See the function execution logs for additional details.". See the function execution logs for additional details.
Warnings
Failed to get members for team @thread.tacv2: Failed to acquire token for client credentials. (A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app .

[v-milibiswas]

Hello @eanthonyq ,
As per the logs invalid client secret, it seems that client secret got expired. To renew the secret below are the detailed steps for creating a new client secret for the CC app:

• Go to Azure Portal
• Go to App registration
• Create a new client secret and grab secret value. Here is the reference.

• Then go to resource group where your CC app is deployed. Select key vault from list of resources.
• Click on Secrets under Objects section.

If the secrets are not visible:-
Go to Access Policies Click on Create

Search for Key, Secret & Certificate Management.

Under Principal, give the Id through which one can view the secrets

3rd step is optional
Go for Review+Create
• Then go to Secrets. Open author/user/graph app password
• Click new version

• Paste the secret value that you have copied and click on create

• Grab the new secret identifier
• Paste it in the configuration sections of app service, 3-function apps. Here is the reference of updating the author app password in app service configuration.

• Click on AuthorAppPassword and paste the secret identifier

• Then Apply
• Follow the same procedure for the user app password and graph app password updating.
• For graph app, please don’t forget to update the password in AzureAd Client Secret as well.