Cortana Skills are Gone

Well, it is over.

Cortana is dead.

In February, Microsoft killed third party skills.  Though you can skill invoke the bots, the interface from Windows or Invoke is dead.

So the question is ... now what.  Relaunch the skills I have as plug ins for Teams chat? Or just wait... continue to build on Google Actions. 

Update: I spent 5 minutes and hooked the bot webchat to the Art of War and Mediations pages. So... at least you can still see the botframework part in action!

Media, White Hats, and Black Hats

ARS Technica has an article with detailed instructions on how to record information, or fake phishing, using Alexa and Google Assistant. It is a blueprint of how to build rogue skills that pass certification but do naughty things.  I know at Microsoft, one of the last things Cortana did was implement developer vetting. It irritates me that articles like this, that have good intentions, not only expose holes to black hats, but make real developers lives harder.

When CERTS expire...

Today I noticed my SSL cert expired. I received notice mid-July, but as with everything, I checked if everything still worked at that time - it did - so conveniently forgot.

This meant likely yesterday, my webs server anounced it was not secured, and all my Cortana skills stopped working with an obscure error message stopped working. (The error message buried at the bottom of the message was [bot service response null].)

So I fixed it by

• downloading my new cert
• figuring out how to use psftp to upload them to my server given I'd obfuscated sshd configuration
• uploading my new crt files and moving them to /etc/ssl/certs
• updating the various entries in /etc/apache2/sites-available conf files that pointed at the cert

So all the Cortana skills are running again.  I am surprised that the automated tests Microsoft runs didn't send me a nasty email saying I was down.  Ah....

Debug C# Cortana Skill Locally

I added a video on how to create a Skill in the cloud, compile it there, then suck it back to a desktop, set debug break points, and continue to debug locally.It doesn't have commentary yet, but I'll get to it.

90 day new user estimates

Looking at how many users on Google or Microsoft are using my skills.

	Microsoft	Google
Art of War	23	540
Safety Moments	54	101
Meditations	12	67
Ambient Sounds	65
AutoMate	5

Art of War wins on Google. Ambient Sounds wins on Microsoft. And hey, for all this effort, at least Google is sending me a T-Shirt. I suspect Colossal Cave will beat them all...

Google tells me on average 15 new users a day try Art of War.

Cortana Hosting Off Azure

Yesterday I decided to start moving my skills off Azure. Messaging is still handled by bot service, but the bot lives elsewhere. This is in no way a comment on Azure App Services. It has everything to do with me paying for a GoDaddy VM and I want to have everything in one place. Sure, its not scalable in its current form, but who cares when you have insignificant TPS.

The things that Azure App Services give you for free... a SSL certificate so you can use HTTPS (that you must for bots messaging) and a preconfigured IIS web server that is always managed (because its a container). But its the free tier restrictions... you can't set your web app and skill to "always on", so you get unloaded every 20 minutes. That means your skill will timeout when invoked, and that's a really crappy experience. I wonder how many people actually try a skill twice after it craters the first time? You also can't use web jobs to tickle the endpoint.

So, first step, install Apache. In hindsight, I should have paid GoDaddy the managed fee (that was reasonable, $50 or so) to configure Apache - with their bundled 509 Cert. What a pain... I don't miss running my own web servers.

Why run Apache? Because you want the web server to take care of the SSL for you. It is a lot of work to build a bot supporting HTTPS without a web server. So the trick is, get Apache to handle the secure request, and then tunnel it via proxy to a port without SSL.My node.js bots - they don't know anything about SSL.

How to have all my bots run in one place? Multiple proxies that redirect based on different paths. Instead of https://myserver/api/messages, we now have http://myserver/mybot/api/messages. And the latter redirects to different local ports via a proxy to http://localhost:3000/mybot/api/messages.

Then you have a choice of registering the Cortana skill via a Bot Channel Registration (the right way), or swap out the endpoint after creating a Web App Bot. Why would you want to do the latter? Updated: Because there is a bug in Bot Channel Registrations. Do not use Bot Channel Registration until the glitch is fixed. Create a web app bot, configure the Cortana channel, and then change the bots end point (and tombstone the Azure app service).

I will add my web.conf here when I have time. It is magic that I'll never remember. And the next trick is running node as a daemon. My VM is running Ubuntu server - so
bash -c 'nohup node app &>/dev/null & jobs -p %1'

Cortana Availability

Ever notice that it can take two invocations to a Cortana skill to get it to work on Azure?

This is because resources are unloaded if they are idle for 20 minutes.  The first invocation; you need to wait for the resources (web app service) to load. You can change this in basic and standard service plans by changing the app service’s settings to “Always On”. 

• https://docs.microsoft.com/en-us/azure/azure-monitor/app/monitor-web-app-availability

But if you are using a free tier, what do you do? Set up an availability test that pings your web server every 15 minutes via an Azure web job or Azure monitor. 

• https://docs.microsoft.com/en-us/azure/app-service/webjobs-create 
• https://docs.microsoft.com/en-us/azure/azure-monitor/app/monitor-web-app-availability

But be aware: keeping resources live can consume your credits!

Two months later...

Hello! It seems I have not posted in almost two months. Life always gets interesting. My in-laws and oldest daughter visited. Then there was the flu, head colds, and administrative issues... so my tinkering slowed down to nothing. I've barely even managed to explore my Christmas present: a Playstation Classic.

I have lots of topics I could write about, because between work and personal enlightenment, I've touched on a lot. I am more than happy to take direction on what to write about - because often times I tell myself no one reads what I write so there is no hurry in getting things down.

Cortana

I did get to push out work related samples on building against bot framework v4. It is different enough to cause people grief in upgrading, and v4 is slightly more complicated and less documented than v3. I've been thinking about writing a guide or giving some sample "search and replace" scripts to help. But that also seems too close to work to use my own time. So expect to see more Cortana stuff here:

• https://github.com/bw-kforce-ms for my experiments related to work (like a file based bot storage for V3)
• https://github.com/Microsoft/cortana-skills-samples (the official repository of samples I'm contributing to)

Unfortunately I hit a policy blocker on publishing skills - that has seen all my Cortana pet projects delayed or shelved.  However, I am happy that Safety Moments averages 6 requests a day on Windows.  Thanks to those who are using it!

Google Actions

I am happy about how Art of War is doing.  It gets about 60 users a week and sends about 200 messages. When I have time, I intend to get Colossal Cave launched on Google.  It was so close to being done on Cortana, and the server is written in Java and platform agnostic.

Enabling Colossal Cave on Cortana

I am back at it. The first step was getting a version of Colossal Cave running. I found Eric Raymond's official public port:

• https://gitlab.com/esr/open-adventure
• http://www.catb.org/~esr/open-adventure/

I compiled it for Windows (using MinGW-64) and made a few changes (removing unix curses and flushing input appropriately so it worked inside a pipe.)

The next step was putting together a Cortana dialog-based shell calling an external process to invoke it. The fun here is the node.js event loop. Reading and writing to the pipe did not happen when I expected, so the dialog was out of sync until I forced the Cortana dialog to happen after pipe write.

So I now how a demo-able version of Colossal Cave on an Invoke with Cortana.

What is next before publishing?

1. Support multiple, concurrent users (yay fork).
2. Support save game.
3. Improve the Windows 10 prompt to repeat the text.

How to invoke Cortana channel actions in node.js

Ugh.  It took me a while to figure it out, and I had to reverse engineer some code to do it. There were breadcrumbs, but of course being a newb it wasn't immediately obvious.

Botframework allows you to connect to channels. You are likely invoked from a channel. That means you can send meta-data back to that channel. In the previous version of the framework, you called a method off the Message called channelData to set the JSON response.  But channelData is deprecated in favor of the new sourceEvent method.  So forget the "Launching apps or websites from a Cortana skill" example (that is in C# and the old way)...

The next hint is in the documentation. sourceEvent takes a map.  The old version just took some action in a JSON with a type : LaunchUri and uri : "http://whatever.com/"

What is this map? Well, you can have your bot connected to one or more channels (or "*" for all.) The source code implementing sourceEvent was the final piece. I found examples setting the facebook channel, or using directline... But what is the channel name for Cortana?  Well, it is cortana.

Wrapping the JSON up underneath the channel name and magically Cortana will launch the app.

msg.sourceEvent( 
{ cortana : {
    action : { 
      type : "LaunchUri",
      uri : "https://www.octavianit.com"  
      }
    } 
  } )

But be careful as Cortana will close the channel after invoking the app (that the protocol maps to).  And you should consider that speakers (screenless devices) will not support this behavior and should have a UX alternative.

Building your own Cortana Music Player

Ever since Amazon killed the cloud Music Storage subscription, I've been annoyed. I uploaded my 2000+ CD collection to the cloud to preserve it and allow me to access it from anywhere.

I've been looking for an alternative, but didn't find anything as cost effective as Amazon's dead program ($25 for the year? a steal). Google had a Google Music Play plan that included YouTube Red for $15/mo. But that cost made me uncomfortable (that, and being on a monthly plan.)

But I believe I found a solution!  OneDrive... you get 1TB of storage with an Office 365 account for $99/yr. That is value, and you get Office too.

But here is what I find more exciting.

I got my Invoke today and had to (like really) see if I could build my own music player... Out of the box Cortana will only let you hook up to streaming services. But you can't play music from your PC or OneDrive (unless you BlueTooth to the device).

How hard can it be to build a skill to play your music from OneDrive? As it turns out - not hard at all.

Amazon Music continually griefed me because it never kept my songs together in their albums when imported (as were imported via iTunes). But when I synced my library to OneDrive - the directory structure is intact. And as it happens, the OneDrive REST API will let you retrieve your directories and walk the files... and you can use your MSA authentication to keep it all personal or share those files...

So, how hard is it to get Cortana and botframework to play an MP3 you have stored on OneDrive?  This easy.

var audioCard = new builder.AudioCard(session)
        .media([
            { url : 'https://onedrive.live.com/download?cid=00E75C36F57E8A5B&resid=E75C36F57E8A5B%216254&authkey=AEAEHi1WUjheHj4' }]);  
var msg = new builder.Message(session)
 .addAttachment(audioCard)
 .text('Now playing Nephatiti by 808 State)
 .speak('<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">Playing Nephatiti by <say-as interpret-as="number_digit">808</say-as> State.</speak>');
session.send(msg).endConversation();

Edit: It is extremely important to end the conversation after sending an audio card on Windows because if Cortana has a dialog going, regardless of any input hint, the volume will be set low. Ending the conversation keeps the volume at 100% at the expense of disconnecting Cortana from the bot.  Also, Cortana will ignore every field in the audio card (like title).

What will be even more fun will be using the language recognition services to solve another Alexa pet peeve... search for my classic and latin music titles!

End to End Developer example video of basic Cortana skill

For those wanting a visual walk through of the process to create a Cortana skill with botframework!

Uses node.js.

Get your Azure QnA Bot to Speak with Cortana

Building a QnA bot and hooking it up to Cortana is simple in node.js under BotFramework V3.

If you have a FAQ that is in Q: A: format you can import it via the Azure QnA Maker tools and auto-create a "knowledge base".  No coding required. You can do this here https://www.qnamaker.ai.

The next step is creating your bot. Microsoft has standardized on their botframework to do this. https://dev.botframework.com is your gateway.

If you are like me and like the simplicity of node.js then pick the QnA template.

Go back to qnamaker and view the code to extract the QnA keys and host.

Then go back to the Azure portal and update your bot for the Application Settings blade sections shown here.

The test web app will now be successfully linked to your QnA bot! But don't forget the last step...

Go to your Channels blade and set up Cortana. Then, go to the build blade and open the online editor. In the app.js code, you will see that the template uses the standard QnA dialog builder - that does not say the resulting answers back with the Cortana speech channel. Add an override like this.

(See GitHub QnAMaker patch for a V4 node example.)

There you have it. Now "Hey Cortana, ask Bernie Question Bot Test what is a dwarf planet?"

For C#, it is slightly more complicated. You need to subclass the the BasicQnAMakerDialog with something that sends a message with speak attached. This is done as an inner class of RootDialog (from the V3 C# template that comes with Azure Web App Bots.

    // Dialog for QnAMaker GA service
    [Serializable]
    public class BasicQnAMakerDialog : QnAMakerDialog
    {
        // Go to https://qnamaker.ai and feed data, train & publish your QnA Knowledgebase.
        // Parameters to QnAMakerService are:
        // Required: qnaAuthKey, knowledgebaseId, endpointHostName
        // Optional: defaultMessage, scoreThreshold[Range 0.0 – 1.0]
        public BasicQnAMakerDialog() : base(new QnAMakerService(new QnAMakerAttribute(RootDialog.qnaAuthKey, RootDialog.qnaKBId, "No good match in FAQ.", 0.5, 1, RootDialog.endpointHostName)))
        { }

        // Override to also include the knowledgebase question with the answer on confident matches
        protected override async Task RespondFromQnAMakerResultAsync(IDialogContext context, IMessageActivity message, QnAMakerResults results)
        {
            if (results.Answers.Count > 0)
            {
                IMessageActivity response = context.MakeMessage();
                response.Text = "Here is the match from FAQ:  \r\n  Q: " + results.Answers[0].Questions[0] + "  \r\n A: " + results.Answers[0].Answer;
                response.Speak = response.Text;
                response.InputHint = "acceptingInput";
                await context.PostAsync(response);
            }
        }
    }

(See GitHub QnAMaker patch for a V4 C# example.)

Building Bots in Azure

I am starting to build Cortana bots. So far its been interesting.

I had trouble building the Azure function bots from template. The node.js example for the simple "echo" bot... First crack, the Azure "Test in Web Chat" didn't work. Errors implied there was a permission issue.

I tried again, and had a deployment error on the bot function template.

I tried again, and on the third try it deployed. To my knowledge, I did nothing differently.

The advantage of a function bot over a web app bot is supposedly pay per invocation (that should be cheaper, right)? The issue with the example code for node.js on the function bot is that every potential used library is embedded in the index.js code where the two line "echo" functionality is embedded in the middle!

WHY?  Why? Well, javascript as a language doesn't have a '#include' statement. Client side, you do the includes on the document for your browser to take care of.

In node, we have requires... that allows us to load modules. But why is this not used in the function bot example? I figure its a work around. But the down side is this: every time I edit the function in the portal for this example, I am touching a 200K line file!