I set the webm profile as the default (something I thought I had tried before) and it works with Plex! I also tried IE and when I clicked to play the channel it asked me to open/save the file so when I clicked "open" it opened VLC (separately from IE) and started streaming video (again, with the webm as default). So that seems to have done the trick. I will say, however that streaming from Plex to a wireless Roku box did have a few buffering interruptions but I tested it with Plex on my wired desktop and it seems to be working fine, so the wireless seems to be the problem there.
I still haven't got it to stream with Firefox (what I usually use). Is Firefox a known issue or any ideas how to make that browser work?