Guest: Kevin Schofield - Microsoft Research
Guest: Eric Horvitz - Microsoft Research
Guest: Andy Wilson - Microsoft Research
Host: Robert Scoble - The ScobleShow

Robert Scoble - The ScobleShow
So, who are you?

Kevin Schofield - Microsoft Research
Who am I? I'm Kevin Schofield.

Robert Scoble - The ScobleShow
I'm (Inaudible) before on Channel 9, but this isn't Channel 9 anymore.

Kevin Schofield - Microsoft Research
This is not Channel 9 anymore. I'm in Microsoft Research and I handle pretty much everything for MSR, does involve actually managing researcher, so...

Robert Scoble - The ScobleShow
Yup.

Kevin Schofield - Microsoft Research
Tech transferring of product groups, talking to customers, or academic programs, all sorts of those stuffs.

Robert Scoble - The ScobleShow
Why am I here, you guys just...

Kevin Schofield - Microsoft Research
Why are you...

Robert Scoble - The ScobleShow
I don't know. I missed the rain.

Kevin Schofield - Microsoft Research
No, this is actually the cool thing about doing in MSR is, we're like the world's largest computer science department, we very openly publish papers, we talk about the stuff we do, we love talking about the stuff that we do.

Robert Scoble - The ScobleShow
But you guys just celebrated you 15th birthday.

Kevin Schofield - Microsoft Research
We just celebrated our 15th anniversary, 15 years, we started in 1991.

Robert Scoble - The ScobleShow
So, let's go in. You got the blue badge, I don't have one of those things.

Kevin Schofield - Microsoft Research
I got the blue badge. Let's actually go through, maybe we can cut through the cafeteria and go to other building next door, and you can see all the deep dark secrets of the Microsoft Research cafeteria.

Robert Scoble - The ScobleShow
Oh, do we really? And (Inaudible) along with us with us too.

Kevin Schofield - Microsoft Research
Super secret high-tech potato chips, and oh yeah, oh yeah very, very secret wraps. We have the coolest cafeteria staff of any cafeteria. Smile, wave. They rock, they're totally, totally awesome.

Robert Scoble - The ScobleShow
You're taking me out to...

Kevin Schofield - Microsoft Research
I'm taking you out, yeah, we're actually to go out, just briefly outside.

Robert Scoble - The ScobleShow
I should know where, we're going to Eric Horvitz office, right?

Kevin Schofield - Microsoft Research
We're going to Eric Horvitz office, yes. So, one of the -- I'll just tell you a quick story here, so, I'm trying to put together sort of an event around our 15th anniversary, and invite some people and then show some stuff. Yeah, it's nice rainy day in Seattle, so, you're going to miss (Voice Overlap).

Robert Scoble - The ScobleShow
I missed the rain. That's why I came here.

Kevin Schofield - Microsoft Research
(Voice Overlap) not special for you today.

Robert Scoble - The ScobleShow
You've got to open the door.

Kevin Schofield - Microsoft Research
So, -- it's really -- I spent a couple of months of the summer trying to figure out, how do we tell the story of what we've done over 15 years in Microsoft Research. So, we actually went and did bunch of math and we started counting stuff, and we found that, we've published over 3700 papers in conferences over 15 years, that's like, almost one paper a day for every day that -- for every business day in 15 years, which is just amazing. If you figure out, it's like an average of ten pages per paper, that's like almost 200 reams of paper. Right, you figured, okay, how do you represent that? So you actually got office depot to bring over 37,000 pieces of paper, and we got a shot of Rick Rashid, our Senior Vice President sitting there on a Living Doc with 37,000 pages of paper, which is pretty amazing. Okay, right behind you, this is another big thing, we did sort of represent what's, what we've really done or what's sort of been the path for 15 years of Microsoft Research. We got this whole wall, maybe you can get the shot here.

Robert Scoble - The ScobleShow
Rocking.

Kevin Schofield - Microsoft Research
This whole wall here starting in 1991, you should love this whole picture of Eric here from 1991, that when big glasses were in. Then if you turn around and look, it's sort of stretches down the wall here for 15 years of everything that we've done. Early graphics work that we're doing, we've been doing all along. Jim Gray starting BARC right back in 1995, there's Roger Needham in our Cambridge Lab, work that we started doing with SQL Server. It just goes on and on -- actually, that end was really hard to do, the first five years because one of the things you learn (Inaudible). Oh, we didn't keep good records for the first five years. We were so busy doing great work and research and papers and everything that -- there's Tony Hoare doing things on Quicksort, works in our Cambridge lab, he joined us...

Robert Scoble - The ScobleShow
I know that guy.

Kevin Schofield - Microsoft Research
Yeah, I know... A.J. Brosh, working calendaring stuff, Jian Wang in our Beijing lab working on (Inaudible) and handwriting, Marc Smith on our social computing group.

Robert Scoble - The ScobleShow
Is there one thing that you think is the defining moment for research so far in the first 15 years?

Kevin Schofield - Microsoft Research
So, somebody asked Eric that, when we did a processing last week, and his sort of defining sea change moment was around 1996, summer of 1996, when at SIGGRAPH, the big annual graphics conference, we had...

Robert Scoble - The ScobleShow
Keep going.

Kevin Schofield - Microsoft Research
We had about 20%, I think it's about 20% of the papers on the top, which was just unheard of any research lab to get that large of the percentage of the papers and like on of the really big highly peer reviewed conferences. He sort of said, right after that happened like, later that day, when he find out that he had a meeting with Nathan Myhrvold, and two of them sat down, and they just spontaneously started laughing, because they felt like, okay, we've actually arrived, this crazy thing that we started doing five years before, it's real, and we're making this huge impact on the field. Now, now we look across, OSDI, The Operating Systems Conference and SIGGRAPH and SIGIR, the big search and information retrieval one, kind of the user interface. We're seeing that, we have a huge amount of papers at every single one of those conferences, and the fact that -- it's amazing to do that for one conference, to run the table of all this conferences, it's almost an embarrassment of riches. We should go upstairs. Let's go.

Robert Scoble - The ScobleShow
So, what's happened since I've been - it's been a few months since I've seen you?

Kevin Schofield - Microsoft Research
What's happened? Well, apart from the 15th anniversary, we just had a ton of researchers, super, super busy, we're always, sort of at the end of every school year (Voice Overlap).

Robert Scoble - The ScobleShow
What you do here?

Speaker
Record Research.

Kevin Schofield - Microsoft Research
He eats lunch.

Speaker
Yeah, I eat lunch though. Text mining and research.

Robert Scoble - The ScobleShow
Oh wow. That's important, we'll come back and see you later.

Kevin Schofield - Microsoft Research
We'll come back and see you later. So let's see, we had 220 interns here over the summer, right. We didn't have enough office space for 220 interns, so we took...

Robert Scoble - The ScobleShow
You had 220 interns just in research?

Kevin Schofield - Microsoft Research
Just here in Redmond.

Robert Scoble - The ScobleShow
You mean Microsoft overall, or in research?

Kevin Schofield - Microsoft Research
Just in research, just in MSR, in this building, and the next building over, 220 interns.

Robert Scoble - The ScobleShow
You must had some good Friday afternoons.

Kevin Schofield - Microsoft Research
So if you look, that used to be a lounge. We love our intern program, it's just this huge wonderful thing, we've got all this great energy and enthusiasm in the lab over the summer doing this, but we didn't have enough office space for this, so we took every single lounge in our two buildings and we converted them all to something cubical. I need to make sure I'm going to right office.

Robert Scoble - The ScobleShow
Isn't Eric up here in the corner?

Kevin Schofield - Microsoft Research
Yup.

Kevin Schofield - Microsoft Research
I thought I had it right. Yeah, so it's like, every spare place we could find where we could actually...

Robert Scoble - The ScobleShow
Now, the people don't know who you are, but you're in charge of moving stuff out of the research and into real products, right.

Kevin Schofield - Microsoft Research
I spend a lot of time working on how we get...

Robert Scoble - The ScobleShow
So, you know everybody on the research team?

Kevin Schofield - Microsoft Research
I know a lot of them, we're so big now, we don't know everybody. Hey, there is Eric?

Eric Horvitz - Microsoft Research
Hello.

Robert Scoble - The ScobleShow
What's the coolest thing you've seen?

Kevin Schofield - Microsoft Research
What's the coolest thing...

Robert Scoble - The ScobleShow
What's the top secret coolest thing because the guy behind me, which you can see it, is probably doing it.

Eric Horvitz - Microsoft Research
Boy! You dress the same way, you don't wear a suit now to work daily?

Robert Scoble - The ScobleShow
No, they kicked me out of the country club in Silicon Valley the other day, if you read my blog.

Kevin Schofield - Microsoft Research
Do you want me to give him the mike?

Robert Scoble - The ScobleShow
Yeah that will be great.

Kevin Schofield - Microsoft Research
This is Eric Horvitz. He is a senior researcher and runs our Adaptive Systems and Interaction Group, and I'm going to give the mike off to him.

Eric Horvitz - Microsoft Research
Thanks Kevin.

Kevin Schofield - Microsoft Research
He is way, way cooler and smarter that I am.

Eric Horvitz - Microsoft Research
Okay, right.

Robert Scoble - The ScobleShow
If he's way cooler and smarter than you then, I know you're supposed to be standing there.

Eric Horvitz - Microsoft Research
We need to get this 'what if' shirts for everybody in my group actually because they're very, very cool.

Robert Scoble - The ScobleShow
Yeah, I like these.

Kevin Schofield - Microsoft Research
It's Microsoft Research Work.

Robert Scoble - The ScobleShow
You've got to put that on a Valleyschwag, yeah.

Kevin Schofield - Microsoft Research
Actually, I'm wearing my (Voice Overlap).

Robert Scoble - The ScobleShow
Those would be, if you send him in -- Valleyschwag is a company down in Silicon Valley that sends schwag around the world for 15 bucks a month, you get all sorts of schwag, but that would be like, that would get people to sign up just for that shirt, because...

Kevin Schofield - Microsoft Research
I like it. We're actually going to do another -- we've got a small handful. Actually, I think I can get you one of these before you leave today.

Eric Horvitz - Microsoft Research
So, Kevin asked me just yesterday, he said, guess what, remember that guy Scoble, he's still doing this stuff, and he said he's coming by to see us (Voice Overlap).

Robert Scoble - The ScobleShow
I've got a bigger and heavier camera now.

Eric Horvitz - Microsoft Research
Oh my God, this guys really know how to treat you well.

Robert Scoble - The ScobleShow
Yeah, we got VC money, Bill Gates, he didn't have real money, the VC money, they buy stuff like this.

Eric Horvitz - Microsoft Research
Maybe they're less -- little bit more free with their money. We ought to run here a little bit careful and directed.

Robert Scoble - The ScobleShow
That's one way to put it.

Eric Horvitz - Microsoft Research
Not to install your VC company (Voice Overlap). Anyway, let's just sort of take you around and show you a couple of -- three new things, three new things today. People work that you've seen before, I think on your Channel 9 in the old days. The first person I want to show you...

Robert Scoble - The ScobleShow
We should talk about who you are first because my new audience might not have seen those earlier. You have the most patents here at Microsoft, right?

Eric Horvitz - Microsoft Research
I hate when people mention that because

Robert Scoble - The ScobleShow
Why?

Eric Horvitz - Microsoft Research
It sounds like you're a patent house or something. As I tell people, in our group, we have a very high (Inaudible) for patents, we only patent things, when we think that, at a leading campus in our field, we really impress and surprise our colleagues in the field, so it's -- there's a whole story here, but read about Ben Franklin and what he said about patents, it's very interesting.

Robert Scoble - The ScobleShow
Why did you say?

Eric Horvitz - Microsoft Research
Well, he and Thomas Jefferson, who actually originated the patent system, it turns out, had some very good ideas that are set at the basis of it all, which is sharing. The idea back then was that people who had really good ideas were not necessarily intended to share them, they'd rather keep them trade secrets at companies, and the whole patent system was set up initially for sharing concepts and ideas. The idea was, you trade a little bit, you get little bit back for your interest and novelty, but you actually shared out the concept with the world, and others could build on them, it was kind of a nice thing. So, we actually think a lot about that and think about we think about academic publication, we think about just publishing and showing ideas, without necessarily patenting them, so on. It's a broad portfolio of the kinds of things we do to get ideas out in the world.

So anyway, this group is called Adaptive Systems and Interaction. We do quite a bit of research in the search area, browsing, we do quite a bit of work in machine learning, so we're one of the teams that Microsoft Research is very interested in the prospects that, computing might change dramatically some day if we see further maturations of technologies that can do decision making autonomously, and reason about what you want to happen next, and your goals, to allow you to maybe someday ask office to just, show me that pivot table like I did last time, but this time, show it to me with the competition highlighted in a new way, natural language and applications, speech vision.

Kevin Schofield - Microsoft Research
So, in another words, you really want to make computers smarter, a lot, lot smarter.

Eric Horvitz - Microsoft Research
We think that computers might someday be sort of like talking to a very bright person you're working with. Like Kevin just now, you get the sense that, you know Eric, you can communicate this much more crispy to this audience, and literally this is called mixed initiative interaction, because he thinks he's helping me out, and a computer should someday also work with humans in some ways (Voice Overlap).

Robert Scoble - The ScobleShow
You think we're going to have a computer embedded in our head someday that's going to say, hey...

Eric Horvitz - Microsoft Research
No, but people do think about the idea that it's not just sitting on a desk but that the big opportunity for software, computing will be kind of in the course of daily life, somehow where the computer system understands your goals and needs and recognizes how it can best help you -- potentially to remind you things you might forget, like what was Robert's wife's name again, we had that party, and just sort of time in and so on, and there have been some prototypes of these things, and to date they're rather clunky, but the vision is there.

Kevin Schofield - Microsoft Research
Let's get you some close up.

Robert Scoble - The ScobleShow
So first of all, you don't have any implants on you? You were the most connected guys at Microsoft Research, but neither of you guys have implants or...

Eric Horvitz - Microsoft Research
I have the PC pocket implant.

Robert Scoble - The ScobleShow
I have a PC pocket implant. Carl's eye lenses are still bigger than the human eye.

Eric Horvitz - Microsoft Research
Now, Robert you met Raman Sarin in the past, when you came in here and saw some really cool stuff, about maybe a year ago now, and some of that technology led to some of the search techniques we're shipping with Windows Vista desktop search. You also saw, I think SmartPhlow and JamBase, the traffic forecasting work we were doing, and now Raman continues to work in this area, a very promising area of small devices, and what can be done with them. Today, he'll tell you a little bit about advances in sharing technologies, so Raman Sarin who is a developer on our team.

Raman Sarin - Microsoft Research
Hello.

Robert Scoble - The ScobleShow
How are you doing?

Raman Sarin - Microsoft Research
Hi Robert, how's it going? Yeah, there's a certain amount of hardware in this room (Voice Overlap)

Raman Sarin - Microsoft Research
Actually, with the LCDs, not so much of a problem. If I can remember way back when I use to have CRTs that, yeah that would be a huge problem, and these are all laptops, so they don't really do much, and then this display is for doing tablet, it's basically a potent tablet. What I'm going to show you is this product prototype.

Kevin Schofield - Microsoft Research
Well, first of all, who are you?

Raman Sarin - Microsoft Research
Who am I? I'm me. My name is Raman Sarin and I'm a developer. I work for Ken Hinckley who works for Eric Horvitz, and we have a small sub groups under Adaptive Systems Interaction called the Adaptive Systems Interaction Focus Group and we focus on interaction prototypes for tablet PCs, some prototypes for search and retrieval and some prototypes for smart phone.

Robert Scoble - The ScobleShow
I think you're one of the few Microsoft employees that could be the guy from Engadget, I got tour of his house the other day. I think you have more gadgets in your office.

Raman Sarin - Microsoft Research
Well, I hope I don't have more than he does. I do have the records at Zune and...

Robert Scoble - The ScobleShow
We just interviewed them an hour ago.

Raman Sarin - Microsoft Research
Yeah, very cool, very cool. I'm -- definitely going to get a good piece of hardware. So, what about 'Blue Rendezvous'? So 'Blue Rendezvous' came about, because I was in the car with my wife, who also has a smart phone because I'm a gadget guy and I got her one and I wanted to send her an address of one of our friends and I discovered that it wasn't really an easy thing to do. So, I though, well, why isn't it easier to share information between these devices, I mean this thing right here has got a 2 gigs array of storage in it, and it's like better than most of the PCs I had until about 2000. So, why not make it really, really easy to share between these, without having to do what I called the infrared tango, where you try and keep the ports lined up and sort of move back and forth. So, what I've done -- can you see that?

Robert Scoble - The ScobleShow
Yeah.

Raman Sarin - Microsoft Research
So this app, basically you're going to start out and I'm going to touch a button and I'll just push one doesn't really matter which one I push, and over here on this device, I'm also going to push one. Then here I'm going to hit fine partner. What this thing is doing is it's now going out and its looking for any Bluetooth discoverable device that understands the Blue Rendezvous Protocol, and when it finds such devices, it's asking, hey, did somebody just press the number one I knew, and when it finds somebody that says yes, somebody just pressed the number one on me, it goes, well, you're the guy I want to talk to. Now, I have this nice little menu popped up where I can pick what it is.

Robert Scoble - The ScobleShow
I just got a new BMW, I'm hooking up my Bluetooth phone to the BMW, it took somebody who was trained. I couldn't even do that.

Raman Sarin - Microsoft Research
So, you understand why this is a problem.

Robert Scoble - The ScobleShow
Can you guys sell this technology to the car company, so we can just walk into Microsoft...

Raman Sarin - Microsoft Research
Well, we can sell it to anybody who wants to buy it. So, yeah, from here I can now just -- I'm going to tap this little icon of a camera, let's actually make a tap there. Tap, screen seems to now be out of the line and I'm going to pick this picture of this goofy looking guy here, and over here what you're going to see, is it says, do you want to receive that picture? So, if I say yes, you'll see it transferring and there it is, I have it on this...

Eric Horvitz - Microsoft Research
The new confirmation signal is on there.

Raman Sarin - Microsoft Research
So, by the same token I got files and I've got contacts and then the last thing I have is this, down here it's this thing called business card swap. What it does is, in the program you can define this business card as me, so, if I were to meet somebody like, let's say I wanted to swap business cards with you, we could just run this and hit that button and it will say, exchange business card. The cool thing it does is, it's unlike just a contact swap, it is bidirectional; I'll get yours, you get mine, but where you define the business card, you can also define which fields go across. So, when I send you my business card, you're not going to get my home address and my home phone number and my home email address, all of which are in there, which -- usually people sort of accomplish this by having multiple copies of their own contact in there, and then they end up sending the wrong copy all the time, so we're trying to just avoid that little problem. When you exit the application, what will happen is that, if the Bluetooth radio was off, it'll go back off; if it was on, it will stay on. There was no pairing that took place. So, if you're not going to find that these two devices are still paired together, so somebody can't comeback later and reconnect you without your knowledge, its just it was a one shot deal.

Robert Scoble - The ScobleShow
Now, you have to load a little software in here to do this?

Raman Sarin - Microsoft Research
Yeah, there is an app that I wrote called Blue Rendezvous. The technology really is the connection stuff not so much the transfer, but the transfer is a useful thing that you can do with that connection that we just looked at (ph).

Robert Scoble - The ScobleShow
Can people in the real world get this and download it?

Raman Sarin - Microsoft Research
Not yet, we're working on that.

Robert Scoble - The ScobleShow
So, you'll put up on the research site sometime?

Raman Sarin - Microsoft Research
Hopefully, yeah.

Robert Scoble - The ScobleShow
Okay. Very cool, what else have you guys -- what else have you been working on -- you do some really cool stuff.

Raman Sarin - Microsoft Research
I do. I have a great job, it's a lot of fun.

Robert Scoble - The ScobleShow
People should know that you wrote the cool little traffic app that I -- when I was here in Seattle, I would use it everyday to figure out which route I should take to work.

Raman Sarin - Microsoft Research
Yeah, and Eric's team is still working on doing some really cool extensions to that. You know, we're doing some really cool stuff, and all I can say is, come back in a couple of months.

Robert Scoble - The ScobleShow
Okay.

Raman Sarin - Microsoft Research
And we'll talk to you about it then.

Robert Scoble - The ScobleShow
I will, thanks.

Kevin Schofield - Microsoft Research
I'm going to drive you down the hallway there. Have you met Robert Scoble before?

Robert Scoble - The ScobleShow
Hey, how are you doing?

Ed - Microsoft Research
Hi, there, hi.

Kevin Schofield - Microsoft Research
That's another researcher in our script.

Eric Horvitz - Microsoft Research
I'm just going to mention him briefly that, we're very interested in making, attention -- attention modeling in our group and now we're going to show you some of the leg breaking stuff we're doing in terms of, getting a sense for gauge patterns during search and browsing applications. Patterns of looking at screen and browsing search results and what that all means and Ed, why don't you just give (Voice Overlap)....

Ed - Microsoft Research
We'll just show you what we're doing. So, basically what we're trying to do here.

Robert Scoble - The ScobleShow
Hold on, just one second. Let me put the mike up here and get this tripod.

Ed - Microsoft Research
Well, yeah so I'm going to try to find a good spot here that I wanted to show you what we're looking for, we'll just stop it there. So, the idea behind this is that, we've done a series of studies to try to understand how people search, for both web search and for personal search, and other kinds of things, and one of the ways of getting a peek at how people are searching is, well I mean, the obvious way, like Google, and Yahoo! Search etcetera, Yahoo! uses is that they have massive, massive amounts of click through data. So, you know what people are clicking on, you have vary things around, and you watch what they do. Now, the problem with that is that you don't understand what people are doing when they're not clicking.

Robert Scoble - The ScobleShow
Yeah.

Ed - Microsoft Research
So, there is no information there. So what we tried to do is to say, okay, lets look at what people are actually looking at, and so that's actually is something -- some people have been doing for little while, you use an Eye-tracker, and you just watch what people look at. Now, the next piece to that is, let's say, let's start changing what we show people and see how that affects how they look for stuff. So, one of the things that we've done with this experiment was, we went through and we started changing the tasks that people had, so we had some very explicit kinds of informational and navigational kinds of tasks, these are sort of standard things people do on the web, and then we vary things like, oh let's change where the number one result is. Let's start moving around how the ranking works, or better yet, let's start changing the kinds of metadata we show with it, and watch how people look for what they're doing. So, I'll just show you here a little...

Robert Scoble - The ScobleShow
Now, what's this device here?

Ed - Microsoft Research
This device here, this is a Tobii Eye-tracker. This is a thing that's made in Sweden. It's one of the top of the line kinds of units here, and what it does...

Robert Scoble - The ScobleShow
How does this work?

Ed - Microsoft Research
Yeah, so what it does is, that it has -- there are some IR LEDs in here, and there's a camera here. What it does is, it looks for the bright spot of the pupil. So that the reflection -- so whenever you take a photo, you get that big red eye thing, all that is, it's the light reflecting back of off the pupil in the cornea. So, what this does is, it does that, but it does it in IR, so that people can't see the lights flashing at them. As you move your eye around the screen, it's able to detect what's on axis and what's not on axis, and it can infer where the eye is looking on the screen, because it's registered to all the position there, so you calibrate it, and then you basically are able to see what people are looking at.

Robert Scoble - The ScobleShow
That's awesome. So, it follows your eye.

Ed - Microsoft Research
Yeah, I'll show you a movie, there's somebody actually doing a -- this is from our experiment, so here what you're seeing is, their task is that...

Robert Scoble - The ScobleShow
So, this isn't on right now, is it?

Ed - Microsoft Research
This is just a movie of somebody else.

Robert Scoble - The ScobleShow
Okay, because I think if you turn that on, the camera actually sees IR, so you're (Voice Overlap).

Ed - Microsoft Research
You would be LED flashing, you would see some flashing stuff here. We might be able to turn it on here in a minute, but I'll show you. So, the deal is that -- so what that person is supposed to do here is that, they're supposed to find when the Titanic set sail for it's only voyage, and what port it left from.

Robert Scoble - The ScobleShow
Yeah.

Ed - Microsoft Research
And then the query is just Titanic. So, what you see is, this little dot that's moving around the screen is their gaze point, and you see their queries, they clicked Titanic, and their eyes are moving around, the size of the thing as it moves, as it gets bigger and bigger is the duration of looking. So, you can see that they're reading. Right now they're reading a bunch of the details about the Titanic and moving down to the next piece, they clicked on it up there and then they're like waiting, oh, what happened. They go up and then -- so now they clicked on that piece, so now then they're going to the Titanic piece. They're reading along, and you can see as they're reading, what they're looking for?

Robert Scoble - The ScobleShow
It doesn't look like it's completely calibrated into the text?

Ed - Microsoft Research
It's a little bit off, you get some noise in there, and so what you have to do, let me find another spot in here, maybe down here, around 14 piece...

Robert Scoble - The ScobleShow
So, this is how people create Heatmaps, right?

Ed - Microsoft Research
Exactly

Robert Scoble - The ScobleShow
They run hundreds of people through...

Ed - Microsoft Research
In fact, I can show you a Heatmap of what we did here.

Robert Scoble - The ScobleShow
Okay.

Ed - Microsoft Research
I can actually go ahead and just show you that. Let's do -- it's going to take a second because this is a very long, this was about an hour long search or session, and so it's having to build this up.

Robert Scoble - The ScobleShow
Now, was it your team that figured out that people don't click the next button on searches. Like Live.com has a new Internet scroll.

Ed - Microsoft Research
Well, actually they have rolled that back now.

Robert Scoble - The ScobleShow
Oh, they did.

Ed - Microsoft Research
They've rolled that back, that was in the beta. Now, the reason is, because they weren't able to get the proof (ph) to be completely smooth.

Robert Scoble - The ScobleShow
Yeah.

Ed - Microsoft Research
I think that that's got huge benefit down the road, but right now, because you've got this, it's all Ajax, and they weren't able to get that to be completely smooth and you'd get this jaggedness to it that would start confusing people, so they rolled it back, the current version that's out there. I think that they're trying to get the bugs worked out of that, so if they can (Voice Overlap).

Robert Scoble - The ScobleShow
I thought that was brilliant, because they found using Eye-track research that people would keep scrolling down the page is brilliant.

Ed - Microsoft Research
The other thing about it is, that it is still there for image search, and for image search it makes a lot of sense, because in fact with image search what you're doing is, you're just looking, basically you're just sort of scanning what's going on and you just scroll, rescroll, rescroll, rescroll like that.

Robert Scoble - The ScobleShow
On my new little start-up company, I'll actually use that research to convince the team that they shouldn't care about what's on the page, on the first part of the page, that they should make it, so the page can scroll for a long way because people will scroll through information and fish like they're fishing through a river something like that (Voice Overlap).

Ed - Microsoft Research
We did some research a while back that showed that people actually prefer to scroll and to scan the information as they're scrolling, over having to, to have more items on the screen, but having to go and get a little bit of information with Hovering for instance. You're better off just actually showing the stuff in line, and -- let me show you some of the Heatmaps here, you were just talking about it. I think they're kind of sexy, people love seeing these.

Robert Scoble - The ScobleShow
Only a geek finds Heatmap sexy.

Ed - Microsoft Research
So, what you see here?

Robert Scoble - The ScobleShow
That's a definition of a geek right there.

Ed - Microsoft Research
What you see here is this Titanic site that we just saw a minute ago, and you can see that the person, first off they're looking up here at the query, and then these little 'X's have to do with where they've clicked. So they've hit the Titanic, and you can see the ones that they've kind of ignored, like they read this piece and they're like, oh, there's no way, Titanic in 30 seconds with Bunnies is actually what I needed. So, they didn't even bother to read the information here. This one, which is I believe -- I don't know what this is exactly, but you can tell there's a lot of detail there, and they're really reading quite a lot of information in that piece. As we move down the page, you can see as they get passed the fold, they didn't even look at anything.

Robert Scoble - The ScobleShow
That argues my theory though, that people don't scroll.

Ed - Microsoft Research
Well it does, but the thing here is that, if they find what they think they need, for instance in this case Wikipedia, they have no reason to go any further. So, what happened in this case.

Robert Scoble - The ScobleShow
So treat it like a milk, like a grocer does with the milk, put the important stuff at the bottom of the page and make them scroll through all your ads. You're going to change the whole row.

Ed - Microsoft Research
Oh yeah, actually, it works. So in this case, one thing that's interesting about the different kinds of tasks that this research showed us is that, how people scan across the page is really varying depending on what they're doing.

Robert Scoble - The ScobleShow
Right.

Ed - Microsoft Research
A lot of people have tried to fit search results to be 'one size fits all', and be all things to everyone. A lot of our research now is suggesting that to the extent you can understand what people are doing, you can really tune what your results are showing you for what people are looking for, and you can have a much better experience with that. A lot of challenge now is now trying to do two things - one, figure out what people are doing and then do that optimization, and the other piece is maybe there are ways to wiggle the stuff around a little bit better, so that it performs better for all those people, so you don't have to worry about it so much. So that's the story here.

Robert Scoble - The ScobleShow
Very cool.

Kevin Schofield - Microsoft Research
So, before we move on, actually I'll probably just a word about this room, because it's kind of a goofy looking room. This is actually a room where we do usability tasks, not only do our product groups do usability tasks, and they have a room similar like this...

Eric Horvitz - Microsoft Research
Oh yeah, we have a one-way mirror here, behind this (Voice Overlap).

Kevin Schofield - Microsoft Research
There is a one-way glass here, and usability tasks are equally important for doing research, so we bring in prototypes in this room and we're actually bringing real outside folks and sort of sit down with them, and we get those (Voice Overlap).

Eric Horvitz - Microsoft Research
For all the decision making and machine learning we do -- our whole team is arrayed around, what I would call the womb of our usability study labs, and this is the really way that we wrote is how people react to our things, or how it's actually build.

Kevin Schofield - Microsoft
So, it's behind the one-way glass?

Robert Scoble - The ScobleShow
I've been in the usability but let's go. There might be other people behind the glass.

Kevin Schofield - Microsoft
There's actually nobody, we just want to check because there's nobody at there right now. You have to go through the really messy storage room for us to get there.

Eric Horvitz - Microsoft
This is (Inaudible) on the way.

Kevin Schofield - Microsoft
This is the (Inaudible)

Eric Horvitz - Microsoft
So, when it's dark in here you can see the other side.

Kevin Schofield - Microsoft
Yeah, you actually have to be careful of the lighting because it has to be darker on this side. Somebody actually walks in and puts on the lights and suddenly everybody on other side knows, oh wait a minute, this is how many people there are behind the glass, but actually one of things we do, we understand the process...

Robert Scoble - The ScobleShow
You can turn on the lights now.

Kevin Schofield - Microsoft
One of the tests that we do for usability testing is, before we start the test, we bring in people to show them rooms; say, hey this is one-way glass. And we take them around the back and just kind of demystify it for a while, so it was like, oh my God, there were 30 people on the glass what are they doing? We want this to be really demystified process for people.

Robert Scoble - The ScobleShow
And you guys record this and actually broadcast it over the Web, sometimes to other researchers or people who watch it.

Kevin Schofield - Microsoft
We didn't broadcast as much here out of our stuff, but a lot of what they do like for product (Inaudible) Usability test is -- we'll broadcast around our internal network, so everybody in the product team can watch the Usability tests because this is just fantastic data; there's nothing like watching real people trying to use your product.

Robert Scoble - The ScobleShow
Tell me about the process of research -- we sort of went through that, I had a theory that Web users use the Web in a certain way. You actually do research with real humans and study hundreds or thousands of people how they actually do that and then you report that at a paper, right?

Kevin Schofield - Microsoft Research.
Yeah and there's a lot of statistics behind that too. If you want to do a research on human-computer interaction, you really have to go, take this statistics class and pay attention because that's how those papers and your results get judged about whether it's statistically significant, whether you are deriving the right conclusions from that. There's a whole set of people that review and pay special attention; they really make sure you are doing it right and drawing the right kinds of conclusions from experiments. Of course, that means you have to design the experiment that way so that you're actually controlled for the right things and really isolate the question.

Robert Scoble - The ScobleShow
Now, the danger of doing that is it's actually just a snapshot of human behavior at one point in time because human behavior changes like we use computers a lot different way today than in 1977.

Kevin Schofield - Microsoft
And even with a new product, you can look at, you can test for sort of the first experience; the out of box experience, the first experience or if you run over the course of an hour, you can see how much somebody -- how could somebody learns use a product over the first hour, but what's their behavior going to be like a month after they have been using this product; they really get comfortable with it. If you look at things like Clippy, a lot of people thought Clippy was really great for the first week or so they're using it. And it got tolerably annoying for a set of people after a long period of time. There's a whole set of people who loved it six months, a year, two years after it, but you have to ask all those interesting questions about how is this diverse set of people all going to react to this thing, longitudinally and not just really on short term. So, we got like one more demo to show.

Eric Horvitz - Microsoft
One comment on that -- Kevin it's a good point. So, we're actually doing -- it's a very good concern having fact that people working with a single piece of software for 20 minutes in a session is not the same thing as living with the software for months..

Robert Scoble - The ScobleShow
Well, their behavior changes every time, right?

Eric Horvitz - Microsoft
It does indeed. So, for example, if we have studies underway right now, for example for something called Customer Love, you may have heard about, 500 PCs out in the real world where people agree to let us look at their activity and behavior, we actually tested our Vista perfecting algorithms on actual user workloads out in the real world and characterize people by the type of person they were, you know, Internet browser type or kind of an accounting type person, productivity software, communications type IM person, game player type and actually looked at how well our algorithms worked with these different people like perfecting into...

Robert Scoble - The ScobleShow
Since all companies do user testing at some level, why don't all products look the same, because when they all discover the same usage pattern and the same...

Eric Horvitz - Microsoft
We might say that we're still in the process of discovering the principles of human-computer interaction and we hope that the good principles are shared across applications even if they might look quite different.

Kevin Schofield - Microsoft
I mean, then again, more to the point. Who says there is only one good design for a particular PC software?

Eric Horvitz - Microsoft
There are some great buildings with great architectures and they look different but they all function nicely.

Kevin Schofield - Microsoft
Yeah, why don't all building look alike, right? It's because there's lots of good ways.

Robert Scoble - The ScobleShow
When architects, 400 years ago were building buildings, they didn't have a chance to study human behavior the way you guys do with the...

Kevin Schofield - Microsoft
I don't think they studied differently. You think they didn't get feedback from people in the buildings?

Robert Scoble - The ScobleShow
That's true.

Kevin Schofield - MS Research
Of course they did.

Eric Horvitz - Microsoft Research
I think it's a good point that there might be some interesting general principals like how to manage attention, how people could recover from disruption, they stop doing one thing and multi-task, doing something else that might -- you might be showing up more astigmatic designs across applications software. So, we'll go and see Andy Wilson's ideal stuff.

Kevin Schofield - MS Research
Andy, you got to see Andy.

Robert Scoble - The ScobleShow
Now, who's Andy Wilson?

Eric Horvitz - Microsoft Research
Andy Wilson is a researcher of our team who has been exploring -- he's exploring TouchLight and surface computing. We call the opportunity he's looking at surface computing and the idea of the (Inaudible) that extends to all surfaces in the world, including Midair (ph) and so Andy has put on the PlayAnywhere technology and the Playtable ideas, and today he'll show you something leg breaking in the area of gesture management and gestures to control computers.

Kevin Schofield - MS Research
Cool, give him the mike.

Robert Scoble - The ScobleShow
What do you do? You do some really interesting and mind-blowing stuff with projection and with surfaces, so tell me a little bit about these.

Andy Wilson - Microsoft Research
I am interested in all kinds of sensing technologies and how those can be used to invent new computing experiences, new form factors, new ways of relating to your computer. So, I am interested in wall displays and table talk sessions.

Robert Scoble - The ScobleShow
Which is why you have some projectors up here?

Andy Wilson - Microsoft Research
Yeah, so display technology, I am very interested in display technologies and projection and the fact that in the future, you'll have displays in all different corners of your office and home and car even. So, how we're going to actually interact with those is one of the central questions in my work and you're not going to be using the keyboard and mouse all the time, we see this already with a Tablet PC and mobile devices we've got, this is a bit of an explosion of form factors that we see already.

Robert Scoble - The ScobleShow
Did you built the demo for Bill Gates to use it CES last year, where he laid a cell phone down on a surface and IMs flew out of the cell phone.

Andy Wilson - Microsoft Research
That's right, and so we're still working on that scenario and that's getting progressively more interesting with that work and the stuff that you saw earlier with Raman Sarin and the Blue Rendezvous Project we're actually incorporating that as one of the functionalities of our surface system. You put the phone down and photos fly out of it.

Robert Scoble - The ScobleShow
Are you building a cool demo for Bill for next year's CES?

Andy Wilson - Microsoft Research
Well, yeah, maybe.

Robert Scoble - The ScobleShow
We're going to have a blog house at the CES to bring it out, we are going to be there with a bunch of bloggers.

Andy Wilson - Microsoft Research
We'll see about it.

Robert Scoble - The ScobleShow
But some of the stuff that -- did you built the thing that they showed up at the TechFast two -- a year and a half ago, which was like a minority report screen. Are we allowed to talk about that?

Andy Wilson - Microsoft Research
Yeah, that's TouchLight and that's the system that we've been working on for the last couple of years and that was also a computer vision system, camera is behind the display and anything there is because the screen is transparent.

Robert Scoble - The ScobleShow
You don't have one of those in your office?

Andy Wilson - Microsoft Research
No, I don't actually, the one that I work on I gave to a developer who is now like kind of brush up my crummy research code and get into the form where other people can use it.

Robert Scoble - The ScobleShow
Just explain what it was? It's a piece of holographic glass right?

Andy Wilson - Microsoft Research
Yeah. Holographic film and -- so you can project on to it and also...

Robert Scoble - The ScobleShow
Right with one of your projectors and then you add cameras behind the screen that you couldn't see, right?

Andy Wilson - Microsoft Research
That's right.

Robert Scoble - The ScobleShow
Now, we're triangulated in on the surface of the glass and so if hold like a Time Magazine up to the glass the camera's could actually take and image of that magazine and keep it up on the screen and then you can manipulate it with you hands.

Andy Wilson - Microsoft Research
Exactly, that's right.

Robert Scoble - The ScobleShow
Unfortunately, I can't show this demo, because you don't have it, but I mean that was mind-blowing for me, it was like, wow, computers could be used for something completely wacky and the stuff that we saw in movies could actually come true.

Andy Wilson - Microsoft Research
Right, so one of the research questions is, how are these interactions actually going to work, how does it -- does it make sense to be able to put your hands directly on the data and manipulate it? These kinds of natural gesture based interaction systems are interesting, because you have a really nice feel, fluid feel, for what you're doing. Right now we have these systems -- they are based on the mouse, when you take all the richness of human emotion and you reduced it to one 2D coordinate.

Robert Scoble - The ScobleShow
How did you figure out -- who came up with the idea of putting two cameras underneath the piece of glass, how did you guys come up with that, because it seems so obvious once you see it, but it probably wasn't obvious before you did it or before you came up with that idea.

Andy Wilson - Microsoft Research
I went to SIGGRAPH a few years ago, and the company that sells that projection screen material was there demoing their -- they were just running a small demo of it, just passively displaying, advertising images on it, and as soon as I saw that, I knew immediately that I wanted a piece of that because I knew exactly what we were going to do with it.

Robert Scoble - The ScobleShow
How did you get it...

Kevin Schofield - Microsoft
Do you have something to show?

Andy Wilson - Microsoft Research
Yeah.

Robert Scoble - The ScobleShow
If you guys actually start showing me stuff, I'm going to here all day, because out of all the stuff I saw at TechFest a year ago, a year and a half ago, that was the coolest stuff. So, I'm honored at meeting you. I'm almost out of battery, so we have 13 minutes of battery left.

Andy Wilson - Microsoft Research
So, what I'm going to show you here is just a real simple desktop version of some of these same ideas, and I'm just going to hop right into it. We have a camera here, this is just a USB webcam, and it's actually looking down at the surface.

Robert Scoble - The ScobleShow
Right. I was looking at your hands.

Andy Wilson - Microsoft Research
Yeah. So here I am, I'm just doing a standard sort of desktop computing kind of thing, and suppose I'm typing or using the mouse whatever, so, if in this system if I put my thumb and forefinger together, the system notices that and allows me to move the cursor.

Robert Scoble - The ScobleShow
Do that again. How did you do that?

Andy Wilson - Microsoft Research
If you look at this view here, this is actually showing what the camera sees and...

Robert Scoble - The ScobleShow
How did it sense -- did you program it to recognize that shape of your hand?

Andy Wilson - Microsoft Research
Well, so the way that it works is kind of neat. It actually reasons about the topology of the image, so it determines what is the background, what is the foreground and basically when you put your thumb and your forefinger together, you're pinching off a piece of the background, if you will. So, all it's doing is counting the number of shapes in the image. So, by putting your thumb and your forefinger together, you're creating a new shape and image. Does that make sense?

Robert Scoble - The ScobleShow
Yeah.

Andy Wilson - Microsoft Research
So, it doesn't know anything about hands or fingers or anything like that and so right now we're just driving the Windows cursor around.

Robert Scoble - The ScobleShow
You're just dragging that Window around with your hand.

Andy Wilson - Microsoft Research
That's right, exactly, not using the mouse.

Robert Scoble - The ScobleShow
You're not doing anything underneath the desk?

Andy Wilson - Microsoft Research
No.

Robert Scoble - The ScobleShow
There's nobody, no remote control?

Andy Wilson - Microsoft Research
No, no guy under the desk.

Robert Scoble - The ScobleShow
When I first saw this, I thought it was so mind blowing, it's so magical, I guess that's good software. When software is really doing something cool, you think its magic, you think there is a guy behind it.

Andy Wilson - Microsoft Research
Magic, right, right. So, here's hooking the same stuff up to Virtual Earth, and so occasionally that screws up a little bit; this is research code. Let's try that again, the rendering. So, you have this nice fluid kind of interaction, where if I rotate my hands, rotates the view, yeah, a little bit of motion there. Now, I can also extend the algorithm to look at the use of two hands, and so, I can zoom in and out very fluidly. Now, this program is pulling tiles off the Virtual Earth server, of the Virtual Earth Web service in real time, so I can zoom in and out as much as you can on the...

Robert Scoble - The ScobleShow
This has deep implications for music, I could be a, what do you call that guy, conductor, and I could conduct an orchestra remotely using this kind of technology. As a surgeon, I can do really complex stuff. Now, could you come up with new algorithms that look at other gestures on your hands?

Andy Wilson - Microsoft Research
Well, sure...

Robert Scoble - The ScobleShow
Could I give the computer the finger?

Andy Wilson - Microsoft Research
So this is really -- what I'm showing you here is just sort of a really quick hack, like I'm skipping around all sorts of problems, recognizing fingers or hands or gestures just by exploring a very particular property of the image as it changes when I put my thumb and the forefinger together. So, right now, this doesn't go very much further in that, but in general, yeah, we can do all sorts of stuffs like that. We can bring in other techniques, other pattern reorganization techniques to understand that. Here, I'm doing a particular shape with my hand and maybe I'm changing the axis and the CAD package or something like that, or I'm resizing something like this, these kinds of...

Robert Scoble - The ScobleShow
And this is just a $100 camera?

Andy Wilson - Microsoft Research
More like 30 bucks, so it's all this software.

Robert Scoble - The ScobleShow
Have you shown this stuff off in public before?

Andy Wilson - Microsoft Research
Have I? No, (Voice Overlap).

Robert Scoble - The ScobleShow
Are you going to go to SIGGRAPH and show this off?

Andy Wilson - Microsoft Research
I'm going to be showing this at a conference in a couple of weeks, for the first time really.

Robert Scoble - The ScobleShow
Okay. Do you want me to hold the video until...

Andy Wilson - Microsoft Research
No, no, it's fine.

Robert Scoble - The ScobleShow
I love software that just blows your mind, even though, once you see it, you understand how you can write an algorithm that does that, but it's just magical.

Andy Wilson - Microsoft Research
Yeah, it just has a really nice fluid feel, and the thing that -- I think the thing that's really nice about this particular -- for me, as a vision guy, who's into writing these kinds of algorithms, so it was really nice to be able to write this code very quickly, just a such a simple technique, just observation that, when you put your thumb and a forefinger together, you're making a new shape in the image. It's kind of a hack, if you will, kind of an inspired hack. So, I like it when these kinds of things happen.

Robert Scoble - The ScobleShow
Do you get called to consult with CSI and help them there?

Andy Wilson - Microsoft Research
No, not really, but...

Robert Scoble - The ScobleShow
Because that's the kind of stuff. This is CSI style stuff, it's so mind-blowing, unless I was seeing it right now, I wouldn't believe it.

Kevin Schofield - Microsoft Research
MSR actually has this thing called (Inaudible) a couple of times...

Andy Wilson - Microsoft Research
Could be a fun thing to do, to see what they think about it and get a little exposure.

Robert Scoble - The ScobleShow
So, what other kind of things do you do, I mean that's so mind-blowing, I should probably stop, but...

Andy Wilson - Microsoft Research
These kinds of methods are just one example of other -- in general, just how do you do fluid sensing of motion and hands and that sort of thing, and the idea of manipulating maps is nice one, because you get this sort of very nice rotation and translation and scaling, pretty analogous to the way real maps on a table work. So those are the kinds of things that excite me the most, is when you're borrowing from what you know already, in reality based interaction, so some of the other systems that I have gathered are on tabletop systems, and walls, and these kinds of things, we're doing largely the same, the same kind of thing.

Robert Scoble - The ScobleShow
How long do you, best guess, do you think it's going to be before you get a system that has enough completeness to it, because obviously that's what -- one little algorithm, you probably need a group of algorithms to make a product, right?

Andy Wilson - Microsoft Research
Yeah, it needs to be a little bit more...

Robert Scoble - The ScobleShow
To get into my house.

Andy Wilson - Microsoft Research
Yeah, I mean that's a complex question. I don't know, it seems to me like these couple of -- this thing in particular might be the kind of thing that could be a power toy, like if you buy a camera, like you would ship with that, and that might actually be an interesting way to sort of test the waters and see how it works. Now the Xbox 360 has a camera, we've been working a little bit with that and trying to figure out what we can -- what new scenarios we can support with that.

Robert Scoble - The ScobleShow
A guy down in Silicon Valley I met has actually bought some really cheap color bands that he puts on you and then he can study your baseball sling or your golf sling. So, you can do other things with colored images or even on your fingers, you can put little colored bands, it probably would help a lot (Voice Overlap).

Andy Wilson - Microsoft Research
Or a little ring or something like that, I mean, it would help a lot actually. I think we have lots of interesting ways to ship this technology and I hope we do it. That will be fun. I think in the next few years -- right now we're in this phase where we finally have the computational horsepower to do this kind of stuff without completely wrecking the rest of the machine. So, with dual-core processors and these kinds of things, this really opens up a lot of possibilities.

Robert Scoble - The ScobleShow
Cool. We have 5 minutes of battery left, so do we have one more thing to see or is this...

Kevin Schofield - Microsoft Research
No, I think this is it.

Robert Scoble - The ScobleShow
This is it.

Kevin Schofield - Microsoft Research
It wasn't enough?

Robert Scoble - The ScobleShow
This was awesome man. I could talk to this guy all day long.

Kevin Schofield - Microsoft Research
You can talk to all these guys.

Eric Horvitz - Microsoft Research
You need kind a menu and you can come back (Voice Overlap)

Kevin Schofield - Microsoft Research
My job is to make sure they actually get work done.

Eric Horvitz - Microsoft Research
Come back again, it was great seeing you. It's really great, we actually miss you around here.

Robert Scoble - The ScobleShow
Thanks.

Kevin Schofield - Microsoft Research
It was great seeing you.

Robert Scoble - The ScobleShow
Thanks for giving me your -- I'm really honored to meet you, because the stuff I saw was just so cool, I wish I could tell you what I saw in the TechFest. That's one of the advantages of being a Microsoft employee, you get to come and see all the cool stuff you guys are building in now.

Kevin Schofield - Microsoft Research
Do you know what, there's a chance we can bring you back for the TechFest this year?

Robert Scoble - The ScobleShow
Yeah, but they don't let press into all the areas.