Robobuilder Controlled by Kinect
Robobuilder Controlled by Kinect
by PedroR
Tue Mar 13, 2012 2:19 pm
Hi all
RoboSavvy will be present at the Big Bang event ( http://www.thebigbangfair.co.uk/home.cfm ) and we'll be showing off NAO controlled by Kinect as well as a Robobuilder controlled by Kinect.
The Robobuilder solution was developed in house (ie by RoboSavvy) and we based our work on:
- VB 2010 Skeletal Samples for kinect
- Robobuilder .Net Library by l3v3rz (very nice library I must add! gone are the days of weird tinkering with byte arrays and checksums)
The solution was actually finished at the last minute before Marco headed off to prepare the event so we didn't have time to do a Video. We'll add this as soon as he returns.
[UPDATE March 15th, 2012: Marco sent a video from the Big Bang event with a brief demo:
[/UPDATE]
[UPDATE 21 March 2012] If you want to get the Pre Compiled version of this software, ready to run, please see bellow http://robosavvy.com/forum/viewtopic.php?p=33945#33943 [/EDIT]
[UPDATE July 9th2012] We've improved our code. The new (V2) with full Upper Body tracking is available here http://robosavvy.com/forum/viewtopic.php?p=34624 [/EDIT]
In our Robobuilder Kinect demo we only track the head and hand joints and then raise and lower the arms of the Robobuilder to mimic your position.
It's a very simple closed loop system but it's an interesting first experience to explore Kinect.
The Aldebaran Kinect demo for NAO is even funnier because you can control the movement of the Robot with your hands: two hand in front = stop, hands down = go; right hand up and right = walk right and left hand up and left = walk left.
It has been a cool Sci Fi experience playing with these things.
I'm finalizing the instructions on how to install all the SDKs and publish the source code for anyone who wants to try out.
The Kinect SDK, especially the Skeleton library makes it very, very simple to do this.
You get an event that is fired whenever there is a new array with positions of the 20 joints that Kinect tracks.
You basically do nothing apart from initializing and retrieving the array with the Skeleton Positions.
I really wish we had the time to make a video as this is really exciting. (both what we did for Robobuilder and what Aldebaran did for NAO)
In the meantime if you have the opportunity of visiting the Big Bang event (it's tailored for the Younger but just in case you want to spend some time with the kids), we'll be there and we'll be happy to receive you.
Big Bang event details:
Website: http://www.thebigbangfair.co.uk/home.cfm
"From 15 - 17 March 2012, The Big Bang Fair comes to The NEC, Birmingham. Always free, visitors can do everything from designing ...."
Regards
Pedro.
RoboSavvy will be present at the Big Bang event ( http://www.thebigbangfair.co.uk/home.cfm ) and we'll be showing off NAO controlled by Kinect as well as a Robobuilder controlled by Kinect.
The Robobuilder solution was developed in house (ie by RoboSavvy) and we based our work on:
- VB 2010 Skeletal Samples for kinect
- Robobuilder .Net Library by l3v3rz (very nice library I must add! gone are the days of weird tinkering with byte arrays and checksums)
The solution was actually finished at the last minute before Marco headed off to prepare the event so we didn't have time to do a Video. We'll add this as soon as he returns.
[UPDATE March 15th, 2012: Marco sent a video from the Big Bang event with a brief demo:
phpBB [media]
[/UPDATE]
[UPDATE 21 March 2012] If you want to get the Pre Compiled version of this software, ready to run, please see bellow http://robosavvy.com/forum/viewtopic.php?p=33945#33943 [/EDIT]
[UPDATE July 9th2012] We've improved our code. The new (V2) with full Upper Body tracking is available here http://robosavvy.com/forum/viewtopic.php?p=34624 [/EDIT]
In our Robobuilder Kinect demo we only track the head and hand joints and then raise and lower the arms of the Robobuilder to mimic your position.
It's a very simple closed loop system but it's an interesting first experience to explore Kinect.
The Aldebaran Kinect demo for NAO is even funnier because you can control the movement of the Robot with your hands: two hand in front = stop, hands down = go; right hand up and right = walk right and left hand up and left = walk left.
It has been a cool Sci Fi experience playing with these things.
I'm finalizing the instructions on how to install all the SDKs and publish the source code for anyone who wants to try out.
The Kinect SDK, especially the Skeleton library makes it very, very simple to do this.
You get an event that is fired whenever there is a new array with positions of the 20 joints that Kinect tracks.
You basically do nothing apart from initializing and retrieving the array with the Skeleton Positions.
I really wish we had the time to make a video as this is really exciting. (both what we did for Robobuilder and what Aldebaran did for NAO)
In the meantime if you have the opportunity of visiting the Big Bang event (it's tailored for the Younger but just in case you want to spend some time with the kids), we'll be there and we'll be happy to receive you.
Big Bang event details:
Website: http://www.thebigbangfair.co.uk/home.cfm
"From 15 - 17 March 2012, The Big Bang Fair comes to The NEC, Birmingham. Always free, visitors can do everything from designing ...."
Regards
Pedro.
Last edited by PedroR on Mon Jul 09, 2012 10:45 am, edited 6 times in total.
by PedroR
Wed Mar 14, 2012 5:50 pm
Hi all
Here comes the some hands-on information about our solution.
Firstly, this is how we designed the solution:
We started off with the demo code here http://www.instructables.com/id/Kinect- ... g-Visual-/ which served as inspiration and kick starter.
(please note the code and instructions are outdated since Kinect SDK v1.0 has since been released and brought very significant changes to the Kinect classes, and the way we start and get skeletons from kinect)
This project itself is making use of one of the samples included in the Kinect SDK Beta 2. This is a Visual basic 2010 solution that runs under Visual basic Express (which is free).
All Visual basic samples have disappeared in the final V1.0 release.
1) What the sample does
What the VB Skeletal sample project does is make use of a Skeleton Tracking Sample that tracks the hands and head positions.
In the Instructables sample, the user has extended it to output a value through the COMM port between 0-180 to an Arduino that would in turn move 2 servos to animate a very simple 2 DoF figure.
He made use of a very cool extension from Coding4Fun for the Kinect which extends each joint class with a "ScaleTo" function.
This function takes the range to which you want to normalize the output (in his case 0-180) and optionally the upper and lower boundaries (in case you want to trim or fine tune the normalization to a specific area of positions)
One cool thing about kinect that is worth mentioning is that all values are normalized in relation to the head I believe (maybe it's a whole body approach...).
This means that you always get the same result regardless of the distance you have from kinect.
2) Adapting to Robobuilder
translating this sample to Robobuilder involved 2 main steps.
Step a) consists in converting the Bytes that are being sent into proper wCK move commands
This one was easily sorted with l3v3rz excellent .Net library for Robobuilder.
- We create an instance of the Robobuilder class,
- open the COMM,
- set the Robot to the Home position
- and enter Direct Control mode.
- next create an instance of the wCK class (which encapsulates commands to talk directly to the servos) and
- from there we just call the functions to set position.
While this involves a complete rewrite of the code that talks to the COMM port is no major deal. Anyone familiar with the RBC and wCK protocol can do this in less than 10 minutes. (yes that's how nice the libraries are!)
Step b) involved mapping the Hand position (1 single value) into a movement of the 3 servos on each arm.
This revealed more complex that we initially anticipated.
From the first moment we wanted to make the movement as natural and human like as possible (i.e. we didn't want a Gull wing effect with stretched arms moving up and down attempting lift off!)
The idea was to have the Robot move from its standing position (elbows slightly bent and pointing backwards) to an elevated position with the arms stretched.
For this reason mapping the Hand Position only to Servos 11 and 14 (that control elevation of the arms) was not going to work for us.
We also needed to move servos 10/12 and 13/15 to go from elbows pointing backwards to stretched arms and vice versa.
First we used Motion Builder in Catch and Play mode and capture the value of the servos in 3 key frames:
- Arms down (home position with elbows pointing backwards)
- Arms at shoulder level (completely stretched)
- Arms up (completely stretched)
From here, we used Excel to Build a Table with these positions. The idea is to.
- map the Position of servo 10/12 in relation to the position of Servo 11
- map the position of servo 13/15 in relation to the position of servo 14
Table Discrete Position Progression For Raising Arm by RoboSavvy, on Flickr
- We also added some intermediate steps to define how we wanted the movement to progress. (these are the ones in black; you can see values progress much faster between the first 2 key frames)
This is especially important to add the "Humanizing Factor": we wanted the arms to stretch and the shoulder to rotate significantly in the very early steps and then stay stretched until the arms are fully up.
We did not want a direct proportion between the positions of the servos as it would make it look too Robotic.
Once we had our table with the Discrete Positions set up, we built an XY Scatter Chart with these values. We then used a cool tool in Excel (that Marco explained to me!) called TrendLine and selected Polynomial, 3rd degree.
Graph - Determining 3rd Degree Functions for Mapping Shoulder and Elbow Position to Servo 11 (raise of the arms) by RoboSavvy, on Flickr
This gave us the position curves (and equations) for servo 10/13 that adjusts to the discrete positions on the table/graph.
The function relates the target servo position (10/12) to the position of servo 11.
(fyi the steps to do this in XL 2007 are: create the Chart (it MUST be set to "X Y (Scatter)"), select the "Layout" tab and click "Trendline". next click "more trendline options" and choose Polynomial).
We now have the functions to move our servos in a smooth, more natural way.
You may be wondering about servos 13/14/15; in practice if you want to mirror the movements just copy the table and calculate 255-[current value] to give you the mirrored position. next follow the exact same procedure.
It was now simply a matter of porting the functions to the Visual basic 2010 code.
The final code is quite simple.
To determine the Position of Servo 11 and 14 (which are needed to call UpdateRobobuilderArms) we use the following code to directly relate the hand position to Servo position:
As you can see we use the "ScaleTo" function from the Coding4Fun Kinect library to directly map hand position to Servo 11 and Servo 14 positions
Next UpdateRobobuilderArms uses the functions we got from Excel to determine the position of the remaining servos (10/12 and 13/15) and uses the .Net library by l3v3rz to send the updated positions to the servos.
I am really anxious to get the Kinect and Robot back in the Office to make a Video that shows all of this coming together!
I will be posting the Source code, a compiled executable and installation instructions on the next post.
Please feel free to post your comments.
Regards
Pedro.
Here comes the some hands-on information about our solution.
Firstly, this is how we designed the solution:
We started off with the demo code here http://www.instructables.com/id/Kinect- ... g-Visual-/ which served as inspiration and kick starter.
(please note the code and instructions are outdated since Kinect SDK v1.0 has since been released and brought very significant changes to the Kinect classes, and the way we start and get skeletons from kinect)
This project itself is making use of one of the samples included in the Kinect SDK Beta 2. This is a Visual basic 2010 solution that runs under Visual basic Express (which is free).
All Visual basic samples have disappeared in the final V1.0 release.
1) What the sample does
What the VB Skeletal sample project does is make use of a Skeleton Tracking Sample that tracks the hands and head positions.
In the Instructables sample, the user has extended it to output a value through the COMM port between 0-180 to an Arduino that would in turn move 2 servos to animate a very simple 2 DoF figure.
He made use of a very cool extension from Coding4Fun for the Kinect which extends each joint class with a "ScaleTo" function.
This function takes the range to which you want to normalize the output (in his case 0-180) and optionally the upper and lower boundaries (in case you want to trim or fine tune the normalization to a specific area of positions)
One cool thing about kinect that is worth mentioning is that all values are normalized in relation to the head I believe (maybe it's a whole body approach...).
This means that you always get the same result regardless of the distance you have from kinect.
2) Adapting to Robobuilder
translating this sample to Robobuilder involved 2 main steps.
Step a) consists in converting the Bytes that are being sent into proper wCK move commands
This one was easily sorted with l3v3rz excellent .Net library for Robobuilder.
- We create an instance of the Robobuilder class,
- open the COMM,
- set the Robot to the Home position
- and enter Direct Control mode.
- next create an instance of the wCK class (which encapsulates commands to talk directly to the servos) and
- from there we just call the functions to set position.
While this involves a complete rewrite of the code that talks to the COMM port is no major deal. Anyone familiar with the RBC and wCK protocol can do this in less than 10 minutes. (yes that's how nice the libraries are!)
Step b) involved mapping the Hand position (1 single value) into a movement of the 3 servos on each arm.
This revealed more complex that we initially anticipated.
From the first moment we wanted to make the movement as natural and human like as possible (i.e. we didn't want a Gull wing effect with stretched arms moving up and down attempting lift off!)
The idea was to have the Robot move from its standing position (elbows slightly bent and pointing backwards) to an elevated position with the arms stretched.
For this reason mapping the Hand Position only to Servos 11 and 14 (that control elevation of the arms) was not going to work for us.
We also needed to move servos 10/12 and 13/15 to go from elbows pointing backwards to stretched arms and vice versa.
First we used Motion Builder in Catch and Play mode and capture the value of the servos in 3 key frames:
- Arms down (home position with elbows pointing backwards)
- Arms at shoulder level (completely stretched)
- Arms up (completely stretched)
From here, we used Excel to Build a Table with these positions. The idea is to.
- map the Position of servo 10/12 in relation to the position of Servo 11
- map the position of servo 13/15 in relation to the position of servo 14
Table Discrete Position Progression For Raising Arm by RoboSavvy, on Flickr
- We also added some intermediate steps to define how we wanted the movement to progress. (these are the ones in black; you can see values progress much faster between the first 2 key frames)
This is especially important to add the "Humanizing Factor": we wanted the arms to stretch and the shoulder to rotate significantly in the very early steps and then stay stretched until the arms are fully up.
We did not want a direct proportion between the positions of the servos as it would make it look too Robotic.
Once we had our table with the Discrete Positions set up, we built an XY Scatter Chart with these values. We then used a cool tool in Excel (that Marco explained to me!) called TrendLine and selected Polynomial, 3rd degree.
Graph - Determining 3rd Degree Functions for Mapping Shoulder and Elbow Position to Servo 11 (raise of the arms) by RoboSavvy, on Flickr
This gave us the position curves (and equations) for servo 10/13 that adjusts to the discrete positions on the table/graph.
The function relates the target servo position (10/12) to the position of servo 11.
(fyi the steps to do this in XL 2007 are: create the Chart (it MUST be set to "X Y (Scatter)"), select the "Layout" tab and click "Trendline". next click "more trendline options" and choose Polynomial).
We now have the functions to move our servos in a smooth, more natural way.
You may be wondering about servos 13/14/15; in practice if you want to mirror the movements just copy the table and calculate 255-[current value] to give you the mirrored position. next follow the exact same procedure.
It was now simply a matter of porting the functions to the Visual basic 2010 code.
The final code is quite simple.
To determine the Position of Servo 11 and 14 (which are needed to call UpdateRobobuilderArms) we use the following code to directly relate the hand position to Servo position:
- Code: Select all
Dim bServo11 As Byte
Dim bServo14 As Byte
Dim ServoRange As Integer = bMaxShoulder - bMinShoulder
' Left Arm
Dim scaledLeftArm = playerSkeleton.Joints(JointType.HandRight).ScaleTo(1, ServoRange, 0.5F, 0.3F)
bServo11 = CByte(255 - (255 - (ServoRange - scaledLeftArm.Position.Y + bMinShoulder)))
' Right Arm
Dim scaledRightArm = playerSkeleton.Joints(JointType.HandLeft).ScaleTo(1, ServoRange, 0.5F, 0.3F)
bServo14 = CByte(255 - (ServoRange - scaledRightArm.Position.Y + bMinShoulder))
UpdateRobobuilderArms(bServo11, bServo14)
As you can see we use the "ScaleTo" function from the Coding4Fun Kinect library to directly map hand position to Servo 11 and Servo 14 positions
Next UpdateRobobuilderArms uses the functions we got from Excel to determine the position of the remaining servos (10/12 and 13/15) and uses the .Net library by l3v3rz to send the updated positions to the servos.
- Code: Select all
Private Sub UpdateRobobuilderArms(ByVal bServo11 As Byte, ByVal bservo14 As Byte)
' left arm
Dim bServo10 As Byte
Dim bServo12 As Byte
Dim dServo11 As Double = CDbl(bServo11)
' 5E-05x3 - 0.0213x2 + 2.9618x - 46.011
bServo10 = CByte(0.00005 * dServo11 ^ 3 - 0.0213 * dServo11 ^ 2 + 2.9618 * dServo11 - 46.011)
' 8E-05x3 - 0.0339x2 + 4.7877x - 108.15
bServo12 = CByte(0.00008 * (dServo11 ^ 3) - 0.0339 * (dServo11 ^ 2) + 4.7877 * dServo11 - 108.15)
' right arm
Dim bServo13 As Byte
Dim bServo15 As Byte
Dim dServo14 As Double = CDbl(bservo14)
' 5E-05x3 - 0.0176x2 + 2.0147x + 87.278
bServo13 = CByte(0.00005 * (dServo14 ^ 3) - 0.0176 * (dServo14 ^ 2) + 2.0147 * dServo14 + 87.27)
' 8E-05x3 - 0.0272x2 + 3.0598x + 24.756
bServo15 = CByte(0.00008 * (dServo14 ^ 3) - 0.0272 * (dServo14 ^ 2) + 3.0598 * dServo14 + 24.756)
SendArmPosition(bServo10, bServo11, bServo12, bServo13, bservo14, bServo15)
End Sub
Private Sub SendArmPosition(ByVal bServo10 As Byte, ByVal bServo11 As Byte, ByVal bServo12 As Byte, ByVal bServo13 As Byte, ByVal bServo14 As Byte, ByVal bServo15 As Byte)
On Error GoTo ErrHandler
If Not bRBConnected Then Exit Sub
rbWCK.wckMovePos(10, bServo10, 1) ' 3rd parameter is Torque 0~4: 0=maximum
rbWCK.wckMovePos(11, bServo11, 1)
rbWCK.wckMovePos(12, bServo12, 1)
rbWCK.wckMovePos(13, bServo13, 1)
rbWCK.wckMovePos(14, bServo14, 1)
rbWCK.wckMovePos(15, bServo15, 1)
Exit Sub
ErrHandler:
MsgBox("Unable to Update Servo Position" & vbCrLf & vbCrLf & Err.Description, vbCritical And vbOKOnly, "Error Connecting")
Err.Clear()
End Sub
I am really anxious to get the Kinect and Robot back in the Office to make a Video that shows all of this coming together!
I will be posting the Source code, a compiled executable and installation instructions on the next post.
Please feel free to post your comments.
Regards
Pedro.
by PedroR
Wed Mar 14, 2012 7:21 pm
For the third post, I will be explaining how to Install all the necessary dependencies for the Kinect SDK.
Please note this code has been written for Kinect SDk v1.0 (the latest at the time of writing the post).
[EDIT 21 March 2012] If you just want to get the Pre Compiled version, ready to run (and not the complete development environment), the instructions are much simpler.
They are available bellow http://robosavvy.com/forum/viewtopic.php?p=33945#33943 [/EDIT]
Step 1. Installing the Software
1a. Microsoft DirectX SDK - June 2010 or later version
http://www.microsoft.com/download/en/de ... en&id=6812
(if you get an error during the Installation, please google for the error code. There is a typical error that occurs during install that is caused by having a version of the C++ redistributable that is too recenet.
Google will be your friend to quickly overcmoe this)
1b. Runtime for Microsoft DirectX 9
http://www.microsoft.com/download/en/de ... g=en&id=35
1c. Install Microsoft Visual Basic Express 2010
http://www.microsoft.com/visualstudio/e ... ic-express
1d. Install .NET Framework 4 (if not yet installed)
http://msdn.microsoft.com/en-us/netframework/aa569263
1e. Install Kinect SDK v 1.0
http://www.microsoft.com/en-us/kinectfo ... rview.aspx
(we used 1.0.3.190 which was the one available at the time of writing the code)
1f. Download Coding4Fun Kinect Toolkit
http://c4fkinect.codeplex.com/
Make sure you get the version for Kinect SDK 1.0
(this offers a class that extends the Kinect classes with a couple of useful functions, including the excellent "ScaleTo" function).
1g. Source code (and compiled executable) for Robobuilder_SkeletalTracking VB
http://robosavvy.com/RoboSavvyPages/Rob ... tSDKv1.zip
2. Testing Kinect
Once everything is downloaded and Installed, Open the Kinect Sample Browser and start the "Skeleton Viewer".
Kinect SDK V1 SampleBrowser - Runnng the Skeletal Viewer Sample by RoboSavvy, on Flickr
If you're able to see your Skeleton moving your almost there!
3. Opening the Robobuilder Solution and Running the Sample code
Extract the files in step 1g.
- Inside the Subfolder SkeletalTracking\Bin\release you will find an Executable, ready to run and to control Robobuilder.
- To view and edit code open SkeletalTracking.sln
If you get errors of missing or Invalid references in VB 2010, make sure you have everything installed from point 1.
Next remove any missing References and add them again.
Relevant References are listed bellow:
Setting Project References for the Robobuilder_SkeletalTracking Solution by RoboSavvy, on Flickr
Make sure you set:
- Coding4Fun.Kinect.Winform
- Coding4Fun.Kinect.WPF
- Microsoft.Kinect
- RobobuilderLib
(for RobobuilderLib you can Browse for the RobobuilderLib.dll that is included in the ZIP file containing the complete VB project or you can download it from http://code.google.com/p/robobuildervc/ )
4. Known issues
- The code will only care about the first or default player (while kinect can track 2 people we deliberately choose to get data from FirstOrDefault only).
- While you are being tracked but NOT yet CONNECTED to the Robot you will see 3 circles being updated in real time and these match your head and hand positions.
- Once you Open the COMM port to connect the Robot, the software will set the Robot in its home position and enter DC mode.
The position of the Robot arms will be updated in Real Time but on some PCs the window with the circles may seem to have frozen (the circles won't move anymore and none of the buttons will work).
This "freezing behavior" of the UI does not really affect the movement of the Robot. It doesn't happen on all PCs though; it seems something related to the PCs USB host and not Kinect or the code.
If your Ui freezes, to be able to close the program, cover the kinect sensor with your hand (so that it looses tracking and interrupts the closed loop feed between kinect and Robobuilder). The Window will now be responsive again.
- Another issue is an unhandled Exception if you try to Open a COMM port number that doesn't exist.
While we handle this exception the error seems to be coming from the RobobuilderLib.dll.
- A final recommendation is that you connect the Robobuilder to the computer using the Serial cable and not over Bluetooth.
From our testing when using Bluetooth on this kind of closed loop control system you can really "see" the latency: the arms won't move fluidly (they'll be jerky).
When you switch to the cable the movement becomes much more fluid.
If you want to learn more about Kinect and explore samples that Make use of the Other sensors (Depth Sensor, Audio, etc.) you may use the samples provided with the Kinect SDK v1. The limitation with the V1 SDK is that there are next to none VB samples (they're mostly C#).
To get VB samples you can alternatively download these http://files.ch9.ms/coding4fun/KinectFo ... starts.zip
Be advised however that these samples are for Kinect SDK Beta 2 and won't work with SDK V1.
This file also contains Powerpoint presentations introducing the concepts and sensors of Kinect. These presentations are definitely worth reading.
Next week Marco will be back with Kinect and the Robobuilders so hopefully we'll be able to shoot and post the video.
Regards
Pedro.
Please note this code has been written for Kinect SDk v1.0 (the latest at the time of writing the post).
[EDIT 21 March 2012] If you just want to get the Pre Compiled version, ready to run (and not the complete development environment), the instructions are much simpler.
They are available bellow http://robosavvy.com/forum/viewtopic.php?p=33945#33943 [/EDIT]
Step 1. Installing the Software
1a. Microsoft DirectX SDK - June 2010 or later version
http://www.microsoft.com/download/en/de ... en&id=6812
(if you get an error during the Installation, please google for the error code. There is a typical error that occurs during install that is caused by having a version of the C++ redistributable that is too recenet.
Google will be your friend to quickly overcmoe this)
1b. Runtime for Microsoft DirectX 9
http://www.microsoft.com/download/en/de ... g=en&id=35
1c. Install Microsoft Visual Basic Express 2010
http://www.microsoft.com/visualstudio/e ... ic-express
1d. Install .NET Framework 4 (if not yet installed)
http://msdn.microsoft.com/en-us/netframework/aa569263
1e. Install Kinect SDK v 1.0
http://www.microsoft.com/en-us/kinectfo ... rview.aspx
(we used 1.0.3.190 which was the one available at the time of writing the code)
1f. Download Coding4Fun Kinect Toolkit
http://c4fkinect.codeplex.com/
Make sure you get the version for Kinect SDK 1.0
(this offers a class that extends the Kinect classes with a couple of useful functions, including the excellent "ScaleTo" function).
1g. Source code (and compiled executable) for Robobuilder_SkeletalTracking VB
http://robosavvy.com/RoboSavvyPages/Rob ... tSDKv1.zip
2. Testing Kinect
Once everything is downloaded and Installed, Open the Kinect Sample Browser and start the "Skeleton Viewer".
Kinect SDK V1 SampleBrowser - Runnng the Skeletal Viewer Sample by RoboSavvy, on Flickr
If you're able to see your Skeleton moving your almost there!
3. Opening the Robobuilder Solution and Running the Sample code
Extract the files in step 1g.
- Inside the Subfolder SkeletalTracking\Bin\release you will find an Executable, ready to run and to control Robobuilder.
- To view and edit code open SkeletalTracking.sln
If you get errors of missing or Invalid references in VB 2010, make sure you have everything installed from point 1.
Next remove any missing References and add them again.
Relevant References are listed bellow:
Setting Project References for the Robobuilder_SkeletalTracking Solution by RoboSavvy, on Flickr
Make sure you set:
- Coding4Fun.Kinect.Winform
- Coding4Fun.Kinect.WPF
- Microsoft.Kinect
- RobobuilderLib
(for RobobuilderLib you can Browse for the RobobuilderLib.dll that is included in the ZIP file containing the complete VB project or you can download it from http://code.google.com/p/robobuildervc/ )
4. Known issues
- The code will only care about the first or default player (while kinect can track 2 people we deliberately choose to get data from FirstOrDefault only).
- While you are being tracked but NOT yet CONNECTED to the Robot you will see 3 circles being updated in real time and these match your head and hand positions.
- Once you Open the COMM port to connect the Robot, the software will set the Robot in its home position and enter DC mode.
The position of the Robot arms will be updated in Real Time but on some PCs the window with the circles may seem to have frozen (the circles won't move anymore and none of the buttons will work).
This "freezing behavior" of the UI does not really affect the movement of the Robot. It doesn't happen on all PCs though; it seems something related to the PCs USB host and not Kinect or the code.
If your Ui freezes, to be able to close the program, cover the kinect sensor with your hand (so that it looses tracking and interrupts the closed loop feed between kinect and Robobuilder). The Window will now be responsive again.
- Another issue is an unhandled Exception if you try to Open a COMM port number that doesn't exist.
While we handle this exception the error seems to be coming from the RobobuilderLib.dll.
- A final recommendation is that you connect the Robobuilder to the computer using the Serial cable and not over Bluetooth.
From our testing when using Bluetooth on this kind of closed loop control system you can really "see" the latency: the arms won't move fluidly (they'll be jerky).
When you switch to the cable the movement becomes much more fluid.
If you want to learn more about Kinect and explore samples that Make use of the Other sensors (Depth Sensor, Audio, etc.) you may use the samples provided with the Kinect SDK v1. The limitation with the V1 SDK is that there are next to none VB samples (they're mostly C#).
To get VB samples you can alternatively download these http://files.ch9.ms/coding4fun/KinectFo ... starts.zip
Be advised however that these samples are for Kinect SDK Beta 2 and won't work with SDK V1.
This file also contains Powerpoint presentations introducing the concepts and sensors of Kinect. These presentations are definitely worth reading.
Next week Marco will be back with Kinect and the Robobuilders so hopefully we'll be able to shoot and post the video.
Regards
Pedro.
Last edited by PedroR on Fri Jun 08, 2012 11:44 am, edited 2 times in total.
by PedroR
Thu Mar 15, 2012 7:01 pm
Hi all
Marco sent us a video a few pictures of of Robobuilder being controlled by kinect as well as the RoboSavvy Robots at the Microsoft booth.
We actually got quite a nice feature there
(this video is also reposted on the first post of the thread as an update)
Here is a photo gallery of our set up at Microsoft Stand:
<object width="600" height="450"> <param name="flashvars" value="offsite=true&lang=en-us&page_show_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2Fshow%2F&page_show_back_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2F&set_id=72157629227024264&jump_to="></param> <param name="movie" value="http://www.flickr.com/apps/slideshow/show.swf?v=109615"></param> <param name="allowFullScreen" value="true"></param><embed type="application/x-shockwave-flash" src="http://www.flickr.com/apps/slideshow/show.swf?v=109615" allowFullScreen="true" flashvars="offsite=true&lang=en-us&page_show_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2Fshow%2F&page_show_back_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2F&set_id=72157629227024264&jump_to=" width="600" height="450"></embed></object>
We are demonstrating the following there.
- 1 Robobuilder being controlled by Kinect (the 5710K with Blue chest)
- 1 Robobuilder being controlled by Microsoft Robotics studio featuring: Speech recognition, Audio Output and PS2 Gamepad controller. (the 5710K with Red Chest)
- 1 Robobuilder Running our embedded OpenWRT solution in conjunction with RoboRealm performing BLOB tracking (the 5720T)
- Finally we're also showing NAO. We're running demonstrations of NAO being controlled by Kinect as well.
Regards
Pedro
Marco sent us a video a few pictures of of Robobuilder being controlled by kinect as well as the RoboSavvy Robots at the Microsoft booth.
We actually got quite a nice feature there
phpBB [media]
(this video is also reposted on the first post of the thread as an update)
Here is a photo gallery of our set up at Microsoft Stand:
<object width="600" height="450"> <param name="flashvars" value="offsite=true&lang=en-us&page_show_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2Fshow%2F&page_show_back_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2F&set_id=72157629227024264&jump_to="></param> <param name="movie" value="http://www.flickr.com/apps/slideshow/show.swf?v=109615"></param> <param name="allowFullScreen" value="true"></param><embed type="application/x-shockwave-flash" src="http://www.flickr.com/apps/slideshow/show.swf?v=109615" allowFullScreen="true" flashvars="offsite=true&lang=en-us&page_show_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2Fshow%2F&page_show_back_url=%2Fphotos%2Frobosavvy%2Fsets%2F72157629227024264%2F&set_id=72157629227024264&jump_to=" width="600" height="450"></embed></object>
We are demonstrating the following there.
- 1 Robobuilder being controlled by Kinect (the 5710K with Blue chest)
- 1 Robobuilder being controlled by Microsoft Robotics studio featuring: Speech recognition, Audio Output and PS2 Gamepad controller. (the 5710K with Red Chest)
- 1 Robobuilder Running our embedded OpenWRT solution in conjunction with RoboRealm performing BLOB tracking (the 5720T)
- Finally we're also showing NAO. We're running demonstrations of NAO being controlled by Kinect as well.
Regards
Pedro
by Kondo
Mon Mar 19, 2012 7:20 pm
Really impressive, congratulations for your excellent work.
I have a question, I have to install all the development environment but it is possible medium-term power to operate the RoboBuilder Kinect in a single executable? And not having to install all the development environment.
Sorry for my English, I'm using the google translator, Greetings
I have a question, I have to install all the development environment but it is possible medium-term power to operate the RoboBuilder Kinect in a single executable? And not having to install all the development environment.
Sorry for my English, I'm using the google translator, Greetings
by PedroR
Tue Mar 20, 2012 4:40 pm
Hi Kondo
If you only want to run the executable you should install the following:
1. Runtime for Microsoft DirectX 9
http://www.microsoft.com/download/en/de ... g=en&id=35
2. IF NOT YET INSTALLED, Install .NET Framework 4
http://msdn.microsoft.com/en-us/netframework/aa569263
3. Install Kinect SDK v 1.0
http://www.microsoft.com/en-us/kinectfo ... rview.aspx
(we used 1.0.3.190 which was the one available at the time of writing the code)
4. Download the ZIP file with source code and Executable
http://robosavvy.com/RoboSavvyPages/Rob ... tSDKv1.zip
Once you download the ZIP file in point 4, look inside the folder "Robobuilder_SkeletalTrackingVB_SDKv1\SkeletalTracking\bin\Release" for the file SkeletalTracking.exe
Run this file and it should now work.
This will install the necessary dependencies for running the EXE file ONLY. (no development environment is installed).
I have not personally tested this solution as all the computers we have for development already have a number of dependencies installed (we don't have a fresh install of windows to validate it).
Please let us know if this works and if not, what si the error that you are getting to be able to help you overcome that.
Kind Regards
Pedro.
If you only want to run the executable you should install the following:
1. Runtime for Microsoft DirectX 9
http://www.microsoft.com/download/en/de ... g=en&id=35
2. IF NOT YET INSTALLED, Install .NET Framework 4
http://msdn.microsoft.com/en-us/netframework/aa569263
3. Install Kinect SDK v 1.0
http://www.microsoft.com/en-us/kinectfo ... rview.aspx
(we used 1.0.3.190 which was the one available at the time of writing the code)
4. Download the ZIP file with source code and Executable
http://robosavvy.com/RoboSavvyPages/Rob ... tSDKv1.zip
Once you download the ZIP file in point 4, look inside the folder "Robobuilder_SkeletalTrackingVB_SDKv1\SkeletalTracking\bin\Release" for the file SkeletalTracking.exe
Run this file and it should now work.
This will install the necessary dependencies for running the EXE file ONLY. (no development environment is installed).
I have not personally tested this solution as all the computers we have for development already have a number of dependencies installed (we don't have a fresh install of windows to validate it).
Please let us know if this works and if not, what si the error that you are getting to be able to help you overcome that.
Kind Regards
Pedro.
by Kondo
Tue Mar 20, 2012 6:18 pm
I've got PedroR!
As you said, it worked the first time.
In the next few days when I can put a video right here.
PedroR Another question, I am administrator of a forum and blogs robotics both in Spanish (Bipedolandia and LimoncelloDigital), I have a translated review everything that is made here of course would put this comes from the Robosavvy forums and cite the author. Do I have to ask permission to somewhere else?
Also I have thought your expound this work in robotics festival in Switzerland next May.
Since comets me and once again thanks, this works great!
As you said, it worked the first time.
In the next few days when I can put a video right here.
PedroR Another question, I am administrator of a forum and blogs robotics both in Spanish (Bipedolandia and LimoncelloDigital), I have a translated review everything that is made here of course would put this comes from the Robosavvy forums and cite the author. Do I have to ask permission to somewhere else?
Also I have thought your expound this work in robotics festival in Switzerland next May.
Since comets me and once again thanks, this works great!
by PedroR
Wed Mar 21, 2012 11:17 am
Hi Kondo
You are authorized to translate and post the interesting bits and pieces from our forum but please make sure to follow these rules:
1) In you copy of the post include:
- the Author (RoboSavvy / user name) and
- include a link to the original post in our forum.
2) In the original post in English on our forum
- Write a post with link to the translated version that you published.
This way users on your forums can come back to RoboSavvy and users on RoboSavvy can go to your forums.
Also note that I can only provide authorization for posts by me, user "robosavvy" or parts published by RoboSavvy.
You should probably check with other users if they let you re post their information/posts as well.
I think that is OK but please be aware of this.
I look forward to seeing your video of your Robot controlled by Kinect.
Regards
Pedro
You are authorized to translate and post the interesting bits and pieces from our forum but please make sure to follow these rules:
1) In you copy of the post include:
- the Author (RoboSavvy / user name) and
- include a link to the original post in our forum.
2) In the original post in English on our forum
- Write a post with link to the translated version that you published.
This way users on your forums can come back to RoboSavvy and users on RoboSavvy can go to your forums.
Also note that I can only provide authorization for posts by me, user "robosavvy" or parts published by RoboSavvy.
You should probably check with other users if they let you re post their information/posts as well.
I think that is OK but please be aware of this.
I look forward to seeing your video of your Robot controlled by Kinect.
Regards
Pedro
by PedroR
Wed Mar 21, 2012 11:54 am
just write the nickname of the Author of the post.
In the case of the posts about Kinect you can read see the author shown is "PedroR" for all of them.
Both Marco and I work for RoboSavvy so in the end it's RoboSavvy's work anyway but as a general rule just quote the username of the person who made the post.
Regards.
In the case of the posts about Kinect you can read see the author shown is "PedroR" for all of them.
Both Marco and I work for RoboSavvy so in the end it's RoboSavvy's work anyway but as a general rule just quote the username of the person who made the post.
Regards.
by PedroR
Mon Mar 26, 2012 9:50 am
Hi Kondo
Thanks for sharing the video. Looks very nice. It is very fun to play with isn't it?
I took the liberty of embedding the video directly on your post. I hope you don't mind about this. It will make it easier for others to view it
Thank you and Regards
Pedro.
Thanks for sharing the video. Looks very nice. It is very fun to play with isn't it?
I took the liberty of embedding the video directly on your post. I hope you don't mind about this. It will make it easier for others to view it
Thank you and Regards
Pedro.