List of Issues that can arise with 3D Reconstruction using Stereo Cameras

3d-recon

3D reconstruction is one of those tasks that depends on getting many things right. If you usually have a bad reconstruction or a very weird looking one, chances are that you might have an issue with one of the issues in this list. I strongly advice using openCV when doing 3D reconstruction, as it provides a robust set of functions for each step of 3D reconstruction. These issues in the list are still valid even if you don’t use openCV for reconstruction. Without further ado:

1. Calibration Errors

The calibration is the MOST important step. This can not be said enough. It will serve you well double or triple check it. If everything goes right, you will usually end with an error less 0.5. Anything larger than 1.0 is usually a sign of things going bad. Make sure to use findCheckerboardCorners, then optimize it further with cornerSubPix. Any slight calibration problem will blow up and make your reconstruction fall flat on its face.

2. Calibration Pad Orientation

opencv-wrong-orientation

This is one problem that I faced and it has no solution except to take another image. If you use a square checkerboard pattern you may get this issue. I was informed that using a rectangle pad avoids this issue (as it checks for a set number of corners to detect per side). Whenever you calibrate double check that by visualizing the detected checkerboard corners that your orientation happens in the same order. Usually it’s from top left to bottom right. As seen in the image above, sometimes, one checkerboard patterns starts from top right to bottom left. This is a weird situation and I found no solution to this problem, except taking new images.

3. Ordering of calibration points

Look in to the 2D points and the 3D points and make sure they correspond in position the way you want them to correspond. I prefer them increasing and decreasing in the same order, i.e. when x of 2D points increase, x of 3D points also increase, and when y of 2D points increase then y of 3D points increase. If you also prefer it this way you usually have to flip the Y coordinate of the 2D points. There is no hard and fast rule about this, but make sure it 3D-2D pairs correspond the way you think they correspond.

4. Reprojection Error

After you get your camera matrices make sure to check for the re-projection error. Like the calibration error this is a good indicator of something going wrong. This is usually less than 2.0 to get a decent reconstruction.

5. Automatic Feature Detection

good-matches

As cool as the Flann detection scheme is, there’s a reason that active 3D reconstruction methods are still in business. These automatic detectors rarely give you the points you want  with the accuracy you want. If you closely check them, you will notice that there are errors of a pixel range, so tread carefully.

5. Triangulation Error

After you triangulate points with your correspondence point, again check the re-projection error. They have to correspond to your correspondence points. This not happening again is clear indication that something is amiss. Mapping the texture from these re-projected points is also a great way to find out whether something is wrong. The texture usually gets warped and small errors can easily be detected.

6. Image resolution

Another problem that I faced when reconstructing objects was with small image resolution. When using an image of 640 x 480 for performance reasons, I noticed that my 3D reconstruction had noticeable errors. After double-checking everything I noticed that my resolution wasn’t accurate enough to wipe out perturbations of 3D points. Switching to a high resolution such as 1600 x 1200, solved this problem. This problem can be detected because other than small but noticeable errors in the reconstruction, the shape and the textures will be preserved well. This is a good indication (given that the correspondence is done well) that the image resolution used is not high enough.

Some more things that can go wrong was pointed out to me by Google+ users Kabir Mohammed and Oliver Hamilton. Here they are:

7. The need for a robust rig

I didn’t use a rig. I just used a hand held camera as this was a temporary exercise. But, if you are going to calibrate once and use the rig throughout, you need to make sure to re-calibrate again. As pointed out by Kabir, the tiniest bit of warping or bending of the rig will screw up reconstruction.

8. Synchronizing frames between cameras

If you are using multiple images (or video), then you need to make sure that the cameras are synchronized. Otherwise, you may end up using different image pairs! As Oliver pointed out if you’re moving the cameras around and the frames aren’t captured at the same time it gives the effect of them not being rigidly linked. Usually USB cameras require an external clock/trigger signal to sync start of image integration. Firewire cameras usually sync across the bus themselves.

I hope these pointers helped some of you to iron out any issues you had with 3D reconstruction using a stereo rig. Please let me know in the comments section of any other issues you faced when doing 3D reconstruction.

Making Visual Studio Intellisense work with QT

Disclaimer: This solution worked for Visual Studio 2013 with Qt 5.3

One of the reasons if your Qt program runs fine but the Visual Studio Intellisense does not detect Qt specific variables can be due to the necessary header files not being included. You will notice if you go to Project Properties->C/C++->Additional Include Directories, that there are several include entries defined with using QTDIR as an environment variable.

I’m sure you have figured out the solution now. You have to define this system variable to make Intellisense work. Add a new system variable called QTDIR, and point it to the QT installation with the version you are working with, ex: C:\Qt\5.3\msvc2012_opengl. This should solve your problem.

Note: There seems to be other problems that can cause Intellisense to not work with Qt, and they are pointed out in: http://qt-project.org/faq/answer/intellisense_does_not_work_for_my_qt_application._whats_wrong

Finding a normal (perpendicular) vector on a 2d plane

There are plenty of times when we need a normal vector in game dev and graphics programming (or if we are simply just learning vector math). It is relatively simple for you to find a normal (perpendicular) vector on a 2D plane. Swap the vector components and negate one, i.e. if the vector components are (u, v) swap it (-v, u) or (v, -u). Negating the first component will make the normal vector counter clock wise to the initial vector. Negating the 2nd term will make the normal clockwise to the initial vector.

Normal Vector illustration
Normal Vector Illustration

Solving the Error 127 from Make when cross compiling to raspberry pi

I was following the excellent tutorial from Alessio on cross compiling to raspberry pi from Windows 7. Then, I hit the the dreaded ‘Error 127’ from make. Now, after hours of searching I couldn’t find how to solve this. Then, Bryan has mentioned that you need to install the 32 bit cygwin version and that would work, and works it does.

If you already installed cygwin 64 bit version like me and wondering how to install it,here are some steps:

  1. download the 32 bit setup to a new directory(ex: C:\cygwin32) and run the setup. Make sure you make every path point to this new directory during the setup process.
  2. Now, copy the crosstools directory to this new directory. So, it will be at C:\cygwin32\opt\cross\x-tools\arm-unknown-linux-gnueabi
  3. Change all paths created in Eclipse in the ‘Path’ variable (and Windows Environment variables too, if you added cygwin there) to point to this new location
  4. Now, clean and build the project and it should compile for you!

Another error I ran across is this:

Could not determine GDB version after sending: C:\cygwin32\opt\cross\x-tools\arm-unknown-linux-gnueabi\bin\arm-unknown-linux-gnueabi-gdb.exe –version

If you ran across this you installed python 2.7 instead of 2.6. Re-run the cygwin setup and change the python version to 2.6.x.x and install this python version. Re-run your program and it should be working.

I wrote this so it helps someone else to save a ton of time. HTH.

world space vs node space on cocos2d

If you are doing a game that will use up maps, or large views that goes beyond the size of the screen you will run into issues like me. These issues will most likely be due to your lack of knowledge of world space vs node space.

Let me try to give an example. You have a character sprite, in a large map at position (60,10). Now, think you move you move the map to the right of the current screen by dragging your finger (think Angry birds). Say, you move it by 10 points. So, now your character is at (50, 10) (if you move your map to the right, the character moves to the left by 10 points). Now, if you want to detect a touch on the sprite will it work? Well, if it does then you already must’ve configured it to work with node space coordinates. So, the answer is no. It will not work.

Why did it not work? Now, if you touch the screen at (50, 10) and try to detect that on the sprite (using the bounding box, or a virtual rectangle around the sprite), you will be not get any result. This is because you are using the world space for the touch. The world space will mean the screen coordinates. So, on the screen you touch (50, 10), that’s all right. But, where is the sprite on the underlying map. We just moved the map, not the character. So, on the underlying map, the character is still at (60, 10). So, what you have to do is convert the world space coordinates to the node space coordinates of the map. Now, when you do this, the touch coordinates will get converted to (60, 10) and we will get a hit on the sprite touch detection.

The convenience methods that does this are:

convertToNodeSpace(CCTouch* touchPoint)
convertToNodeSpace(CCPoint point)

Mastering copy paste with vim

When your yanking and deleting, sometimes you come across “oh, shit” situations, where you replaced the text that you yanked with something you deleted or vice versa. Fear not, these two commands will save your day.

What you yanked (or copied) will be saved in “0 register, so to paste what you yanked, the key sequence is: “0p

What you deleted (or cut) will be saved in the “1 register, so to paste what you deleted, the key sequence is: “1p

Of course, the generic version of this is for yanking to the “x register: “xy and pasting is: “xp

The generic version of deleting is kept as an exercise 😛

Building Axis2/C on Mac OS X Mountain Lion 10.8.4

This should work on other Mountain Lion versions as well, but I have not tested it. First of all, let’s install a set of dependencies required for the building process.

The easiest way (at least for me) was to install them through macports.

Note: You need to install Xcode and the Xcode command lilne tools for macports to work. Xcode is a (very) large download. So, it will take some time to install. But, an advantage is many tools ask you to install Xcode since it brings in a lot of useful dependencies useful for dev. provided by Apple itself. So, you’ll need it anyway if you’re planning to do some C development 🙂

After installing macports, you need to install automake, autoconf, libtool, libxml2 and pkgconfig. Type in the following command:


sudo port install automake autoconf libtool libxml2 pkgconfig

Now, we are set to build the source. Check it out using the following command:


svn co https://svn.apache.org/repos/asf/axis/axis2/c/core/trunk axis2-c-trunk

This will checkout the source code the a directory named axis2-c-trunk. Now, we need to apply this patch that fixes a constants issue. Download it and put it inside the directory axis2-c-trunk. Now execute the following commands to patch the source:


cd axis2-c-trunk

patch -p0 < unavailable_constants_macosx_fix.patch

Now, you are ready to build the source. Just execute the build.sh with the following command:


./build.sh

That should create a successful build in the axis2-c-trunk/deploy directory. If it didn’t you probably would need this patch as well. Patch your source the same way as above, and try running the build script again:

patch -p0 < mac_os_build_fix.patch
./build.sh