OpenCV Stereo Matching/Calibration

camera-calibrationopencvstereo-3d

I'd initially posted this on the OpenCV forums, but unfortunately, I didn't get too many views/replies so I'm posting here with the hopes that someone might have a direction to please suggest?

I am using the Bumblebee XB3 Stereo Camera and it has 3 lenses. I've spent about three weeks reading forums, tutorials, the Learning OpenCV book and the actual OpenCV documentation on using the stereo calibration and stereo matching functionality. In summary, my issue is that I have a good disparity map generated but very poor point-clouds, that seem skewed/squished and are not representative of the actual scene.

What I have done so far:

Used the OpenCV stereo_calibration and stereo_matching examples to:

Calibrated my stereo camera using chess board images

1) Raw Scene Images: http://answers.opencv.org/upfiles/1380850337241986.jpg
2) Rectified the raw images obtained from the camera using the matrices after camera calibration
: http://answers.opencv.org/upfiles/13808502665723237.png
3) Generated a disparity image from the rectified images using Stereo Matching (SGBM)
:
4) Projected these disparities to a 3D Point Cloud

What I have done so far as elimination towards my problem:

I have tried the 1st and 2nd images, then the 2nd and 3rd lenses and
finally the 1st and 2nd.
I've re-run calibration of my chess-board captures by varying the
distance (closer/farther away)
I have used over 20 stereo pairs for the calibration
Varying Chessboard size used: I have used a 9×6 Chessboard image for
calibration and now switched to using a 8×5 one instead
I've tried using the Block Matching as well as SGBM variants and get
relatively similar results. Getting
better results with SGBM so far.
I've varied the disparity ranges, changed the SAD Window size etc. with
little improvement

What I suspect the problem is:

My disparity image looks relatively acceptable, but the next step is to go to 3D Point cloud using the Q matrix. I suspect, I am not calibrating the cameras correctly to generate the right Q matrix. Unfortunately, I've hit the wall in terms of thinking what else I can do to get a better Q matrix. Can someone please suggest ways ahead?

The other thing that I think might be problematic is the assumptions I am making when using the cv::stereoCalibrate function. For the moment, I individually calibrate each camera to get the camera and distortion (cameraMatrix[0], distCoeffs[0] and cameraMatrix[1], distCoeffs[1]) matrices so it makes the complexity for the stereoCalibrate function a little easier.

stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    imageSize, R, T, E, F,
                    TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                    //CV_CALIB_FIX_ASPECT_RATIO +
                    //CV_CALIB_ZERO_TANGENT_DIST +
                    //CV_CALIB_SAME_FOCAL_LENGTH +
                    CV_CALIB_RATIONAL_MODEL 
                    //CV_CALIB_FIX_K3 + CV_CALIB_FIX_K4 + CV_CALIB_FIX_K5
                    );

Additionally, I think it might be useful to mention how I am going from disparity to point cloud. I am using OpenCV's cv::reprojectImageTo3D and then writing the data to a PCL Point cloud structure. Here is the relevant code:

cv::reprojectImageTo3D( imgDisparity16S, reconstructed3D, Q, false, CV_32F);
  for (int i = 0; i < reconstructed3D.rows; i++)
  {
    for (int j = 0; j < reconstructed3D.cols; j++)
    {
        cv::Point3f cvPoint = reconstructed3D.at<cv::Point3f>(i, j);  
            //Filling in a PCL structure
            pcl::PointXYZRGB point;
            point.x = cvPoint.x;
            point.y = cvPoint.y;
            point.z = cvPoint.z;
            point.rgb = rectified_imgRight.at<cv::Vec3b>(i,j)[0]; //Grey information

            point_cloud_ptr->points.push_back (point);
    }
  }

  point_cloud_ptr->width = (int) point_cloud_ptr->points.size();
  point_cloud_ptr->height = 1;
  pcl::io::savePCDFileASCII("OpenCV-PointCloud.pts", *point_cloud_ptr);

PS: The reason I chose to upload these images are that the scene has some texture, so I was anticipating a reply saying the scene is too homogenous. The cover on the partition and the chair as well are quite rich in terms of texture.

Few Questions:

Can you help me remove the image/disparity plane that seems to be part of the point cloud? Why is this happening?

Is there something obvious I am doing incorrectly? I would post my code, but it is extremely similar to the OpenCV examples provided and I do not think I'm doing anything more creatively. I can if there is a specific section that might be concerning.

In my naive opinion, it seems that the disparity image is OK. But the point cloud is definitely nothing I would have expected from a relatively decent disparity image, it is WAY worse.

If it helps, I've mentioned the Q matrix I obtain after camera calibration, incase something obvious jumps out. Comparing this to the Learning OpenCV book, I don't think there is anything blatantly incorrect …

Q: rows: 4
   cols: 4
   data: [ 1., 0., 0., -5.9767076110839844e+002, 0., 1., 0.,
       -5.0785438156127930e+002, 0., 0., 0., 6.8683948509213735e+002, 0.,
       0., -4.4965180874519222e+000, 0. ]

Thanks for reading and I'll honestly appreciate any suggestions at this point …

Best Answer

I found similar problems when using OpenCV (v.2.4.6) 3D reconstruction function. Some people, like Martin Peris, have implemented again by themselves. http://blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html

Sadly, sometimes I have found problems in both implementations with my data. So, I supposed my problem in these cases is due to a poor extrinsic camera parameter description. Perhaps, it is your case too. :-?

PS. In order to get rid of the background you'll need segment it. Or, at least, check depth values bigger than a threshold once the reconstruction works. handleMissingValues flag only eliminates points at "infinity" PS2. Please, tell us if you solve this problem. I think it is valuable for all comunity. Thnx

The original papers

C++ – How to generate a valid point cloud representation of a pair of stereo images using OpenCV 3.0 StereoSGBM and PCL

After doing some work and some research I found my answer and I am sharing it here so other readers can use.

Nothing was wrong with the conversion algorithm from the disparity image to 3D XYZ (and eventually to a point cloud). The problem was the distance of the objects (that I was taking pictures of) to the cameras and amount of information that was available for the StereoBM or StereoSGBM algorithms to detect similarities between the two images (image pair). In order to get proper 3D point cloud it is required to have a good disparity image and in order to have a good disparity image (assuming you have performed good calibration) make sure of the followings:

1- There should be enough detectable and distinguishable common features available between the two frames (right and left frame). The reason being is that StereoBM or StereoSGBM algorithms look for common features between the two frames and they can easily be fooled by similar things in the two frames which may not necessarily belong to the same objects. I personally think these two matching algorithms have lots of room for improvement. So beware of what you are looking at with your cameras.

2- Objects of interest (the ones that you are interested to have their 3D point cloud model) should be within a certain distance to your cameras. The bigger the base-line is (base line is the distance between the two cameras), the further your objects of interest (targets) can be.

A noisy and distorted disparity image never generates a good 3D point cloud. One thing you can do to improve your disparity images is to use track-bars in your applications so you can adjust the StereoSBM or StereoSGBM parameters until you can see good results (clear and smooth disparity image). Code below is a small and simple example on how to generate track-bars (I wrote it as simple as possible). Use as required:

 int PreFilterType = 0, PreFilterCap = 0, MinDisparity = 0, UniqnessRatio = 0, TextureThreshold = 0,
    SpeckleRange = 0, SADWindowSize = 5, SpackleWindowSize = 0, numDisparities = 0, numDisparities2 = 0, PreFilterSize = 5;


            Ptr<StereoBM> sbm = StereoBM::create(numDisparities, SADWindowSize);  

while(1)
{
            sbm->setPreFilterType(PreFilterType);
            sbm->setPreFilterSize(PreFilterSize);  
            sbm->setPreFilterCap(PreFilterCap + 1);
            sbm->setMinDisparity(MinDisparity-100);
            sbm->setTextureThreshold(TextureThreshold*0.0001);
            sbm->setSpeckleRange(SpeckleRange);
            sbm->setSpeckleWindowSize(SpackleWindowSize);
            sbm->setUniquenessRatio(0.01*UniqnessRatio);
            sbm->setSmallerBlockSize(15);
            sbm->setDisp12MaxDiff(32);

            namedWindow("Track Bar Window", CV_WINDOW_NORMAL);
            cvCreateTrackbar("Number of Disparities", "Track Bar Window", &PreFilterType, 1, 0);
            cvCreateTrackbar("Pre Filter Size", "Track Bar Window", &PreFilterSize, 100);
            cvCreateTrackbar("Pre Filter Cap", "Track Bar Window", &PreFilterCap, 61);
            cvCreateTrackbar("Minimum Disparity", "Track Bar Window", &MinDisparity, 200);
            cvCreateTrackbar("Uniqueness Ratio", "Track Bar Window", &UniqnessRatio, 2500);
            cvCreateTrackbar("Texture Threshold", "Track Bar Window", &TextureThreshold, 10000);
            cvCreateTrackbar("Speckle Range", "Track Bar Window", &SpeckleRange, 500);
            cvCreateTrackbar("Block Size", "Track Bar Window", &SADWindowSize, 100);
            cvCreateTrackbar("Speckle Window Size", "Track Bar Window", &SpackleWindowSize, 200);
            cvCreateTrackbar("Number of Disparity", "Track Bar Window", &numDisparities, 500);

            if (PreFilterSize % 2 == 0)
            {
                PreFilterSize = PreFilterSize + 1;
            }


            if (PreFilterSize2 < 5)
            {
                PreFilterSize = 5;
            }

            if (SADWindowSize % 2 == 0)
            {
                SADWindowSize = SADWindowSize + 1;
            }

            if (SADWindowSize < 5)
            {
                SADWindowSize = 5;
            }


            if (numDisparities % 16 != 0)
            {
                numDisparities = numDisparities + (16 - numDisparities % 16);
            }
        }
}

If you are not getting proper results and smooth disparity image, don't get disappointed. Try using the OpenCV sample images (the one with an orange desk lamp in it) with your algorithm to make sure you have the correct pipe-line and then try taking pictures from different distances and play with StereoBM/StereoSGBM parameters until you can get something useful. I used my own face for this purpose and since I had a very small baseline, I came very close to my cameras (Here is a link to my 3D face point-cloud picture, and hey, don't you dare laughing!!!)1.I was very happy of seeing myself in 3D point-cloud form after a week of struggling. I have never been this happy of seeing myself before!!! ;)

Best Answer

Related Solutions

C++ – Image Processing: Algorithm Improvement for ‘Coca-Cola Can’ Recognition

The original papers

C++ – How to generate a valid point cloud representation of a pair of stereo images using OpenCV 3.0 StereoSGBM and PCL

Related Topic