Magento – Checkout option of Ajax Cart Popup does not redirect to one step checkout

Are you using caching of some sort, e.g. Varnish? If so please check your whitelist options. Sometimes ajax requests are cached thus returning different responses. Also, since you seem to be using an extension for the ajax cart make sure to check the developer docs.

I would suggest to check javascript file for native minicart functionality, it is located in: /skin/frontend/rwd/default/js/minicart.js

check removeItem function, it has the following AJAX logic:

$j.ajax({
            type: 'POST',
            dataType: 'json',
            data: {form_key: cart.formKey},
            url: el.attr('href')
        }).done(function(result) {
            cart.hideOverlay();
            if (result.success) {
                cart.updateCartQty(result.qty);
                cart.updateContentOnRemove(result, el.closest('li'));
            } else {
                cart.showMessage(result);
            }
        }).error(function() {
            cart.hideOverlay();
            cart.showError(cart.defaultErrorMessage);
        });

just copy this logic with some amends in your confirm's callback.

You might have something like this:

    var minicartOptions  = {
    formKey:           "<?php echo $this->getFormKey();?>"
}

var $x=jQuery.noConflict();
$x(document).ready(function() {
    $x("#dialog123").dialog({
        autoOpen: false,
        modal: true
    });
});

$x(".remove-this-item").click(function(e) {
    e.preventDefault();
    var targetUrl = $x(this).attr("href");

    $x("#dialog123").dialog({
        buttons : {
            "Confirm" : function() {
                $j.ajax({
                    type: 'POST',
                    dataType: 'json',
                    data: {form_key: minicartOptions.formKey},
                    url: targetUrl
                }).done(function(result) {
                    /**
                     * @TODO customize success callback according to your needs.
                     */
                    cart.hideOverlay();
                    if (result.success) {
                        cart.updateCartQty(result.qty);
                        cart.updateContentOnRemove(result, el.closest('li'));
                    } else {
                        cart.showMessage(result);
                    }
                }).error(function() {
                    /**
                     * @TODO customize error callback according to your needs.
                     */
                    cart.hideOverlay();
                    cart.showError(cart.defaultErrorMessage);
                });
            }
                window.location.href = targetUrl;
            },
            "Cancel" : function() {
                $x(this).dialog("close");
            }
        }
    });

NOTE: this is ugly version that needs to be refactored first :)