Magento 2 – How to Extend rules.js to Add Custom Validation Rules

In your case, you need to create a custom theme. You should read more here: http://devdocs.magento.com/guides/v2.1/frontend-dev-guide/themes/theme-create.html

Put the validation rule in your custom theme with following path:

app/design/frontend/Package/themename/Magento_Ui/web/js/lib/validation/rules.js

It's hard to answer your question without creating an example. So I suppose that you need to add your custom validation rule into your custom payment method in Magento 2.

Then, your method render js will look like this:

/**
 * Copyright © 2017 StackOverflow. All rights reserved.
 * See COPYING.txt for license details.
 */
/*browser:true*/
/*global define*/
define(
    [
        'jquery',
        'ko',
        'Magento_Checkout/js/view/payment/default',
        'mage/validation'
    ],
    function ($, ko, Component) {
        'use strict';

        $.validator.addMethod('validate-input-one', function (value) {
            return (value === $('#hidden-input-one').val());
        }, 'You have to select input 1 before place order.');

        $.validator.addMethod('validate-input-two', function (value) {
            return (value === $('#hidden-input-two').val());
        }, 'You have to select input 2 before place order.');

        return Component.extend({

            ...

            /**
             * Custompayment form validation
             *
             * @returns {Boolean}
             */
            validate: function () {
                var form = 'form[data-role=custompayment-form]';
                return $(form).validation() && $(form).validation('isValid');
            },

            ...

        });
    }
);

Now, in your custom payment method template file:

<input type="text"
       id="input-one"
       name="payment[input-one]"
       data-validate="{required:true, 'validate-input-one': true}"
       aria-haspopup="false"
       aria-autocomplete="both"
       autocomplete="off"
       class="input-text"/>
<input type="hidden" id="hidden-input-one" data-bind="value: inputOne" />

As you see, whenever the value attribute of #hidden-input-one is empty, #input-one will not pass the validation.

CONCLUSION

mage/validation is the key.