LTSpice has a model for the controller built in, so you can model the system fairly easily. This doesn't take into account effects from your actual PCB layout. While you could try and model those effects, it's best to just build the circuit go from there.
I generally start with the default values and go from there, unless there's some known stuff I can simulate which will let me adjust them.
I like to simulate converters with various kinds of loads. Simple switched loads like what's in the screenshot can give you some basic indications as to what will happen under certain conditions. You can also use arbitrary sources to force certain loads and see how the converter handles those, but they can cause odd situations to happen if you're not careful.
If you wind up with crazy voltage spikes, try lowering your timestep size.
This also lets you try a whole range of different configurations at once. This screenshot shows the output voltage for 10k steps in compensation resistor from 1k to 300k.
The datasheet actually shows how the system works at the bottom of page 5 in the block diagram. The compensation network sits at the output of the error amplifier.
The error amplifier outputs how far away the output voltage is from a target at any given time, which in this instance is 1.24V. At the target voltage your feedback divider would give 1.24V at the output. It uses this as a part of a calculation to adjust the duty cycle of the MOSFET to achieve the desired output.
Adding the RC filtering to this error signal helps keep the loop stable. If there was no compensation the loop would react so fast it would start to oscillate uncontrollably, as the output shot up and down the feedback system would keep overreacting to correct the output. This would be an underdamped condition.
If the compensation network slows down the error response too much on the other hand the regulator will be slow to react to changes at the output. For instance if something suddenly placed a heavy load on the regulator the output could sag very low before the regulator catches up. This would be an overdamped condition.
The goal of this network is to make sure the regulator can respond fast enough to react to the loads you will be placing on it, but not so fast enough it starts jerking itself around.
Linear and TI both have excellent app notes on current mode boost converter compensation which are a good place to start.
Here's one from linear: http://cds.linear.com/docs/en/application-note/AN149fa.pdf
From my personal experience many people use the given application circuit as a starting point, and then build out from there.
This document by TI is a great resource for understanding current mode control theory at a deeper level. Page 10 has an example of the control loop and transfer function. You can use that as a base and start add more circuit elements in.
Best Answer
I took a look at the datasheet. The value of R3 and both of the capacitors don't seem to be very important. At page 2, in a typical application, R3 is 1kOhm and both capacitors ar 4.7nF. You should go with those values.