MAX232 chip by TI not functioning correctly

The MAX232 is a charge pump. It fills a cap with charge, then moves it to the other side of the supply rail. For example, if you charge a cap on the 5V rail, then disconnect it and connect the low side of the cap to +5V, the high side will now be at +10V.

In short, the capacitors are how the MAX232 makes its high voltages from the lower supply voltage. It will not function without caps.

If you need to connect the AVR to the computer using the serial port and a "standard" DB9-DB9 serial cable, yes, you need to use an RS232 transceiver in circuit. The AVR's UART outputs 0-5V signaling. The RS232 spec is +/-15V signaling, and I believe it's also "inverted" with compared to UART signaling. You don't have to use specifically the MAX232, but you do have to use a RS232 transceiver. These chips (plus some external capacitors) handle the voltage level conversions (via charge pumping) in either direction as well as the signal inversion, and act as "line drivers" driving longer lengths of wire allowed by the RS232 standard.

Alternatively, you can use a cable that includes such a transceiver circuit embedded in the DB9 shell if you are space constrained on your board. Yet another alternative involving a cable is to use a cable that includes an FTDI FT232 chip in it, which has the advantage of making it look like a COM port over USB to the computer, but you're looking at upwards of $15 for that cable.