Summary:
You can probably double brightness by using a few more port pins.
Use of 4 transistors as column drivers will probably make a significant difference.
Use of a low side segment buffer IC plus high side transistors will give you all the drive you want.
Ports are presently being used "illegally" but damage is not occurring due to low drive of processor relative to LEd current ratings. Once proper drive is available series resistors will be needed to prevent damage.
Here is the data sheet for the 74HC244 buffer mentioned by arminb.
It provides 8 x inverting buffers.
The non inverting version is the 74HC240 - datasheet here .
Both these are OK in principal but notionally only provide 8 mA drive per circuit.
You can get various higher current versions such as these SN74LVC244a in stock at Digikey at 26 cents in ones - a bargain! :-).
Driving LEDS without resistors from a processor is bad practice whether it "works" or not. It loads the IC into "illegal" areas if it doesn't work and risks damaging the display if it does work (and still loads the IC illegally).
Your display currently drives 4 segments at a time x 4 columns.
IF you have 4 more port pins available you can (probably) double the brightness of your display by driving 8 segments at a time x 2 columns x 2 port pins per column. This allows you to multiplex your display at twice the rate so twie as bright IF current is maintained - which is why I doubled up the column drives.
At present you are driving a column of 4 LEDs via a portA pin but sinking it via 4 x port B pins. If you have 4 segments on then the brightness is limited by the ability of the single Port A pin to supply 4 x LEDs. If there is only one segment per column on then PortyA supplies one LED and port B sinks 1 LED BUT sink capacity is usually greater than source capacity on many processors, so PORT A drive will still probably be the limit.
SO by providing 4 x port A high side drivers you should increase brightness. These can be as simple as 4 x transistor emitter followers.
4 x NPN transistor (BC337 or similar)
Collectors to V+
Bases to Port A pins
Emitters to LED Anodes.
Adding a series resistor per LED would be "a good idea" [tm].
Knowing what your port pin source and sink capabilities are is essential if you wish to design the result.
Telling us what your processor is and providing links is always an extremely valuable thing to do.
If using transistors as above does not work well enough you can add per segment drive as well by using an octal buffer, as above, or by using segment drive transistors.
For a low side driver an eg ULN2803 octal driver would do an extremely good job. Using one of these plus high side emitter followers without a series resistor would produce a very very very bright display, but only for one cycle :-(. Yu can get other members of the ULN280x family which have different input drive requirements but the '2803 is generally the most useful.
It looks OK, but the resistors should go on the cathode side. With the resistors on the anode side your display will change in brightness depending on the number of LEDs which are on. A "1" will appear brighter than an "8".
Also keep in mind that the TLC59213 is a registered device: you'll have to latch the data on the inputs to the actual driver with a positive pulse on the CLK input.

I've marked the ULN2003A as optional. If your microcontroller's I/O ports can sink 20 mA (typical for a LED display) then you don't need it. If the LEDs need more you'll probably need the ULN2003A; 20 mA is the limit for most microcontrollers. But check if the total power sink for the device isn't exceeded.
I just wonder why you would use the TLC59213? The latch requires another I/O pin, and you can do what you need from it with just 3 PNP transistors and 3 resistors. That would be a much cheaper solution.
edit re your comment
The Sziklai pair works like a Darlington: current gain is the product of the two transistors' separate current gains, and therefore can be very high. But you probably won't need that. I'm still supposing you have 20 mA LEDs/displays. Then the Port B current is 20 mA per pin. (The four indicator LEDs should be multiplexed, just like the displays, so the sequence is display 1, display 2, indicators, display 1, etc.) It you can't drive that directly (further down more about that) a simple transistor will give you enough gain to get the 20 mA.
For the high side you'll have to drive maximum 7 LEDs simultaneously, at 20 mA that's 140 mA. Many general purpose PNPs will have hFEs of 100 and higher, so a 5 mA base current will be sufficient. Also here no Sziklai or Darlington required.
But there's the 12 V! Means you can't drive the PNP directly from a microcontroller's I/O pin: with the output low there's not a problem, but a high will still drive the transistor, and if you switch to high-impedance input you will have the 12 V on that input, which is, er, very bad!

This is a typical circuit to solve this. It's a bit more involved and for a 12 V supply the TLC59213 may be the better solution. When I mentioned the discrete alternative I was thinking about a 5 V supply. Let me explain it anyway.
Q1 drives 1 display, of which two LEDs are shown. To activate Q1 you place a high level at Q2's base. 5 V input will result in about 1 mA of base current. A typical NPN general purpose transistor like the BC547A will have an hFE of minimum 100, then the collector current would be 100 mA if there wasn't resistor R2. Q1's base will be at 11.3 V and R2 will limit the base current to about 5.5 mA. A BC327 has an hFE of minimum 100, so will be able to drive the required 140 mA.
What's R1 for? If Q2 is off it may draw a small leakage current, though especially at higher temperatures this can be a couple of µA. Without R1 these would flow out of Q1's base, and cause several tenths of mA through the LEDs, which may glow lightly despite being off. R1 will take care of the leakage current. Currents below 60 µA will cause a voltage drop across the resistor less than 600 mV, which is too low for the emitter-base junction to conduct, so all will flow through R1, none through Q1, and the LEDs will remain off.
You would need 3 times Q1, Q2, R1, R2 and R5, and for an 8-digit display even 8 of them. The TLC59213 costs 2.23 dollar at Digikey, so for 3 displays the discrete solution would probably still be cheaper.
With the 12 V supply you'll also need the open-collector ULN2003A driver: when off the PNP's leakage current will otherwise cause 10 V to appear on the microcontroler's I/O pins.
Best Answer
Your minimum value will cause a current of 4x the desired average current to flow, while not exceeding the maximum peak current (both constraints must be applied). Since the peak current is less than 4x the average current, that is the limiting constraint.
So 20mA is your maximum average segment current (that's a heck of a lot of current for a modern display unless you require sunlight visibility). That's the display limitation. For reliability, you should not run the LEDs too close to the maximum, and you should take ambient temperature into account.
That means your whole display will take 8 * 4 * 0.02 = 640mA when the display itself is driven to maximum brightness.
Now you need the driving circuit voltage drops and capability to determine the resistor value. If you are driving directly with the 16F648, you will not be able to get anything like 80mA * 7 (or 8 with the decimal point) from the digit driving output. The digit driver has to drive all the segments simultaneously, preferably without changing voltage drop too much, otherwise you'll see an '8' as much dimmer than a '1' or '-'.
If you limit the digit current to 20mA (read the MCU data sheet for port pin limitation and resulting voltage drop) that means you can only have 20mA/8= 2.5mA per segment peak (625uA average), and your resistors work out to about 1.3K, ignoring the driver drops. The whole display draws 20mA or 1/32 of the maximum. The minimum resistor value is just (5V - 1.8V)/0.0025A = 1.28K
If that display brightness is insufficient, you can add digit drivers only (4 of them) and drive the segments with the MCU. There is a maximum current limitation of the entire chip to sink (or source for common cathode) current in the data sheet which will come into play. Probably 150mA or something like that. So if 150mA is correct, you could have 150mA/64 = 4.6mA/segment average tops without adding segment drivers as well as digit drivers. Your display would then draw 150mA maximum (displaying 8.8.8.8.)
Visual brightness is determined by log(average current). Some displays are much brighter than others at the same current. Driving a 4-digit display to acceptable brightness with just resistors and a PIC requires a fairly good display.
Edit: Thank you for adding the schematic. Now it is clear that you have digit drivers and are using a common-cathode display type. Assuming the transistors are properly saturated they should drop perhaps 0.15V. The absolute maximum current sourced by all ports is 200mA so we should stay well away from that. Pick 150mA for the sake of argument. That's 18mA per segment. The output characteristics of the PIC pins when sourcing are not guaranteed above 3mA (8.5mA when sinking). This is parameter D090 in the datasheet. So if you are going to approach the maximum allowable current you will be at the mercy of the PIC variation from unit-to-unit. Not even 'typical' numbers are given. Older PICs sometimes gave the typical curves.. 15mA would drop about a volt, but it's going to be much worse at high temperatures. PICs sink current much better than they source it.
So if you want to stay within guaranteed characteristics for a conservative design you would use 3mA peak (0.75mA average) and the resistor would be (5V - 0.15V - 0.7V - 1.8V)/0.003 = 1.3K. If you want to go into the nether world of typical performance but guaranteed not damaging (it's just a display after all) you could try 15mA peak (3.5mA average) and a resistor of 130 ohms. The worst-case current should still be less than 20mA.
It would be better to switch to a common anode display and use PNP transistors as digit drivers.