Electronic – Digitally-Controlled Analog Processing for VGA Video Signal

This is why VGA displays didn't happen until long after the Z80.

Block graphics with sprites, or 1 bit per pixel monochrome graphics, or character ROM based display modes were the order of the day. Sometimes you could switch modes between them.

Using character ROMs, the ASCII character code provided most of the address into ROM, with (scan line mod 10) providing 4 LSBs of address. So you stored ASCII character codes into a small (maybe 2kbyte) RAM, and the video controller hardware read a string of 80 from this RAM, (10 times in succession for 10 successive scan lines) to deliver 80 bytes (640 bits) per line.

Those 80 bytes may come from a 128 byte section of the 2K RAM, to simplify the video addressing. Likewise the 10 scan lines come from a 16 byte section of the character ROM to simplify addressing. With a suitable design you can select a different page of character ROM with 14 or 16 bytes per character for a prettier font (and fewer lines of text on screen!) with only minor changes to the video hardware.

The 6845 video controller was a popular device that could handle this sort of addressing and simple bitmapped graphics; it should still be easy to find a lot of information on it.

One thing that used to be common for video graphic controllers is Video RAM or VRAM.

VRAM has two sets of data output pins, and thus two ports that can be used simultaneously. The first port, the DRAM port, is accessed by the host computer in a manner very similar to traditional DRAM. The second port, the video port, is typically read-only and is dedicated to providing a high throughput, serialized data channel for the graphics chipset.

Internally, VRAM reads an entire DRAM row and shifts it out sequentially to the video circuitry. This leave the DRAM available for use by the MPU. VRAM has largely been replaced by the use of SDRAM, "even though it is only single-ported and more overhead is required".

A technique I have used in the past is to use interleaved access to memory. It's a bit complex to explain (the devil is in the details), but I will outline the basics:

Basically the MPU accesses video memory in between pixel accesses by the video-controller. If this timing gets too tight, there are a couple things you can do that will greatly relieve the timing (usually only 1 of these is necessary):

1. You can use 2 RAM chips (or banks) and interleave those using each chip for every-other pixel. In your case, this would effectively slow your pixel clock to 80ns per chip allowing MPU and video-controller access to have windows of 40ns each. This could be extended to more banks interleaving more pixels if necessary. This technique is called Interleaved Memory.
2. You can increase the data-bus size of the video memory. The video-controller would read multiple pixels in a single access and use them sequentially. The MPU would either have a larger data-bus, each access would be directed to the appropriate byte (or word) and byte-selects would be used on the video memory, or a read-modify-write would have to be performed to write to the larger data size. In your case, it would probably be simplest to increase the video memory data bus to 16 or 32 bits (2 or 4 pixels), and probably then use an MPU with the same bus size.

If you interleave video accesses, you may want to consider the use of an FPGA or CPLD for your video memory controller.

Another method is to have 2 separate video memories and use bank-select. The MPU writes to one bank while the other is being used by the video-controller for display. When the MPU is finished writing, the bank accesses are swapped (usually during a sync pulse).