This is a great example of why the same engineer that designed the circuit should do the layout and routing. The schematic is for communicating the circuit, so pins should definitely be arranged functionally without regard to physical pin order. This is much better for most uses of the schematic.
However, layout and routing is not one of those uses. The solution is to see this as part of the layout and routing problem, not something to be pushed back on the schematic. Most of the time, pins of a port of a microcontroller for example, are near each other. But sometimes not. You have to look at the ratsnest when doing layout, and see if there might be ways to simplify that without impacting the circuit. Yes, you might end up going back and forth between the schematic and the layout, and moving nets around between pins.
Fortunately, this sounds like a bigger deal that it really is. Most of the time, arbitrary connections can be accomodated well enough. Sometimes you have a really tight board with few layers that is really cost-sensitive. In that case, you spend the extra engineering effort to swap things around to simplify the board. Again, that's the unusual case. Most products aren't high enough volume to justify this level of optimization during engineering.
Added
I should have mentioned this earlier, but for dense designs I take some care assigning pins up front. I recently put a 64 pin microcontroller on a 4-layer board. With so many connections in a tight space, routing within a inch or so of the chip is a serious issue. It's easy for signals to get blocked in, requiring significant re-routing of other signals.
What I did was print out the pinout diagram from the datasheet as large as possible. Then I wrote labels around the chip indicating in what directions the various other subsystems would be on the board. For example "EEPROM" at top right, display processor at top left, service port at lower, left, etc. I also created a list of all I/O signals required by each block.
To assign pins, I first crossed off those that had to be fixed. Some of the subsystem positions were suggested by the fixed pin assignments. Yes, this is a iterative process. Once that was all set, I started assigning nearby pins to the I/O lines of the various subsystems. You want to do this in pencil since this is also somewhat iterative. For example, you may find that you should have started assigning the UART pins more to the left, since other things have higher demand to the right and you are running out of pins there.
For a micro this complicated, I dedicate a whole sheet just for the immediate connections of the micro. This shows the power, ground, bypass caps, crystal, programming header, and the like. The I/O connections are just named lines that go to other pages in the schematic. This page is labeled something like "Main controller". The next page perhaps "Main controller peripherals", which shows the things connected to I/O lines that don't need to go anywhere else in the schematic. Examples might be the external EEPROM, the status LED, a relay with contacts brought out to a customer connector, etc.
Note that the above method requires some idea of the layout before assigning pins. Again, this is a iterative process. In this case, I left the flexible I/O signals unconnected to the processor when starting layout. I chose locations for subsystems according to where they connected to the external world, where they would be out of the way when their location didn't matter, etc. The processor was then oriented based on where the fixed things around it, like the crystal and load caps, could fit best. That's when the process described above was started.
This was a rather extreme case. Most of the time I don't need to be this deliberate about assigning I/O pins. For smaller processors where there is less routing congestion, I usually just assign pins in the schematic, then deal with it in the layout. That may mean a few extra vias, but for many boards that's not a big deal.
In RAM chips, you often want all address/data lines to have the same length (the shorter the better), so that the signals travel along simultaneously and the clock frequency could be higher. Address and Data lines have to cover the entire area of the chip, so they are grouped by the physical location of the storage cells they feed, not by logical function.
Heterogeneous devices like micro-controllers have different kind of constraints. Some signals could interfere with each other (like a clock and a sensitive ADC) and cannot go to adjacent pins. Furthermore, distributing similar pins of both sides of the chip can actually improve routing of the PCB. Chances are, the chip will sit in the middle and have to talk to neighbors on both sides. Having all similar pins on one side would make PCB routing a nightmare.
Best Answer
In earlier, simpler days the pinout was dictated by what the silicon layout designer could achieve, the external pin order following closely to the bond pads on the silicon die. For example, in early TTL logic an attempt was made to always have ground on pin 7 and 5V on pin 14 of a 14 pin package, however on the more complex devices this was not achievable at the time. For instance, the 7493 counter had 5V on pin 5 and ground on pin 10.
I have been involved in the design of thick film hybrids, and therefore have seen many logic gates, op-amps, multiplexers, ADCs and DACs etc, in bare silicon, and seen how you can't have bond wires crossing. The modern logic gates are very tiny, not like the old TTL. You can't re-order the pins, but you can rotate the chip into your preferred orientation, which helps a lot.
In the analogue world, most dual or quad op-amps were given a symmetrical layout, like the LM124, but a few were not. But the die layout is critical, and performance gains came from avoiding thermal gradients across the input transistors to minimise offset voltage. Usually a quad of transistors were used, diagonally opposite devices in parallel making the differential pair, but detail like that would not show in the external pinout. However, the position of the output stage, that can dissipate significant heat, will influence the pinout.
The LM124 of about 1974 was a mediocre device (still has some uses), and the need to replace it in high performance designs ensured that almost all high performance quad op-amps follow the same pinout. And so it is with other packages, single and dual, with the exception that the 747 (dual 741) was not replicated, and duals went to an 8 pin package.
Although modern integrated circuits are generally much more complex, faster and lower power CMOS equivalents of most 73 series logic devices are still made. The same pinout is retained, after 50 years, for compatibility reasons, even though the devices are available in much smaller SMD packages.
8 bit memory devices came along when silicon layout techniques had improved, and followed a scheme with the address bits and data bus arranged in order around the device to facilitate laying out the bus on the pcb, with ground and 5V on the corner pins, as usual. The package grew from 24 to 28 and then 32 pins as the memory capacity increased, the original pins being kept in the same physical places with respect to the ground pin, so the added address lines ended up in strange places, and not in a nice sequence. Fortunately ROM/EPROM/FLASH retained the same pins for the main functions, and RAM kept to the same convention as far as possible, so it was fairly easy to change the relative amounts of ROM and RAM in a system, and the bus was consistent between all the memory devices.
Early microprocessors were in some cases arranged to make the bus physically reasonably compatible with the memory devices, but others were notoriously difficult for the layout engineer. Some companies seemed to be more capable than others, but layouts were still being done by hand at that time.
When the time came to mix analogue and digital stuff on one chip, starting with ADCs and DACs, noise was all-important and the pinout reflects that. Only one standard developed for parallel input CMOS DACs of variable resolution, the package getting longer as the number of bits increased. Fortunately most such devices use a serial digital interface nowadays.
Analogue switches again tended to be symmetrical with 4 in a package, one at each set of 3 pins at the end of a row, with the power and grounds in the middle, driven by noise considerations.
By the time the first microcontrollers with analogue circuitry onboard appeared, it is likely that everyone was using CAD for chip design, and with the need to avoid crosstalk between complex digital stuff and analogue the entire layout of the chip had to be carefully considered, and the external pin layout went back to what was possible internally, much as it began. The difference is that now, there is a fair amount of freedom to lay out the digital I/O ports sensibly, with an 8 bit port usually in correct numeric sequence, keeping the analogue stuff well away (ground pin(s) are also very important) so the external layout may end up looking a bit messy. This is not helped by the tendency of some chip designers to try to follow the pinout of some unrelated ancient device like the Intel 8051. But for good analogue performance you don't use the internal facilities in your microcontroller anyway.
When it comes to smart power drivers, class D audio amplifiers and similar, an entire new set of criteria come into play, and the general principle of keeping analogue and digital as far apart as possible still applies, but with the need to avoid thermal gradients in sensitive areas such as the input end of op-amps. So if you see a weird pinout, the manufacturer has designed it that way for a good reason. It is no longer because they were struggling to lay the chip out manually.
The advent of BGA packaging has basically ruined any attempt at rational pin layout. It does give you the opportunity for proper supply distribution and decoupling, as there are multiple ground and supply balls, but you are going to be using some good CAD software and a PCB with many layers to get the I/O where you want it. But at least you don't tend to use parallel data busses in most modern equipment, unless external memory is required.
In summary, the apparently irrational pinouts were, were not, and again are, due to what could be physically designed into the silicon given the design methodology and manufacturing processes of the day, but some of them are solely due to the need to be standard, i.e. copy someone else.
It was also so with valves (tubes) but the complexity level was of course much less.