I'd say you're dreaming. The main problem will be the limited RAM.
In 2004, Eric Beiderman managed to get a kernel booting with 2.5MB of RAM, with a lot of functionality removed.
However, that was on x86, and you're talking about ARM. So I tried to build the smallest possible ARM kernel, for the 'versatile' platform (one of the simplest). I turned off all configurable options, including the ones that you're looking for (USB, WiFi, SPI, I2C), to see how small it would get. Now, I'm just referring to the kernel here, and this does not include any userspace components.
The good news: it will fit in your flash. The resulting zImage is 383204 bytes.
The bad news: with 256kB of RAM, it won't be able to boot:
$ size obj/vmlinux
text data bss dec hex filename
734580 51360 14944 800884 c3874 obj/vmlinux
The .text segment is bigger than your available RAM, so the kernel can't decompress, let alone allocate memory to boot, let alone run anything useful.
One workaround would be to use the execute-in-place support (CONFIG_XIP), if your system supports that (ie, it can fetch instructions directly from Flash). However, that means your kernel needs to fit uncompressed in flash, and 734kB > 700kB. Also, the .data and .bss sections total 66kB, leaving abut 190kB for everything else (ie, all dynamically-allocated data structures in the kernel).
That's just the kernel. Without the drivers you need, or any userspace.
So, yes, you're going to need a bit more RAM.
A textbook 32-bit RISC processor core capable of running the no-mmu version of linux doesn't actually need to be that large - the real resource you need is far more RAM (10s of megabytes) than available in any FPGA, so you'll probably want SDRAM on the board and a controller for that in the FPGA.
That said, if you want anything more than a trivial level of performance, you probably want a core with some optimizations (pipelining, etc), and that starts to increase the size somewhat. Adding a full mmu will make memory (re-)allocation more efficient and enable the usual copy-on-write fork() behavior.
Both major FPGA vendors have soft processor cores with available linux ports - Microblaze for Xilinx, Nios II for Altera. You should probably read their docs for specific platform recommendations as it is of course a target that moves with time. A third party core design might be somewhat larger for similar performance, if it is written in a more portable way and not as specifically optimized for a given FPGA family.
Historically there have been chips available combining both a hard processor core (often powerpc) with a region of configurable FPGA fabric. Another option to look at would be a separate processor (likely ARM) on the same board as an FPGA.
A lot of the decision will depend on how tightly you need to couple the processor and FPGA. If you can reduce the problem to configuration registers and a stream of data, it could be as modular as hanging an FPGA board with a fast USB chip off the USB host port of an embedded linux board like a BeagleBoard or RasberryPi. For tighter integration, you may want the FPGA on the same board and sitting on the processor's external bus. Or for low data rates, it's trivial to put an SPI register interface in an FPGA, and UART interfaces are entirely do-able though a bit trickier.
Finally, there is the question if you actually need a full operating system such as linux, or if a more "micro-controller sized" embedded TCP stack would solve your problem while requiring less memory.
Best Answer
As is the same with Linux on full-size machines, the bootloader passes the location of the root as the
rootargument to the kernel. The bootloader itself (usually Das u-Boot) is responsible for checking NOR flash, NAND flash, SD, USB, SATA, PXE, etc. to see if a kernel and optional initrd can be found, and passing options as defined in the board's environment variables.TL;DR: Check the bootloader configuration and kernel command line.