Linux – Concatenating Files to a Virtual File

fileslinuxloopback

On a Linux system, is there any way to concatenate a series of files into one exposed file for reading and writing while not actually taking up another N bytes of disk space? I was hoping for something like mounting these files via loopback/devmapper to accomplish this.

I have a problem where there are split binary files that can get quite large. I don't want to double my space requirements with massive disk IO just to temporarily read / write contents from these files by cating them all together into one enormous file.

I found this project here, but it seems to have a very specific use case and also depends on perl

Best Answer

You can do that by combining loop devices and device mapper, thanks to "everything is a file". But with one limitation: The file size cannot be changed (as you cannot increase block devices by writing to them). So if you want to append data then you have to create a bigger block device and overwrite the appended dummy data.

# for testing, Create 4 files
echo foo | dd of=block0 cbs=512 conv=block
echo bar | dd of=block1 cbs=512 conv=block
echo bat | dd of=block2 cbs=512 conv=block
echo baz | dd of=block3 cbs=512 conv=block
# Create a loop device for each of the 4 files
losetup /dev/loop0 block0
losetup /dev/loop1 block1
losetup /dev/loop2 block2
losetup /dev/loop3 block3
# Create a device map named "test" using those loop devices
(
    echo "0 1 linear /dev/loop0 0"
    echo "1 1 linear /dev/loop1 0"
    echo "2 1 linear /dev/loop2 0"
    echo "3 1 linear /dev/loop3 0"
) | dmsetup create test
$EDITOR /dev/mapper/test # use overwrite mode only

For extending the file you may create a big sparse file and use that as additional device.

Related Solutions

Linux: How to break a large file into smaller files

Use the split command.

split -b 22 m newfile.txt new would split the file "newfile.txt" into three separate files called newaa, newab and newac each file the size of 22 MB.

Linux – How to run a server on port 80 as a normal user on Linux

Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.

The long answer: you can redirect connections on port 80 to some other port you can open as normal user.

Run as root:

# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):

# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).

EDIT: as per comment question - to delete the above rule:

# iptables -t nat --line-numbers -n -L

This will output something like:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080 redir ports 8088
2    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080

The rule you are interested in is nr. 2, so to delete it:

# iptables -t nat -D PREROUTING 2

Best Answer

Related Solutions

Linux: How to break a large file into smaller files

Linux – How to run a server on port 80 as a normal user on Linux

Related Topic