We have a distributed sensor network (12 nodes), using Ethernet as the backend protocol. These are networked at 100Mb/s to a single 1Gb/s link using a 16-port switch.
I've previously seen Java and C code to listen to a port. However our aim is to get data into the hard drive ASAP. The nodes will be transmitting with >50% network utilization meaning the Gb/s link will have a very high utilization. The data throughput will be too high to put into local RAM without issues write-read management.
My question is, what issues do I need to access when piping data as fast as possible to a hard drive. We are a single remit, that of real time data acquisition. Therefore filling RAM and later dump to HD with a gap in data acquisition is unacceptable.
Thanks.
Best Answer
I see no problem in your case except the capacity of HDD is limited.
50 % of 1000 Mbps uplink is about 62.5 MB/s to store. A 1 Tb HDD will be filled in full during less than 16000 seconds (about 4 h 30 min).
62.5 MB/s is not guarantied for an HDD as it write speed is about 50-150 MB/s (from here) therefore you need to parallel 2 or 3 of HDDs (e.g. RAID 0). Even low-cost Mini-ATX motherboards integrates two and more SATA interfaces.
Any modern Core-i3 PC could run that task. You can try/test your PC with Wireshark as a software to run the packet capturing.
A LOM Gigabit Ethernet is typically OK, but if you want stronger time stamping you could use specialized capturing hardware like the Endace DAG capturing card.