Meant by “streaming data access” in HDFS

filesystemshadoophdfsstreaming

According to the HDFS Architecture page HDFS was designed for "streaming data access". I'm not sure what that means exactly, but would guess it means an operation like seek is either disabled or has sub-optimal performance. Would this be correct?

I'm interested in using HDFS for storing audio/video files that need to be streamed to browser clients. Most of the streams will be start to finish, but some could have a high number of seeks.

Maybe there is another file system that could do this better?

Best Answer

Streaming just implies that it can offer you a constant bitrate above a certain threshhold when transferring the data, as opposed to having the data come in in bursts or waves.

If HDFS is laid out for streaming, it will probably still support seek, with a bit of overhead it requires to cache the data for a constant stream.

Of course, depending on system and network load, your seeks might take a bit longer.

Related Solutions

Deleting temp directory from HDFS

hadoop fs -stat "%Y" "/path/*" Will output timestamps of everything in /path/. Use that along with a cut off as to what you consider too young and you can have this clean up in a shell script kicked off by cron.

This might be smarter then parsing other things outputted by hadoop fs.

Docker – Running HDFS with only 1 data node – appending fails

I fixed it by setting the dfs.replication property in the client as well as the servers. Anyone who runs into this issue should try this as well.

For the curious, here's the docker-compose file I ended up using for them (should save some time if you need to set up a quick hdfs in docker):

version: "2"
networks:
  platform: {}
services:
  hdatanode:
    image: "uhopper/hadoop-datanode"
    networks:
      platform:
        aliases:
        - "hdatanode"
    environment:
      CORE_CONF_fs_defaultFS: "hdfs://hadoop:8020"
      CLUSTER_NAME: "cluster1"
      HDFS_CONF_dfs_replication: "1"
    depends_on:
    - "hadoop"
  hadoop:
    image: "uhopper/hadoop-namenode"
    networks:
      platform:
        aliases:
        - "hadoop"
    ports:
    - "50070:50070"
    - "8020:8020"
    environment:
      CLUSTER_NAME: "cluster1"
      HDFS_CONF_dfs_replication: "1"

Then, on the client set these three config properties if running inside the docker network:

<property><name>fs.defaultFS</name><value>hdfs://hadoop:8020</value></property>
<property><name>fs.hdfs.impl</name><value>org.apache.hadoop.hdfs.DistributedFileSystem</value></property>
<property><name>dfs.replication</name><value>1</value></property>

If running outside the docker network, but your docker is on localhost, you'll want to change those to:

<property><name>fs.defaultFS</name><value>hdfs://localhost:8020</value></property>
<property><name>fs.hdfs.impl</name><value>org.apache.hadoop.hdfs.DistributedFileSystem</value></property>
<property><name>dfs.replication</name><value>1</value></property>

Instant HDFS for testing/dev purposes!

Best Answer

Related Solutions

Deleting temp directory from HDFS

Docker – Running HDFS with only 1 data node – appending fails

Related Topic