Hadoop – How to specify username when putting files on HDFS from a remote machine

authenticationauthorizationhadoophdfs

I have a Hadoop cluster setup and working under a common default username "user1". I want to put files into hadoop from a remote machine which is not part of the hadoop cluster. I configured hadoop files on the remote machine in a way that when

hadoop dfs -put file1 ...

is called from the remote machine, it puts the file1 on the Hadoop cluster.

the only problem is that I am logged in as "user2" on the remote machine and that doesn't give me the result I expect. In fact, the above code can only be executed on the remote machine as:

hadoop dfs -put file1 /user/user2/testFolder

However, what I really want is to be able to store the file as:

hadoop dfs -put file1 /user/user1/testFolder

If I try to run the last code, hadoop throws error because of access permissions. Is there anyway that I can specify the username within hadoop dfs command?

I am looking for something like:

hadoop dfs -username user1 file1 /user/user1/testFolder

Best Answer

If you use the HADOOP_USER_NAME env variable you can tell HDFS which user name to operate with. Note that this only works if your cluster isn't using security features (e.g. Kerberos). For example:

HADOOP_USER_NAME=hdfs hadoop dfs -put ...

Related Solutions

In Subversion can I be a user other than the login name

Most Subversion commands take the --username option to specify the username you want to use to the repository. Subversion remembers the last repository username and password used in each working copy, which means, among other things, that if you use svn checkout --username myuser you never need to specify the username again.

As Kamil Kisiel says, when Subversion is accessing the repository directly off the file system (that is, the repository URL is of form file:///path/to/repo or file://file-server/path/to/repo), it uses your file system permissions to access the repository. And when you connect via SSH tunneling (svn+ssh://server/path/to/repo), SVN uses your FS permissions on the server, as determined by your SSH login. In those cases, svn checkout --username may not work for your repository.

Hadoop – how can I increase hdfs capacity

Set the location of the hdfs to a partition with more free space. For hadoop-1.2.1 this can be done by setting the hadoop.tmp.dir in hadoop-1.2.1/conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
   <property>
      <name>fs.default.name</name>
     <value>hdfs://localhost:9000</value>
     </property>
   <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/myUserID/hdfs</value>
    <description>base location for other hdfs directories.</description>
   </property>
</configuration>

Running

df

had said my _home partition was my hard disk, minus 50GB for my /
( _root) partition. The default location for hdfs is /tmp/hadoop-myUserId which is in the / partition. This is where my initial 50GB hdfs size came from.

Creation and confirmation of the partition location of a directory for the hdfs was accomplished by

mkdir ~/hdfs
df -P ~/hdfs | tail -1 | cut -d' ' -f 1

successful implementation was accomplished by

stop-all.sh
start-dfs.sh
hadoop namenode -format
start-all.sh
hadoop dfsadmin -report

which reports the size of the hdfs as the size of my _home partition.

Thank you jtravaglini for the comment/clue.

Best Answer

Related Solutions

In Subversion can I be a user other than the login name

Hadoop – how can I increase hdfs capacity

Related Topic