Here, you have to provide the same path that you did when you installed Java earlier. Next, find the line that is exporting the $JAVA_HOME variable and uncomment it. Run the following command to open the file in the editor: sudo vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh Let’s start that with the file hadoop-env.sh. Next, we’ll have to edit a few files to change the config for various Hadoop components. Once you place it in the file, make sure you source it so that it takes effect immediately: zshrc file so that these variables are exported every time automatically instead of you having to do it. Also, it is a good idea to place these export statements in the. Set it to reflect the path where you have placed the Hadoop directory. Anyway, following are the variables I’m talking about: export HADOOP_HOME=/mnt/d/bigdata/hadoop-3.3.1export HADOOP_INSTALL=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/binexport HADOOP_OPTS"=$HADOOP_HOME/lib/nativ"Īs you can see, you only have to change the value of the first environment variable, HADOOP_HOME. The best part is, you have to customize only one variable. We have to set a bunch of environment variables. Because we’re installing Hadoop on our local machine, we’re going to do a single-node deployment, which is also known as pseudo-distributed mode deployment. This will create a directory named hadoop-3.3.1 and place all files and directories inside that directory. To decompress that file, use the following command: tar xzf hadoop-3.3.1.tar.gz As of this writing, the latest version of Hadoop is version 3.3.1, and you can download it from here. So, run the following command to cat the file contents of the key file we just created, and then copy that to the authorized_keys file:įirst step to installing Hadoop is to actually download it. At least I haven’t seen an option to change this behavior. This is especially important because this is what Hadoop expects. Once the keys are generated, you have to copy them over to the list of authorized keys so that you don’t have to enter a password each time you SSH into the machine. For this, run the following command and go through the instructions that you’ll get: ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Once we have both the server and the client installed for SSH, we have to generate keys for authentication. To install OpenSSH, run the following commands in the terminal: sudo apt install openssh-server openssh-client -y Without SSH into localhost, most components of Hadoop wouldn’t work. The next dependency to install is OpenSSH so that Hadoop can SSH into the localhost. To make it permanent, you’ll have to add this command to the. If you just run this command in your terminal, the variable will be exported only for the current session. Export JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |