How to install HDFS on Fedora Server Latest

HDFS is a distributed file system designed for storing and processing large datasets across clusters of computers. In this tutorial, we will guide you on how to install HDFS on a Fedora Server Latest.

Requirements

• A Fedora server latest operating system with sudo privileges.
• Java Development Kit (JDK) version 7 or later.

Step 1: Installing Java Development Kit (JDK)

Before installing HDFS, you need to install the Java Development Kit (JDK), which is required to run Hadoop. You can follow the below steps to install JDK on your Fedora server.

1. Update the package index.

sudo dnf update -y

2. Install the JDK.

sudo dnf install java-11-openjdk-devel

3. Verify the installation using the command.

java -version

If the installation is successful, it will display the version of the JDK installed on your system.

Step 2: Downloading and Installing Hadoop

After installing JDK, you can proceed with downloading and installing HDFS.

1. Download the latest stable release of Hadoop from the Apache Hadoop website.

wget https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-X.X.X/hadoop-X.X.X.tar.gz

Note: Replace X.X.X with the latest version number of Hadoop.

2. Extract the downloaded Hadoop archive.

tar -xvzf hadoop-X.X.X.tar.gz

3. Move the extracted Hadoop folder to the /usr/local directory.

sudo mv hadoop-X.X.X /usr/local/hadoop

4. Set the environment variables for Hadoop. Open the /etc/profile file.

sudo nano /etc/profile

Add the following lines to the end of the file.

#HADOOP VARIABLES START
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export PATH=${PATH}:${HADOOP_HOME}/sbin:${HADOOP_HOME}/bin
#HADOOP VARIABLES END

5. Save and close the file.

6. Load the environment variables using the following command.

source /etc/profile

Step 3: Configuring HDFS

After installing Hadoop, you need to configure HDFS by editing the configuration files.

1. Open the hadoop-env.sh file in the Hadoop configuration directory.

sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

2. Add the following line to set the Java home path.

export JAVA_HOME=/usr

3. Save and close the file.

4. Open the core-site.xml file in the Hadoop configuration directory.

sudo nano /usr/local/hadoop/etc/hadoop/core-site.xml

5. Add the following properties inside the configuration tags.

<configuration>
   <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

6. Save and close the file.

7. Open the hdfs-site.xml file in the Hadoop configuration directory.

sudo nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

8. Add the following properties inside the configuration tags.

<configuration>
   <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>
   <property>
      <name>dfs.name.dir</name>
        <value>file:///usr/local/hadoop/hdfs/namenode</value>
   </property>
   <property>
      <name>dfs.data.dir</name>
      <value>file:///usr/local/hadoop/hdfs/datanode</value>
   </property>
</configuration>

9. Save and close the file.

Step 4: Formatting HDFS

Before starting the HDFS, you need to format the HDFS file system. You can use the following command to format the HDFS file system.

hdfs namenode -format

Step 5: Starting HDFS

After configuring and formatting HDFS, you can start the HDFS file system.

1. Start the HDFS daemons using the following command.

start-dfs.sh

2. Verify the HDFS service is running correctly by accessing the web interface at http://localhost:50070.

Conclusion

In this tutorial, we have shown you how to install and configure HDFS on a Fedora Server Latest. You can now use HDFS to store and process large datasets across clusters of computers.

If you want to self-host in an easy, hands free way, need an external IP address, or simply want your data in your own hands, give IPv6.rs a try!

Alternatively, for the best virtual desktop, try Shells!