{"id":1526,"date":"2024-09-13T16:57:58","date_gmt":"2024-09-13T15:57:58","guid":{"rendered":"https:\/\/france.magicsoftware.com\/blog\/?p=1526"},"modified":"2024-09-16T16:50:10","modified_gmt":"2024-09-16T15:50:10","slug":"hadoop-magic-xpi","status":"publish","type":"post","link":"https:\/\/france.magicsoftware.com\/blog\/magic-xpi\/hadoop-magic-xpi\/","title":{"rendered":"Hadoop – Magic xpi"},"content":{"rendered":"\n

Goal : Handle Hadoop with Magic xpi using Hadoop REST API<\/strong> (Put and Delete file)<\/p>\n\n\n\n

1\/ Download and install Hadoop 2.9.1 to C:\\bigdata\\hadoop-2.9.1 (https:\/\/hadoop.apache.org\/release\/2.9.1.html<\/a>)<\/p>\n\n\n\n

2\/ Set JAVA_HOME<\/strong> to jdk1.8.0_221 (example : c:\\progra~1\\java\\jdk1.8.0_221)<\/p>\n\n\n\n

Set HADOOP_BIN<\/strong> to <hadoop installation directory>\\bin (example : C:\\bigdata\\hadoop-2.9.1\\bin )<\/p>\n\n\n\n

Set HADOOP_BIN_PATH<\/strong> to <hadoop installation directory>\\bin<\/p>\n\n\n\n

Set HADOOP_HOME<\/strong> <hadoop installation directory> (example : C:\\bigdata\\hadoop-2.9.1)<\/p>\n\n\n\n

Add in the PATH<\/strong> : <hadoop installation directory>\\bin, <hadoop installation directory>\\sbin and <hadoop installation directory><\/p>\n\n\n\n

(example : C:\\bigdata\\hadoop-2.9.1\\sbin, Add in the PATH : C:\\bigdata\\hadoop-2.9.1\\bin and Add in the PATH : C:\\bigdata\\hadoop-2.9.1 )<\/p>\n\n\n\n

3\/ Verify that you have these files under bin.<\/p>\n\n\n\n

If hadoop.dll and winutils.exe and winutils.pdb are not present , download and copy these files under bin.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

4\/ Configure these 4 files below (hadoop-env.cmd, core-site.xml, hdfs-site.xml, mapred-site.xml<\/strong>)<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

5\/ Create Data Node and Name Node<\/p>\n\n\n\n

Goto C:\/BigData\/hadoop-2.9.1 and create a folder \u2018data\u2019. Inside the \u2018data\u2019 folder create two folders \u2018datanode\u2019 and \u2018namenode\u2019. Your files on HDFS will reside under the datanode folder.<\/p>\n\n\n\n

6\/ Format Name Node<\/p>\n\n\n\n

Open a new Windows Command Prompt and run below command. It may give you some warnings, ignore them<\/p>\n\n\n\n

hadoop namenode -format<\/strong><\/code><\/pre>\n\n\n\n
7\/ Open a new Windows Command Prompt (admin privilege)<\/p>\n\n\n\n
Create a New Hfds directory<\/p>\n\n\n\n
    hadoop fs -mkdir \/test<\/strong><\/p>\n\n\n\n
Set rights on \/test<\/p>\n\n\n\n
    hdfs dfs -chmod 777 \/test<\/strong><\/p>\n\n\n\n
Check by listing the directory<\/p>\n\n\n\n
    hdfs dfs -ls \/test<\/strong><\/p>\n\n\n\n
8\/ Start Hadoop in a Command Line (with admin privilege)<\/p>\n\n\n\n
enter : start-all.cmd<\/strong><\/p>\n\n\n\n
9\/ Check on your browser that you can have access to Hadoop (http:\/\/localhost:8088)<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
10\/ Check the node<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Check that the node is running (Start a browser and check the data nodes (http:\/\/xpi414-training1:50075\/datanode.html#tab-overview)<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
11\/ Check with Postman that you can upload a file on your Hdfs directory<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can check that on you Hdfs the file is present<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Technical documentation is available here : <\/p>\n\n\n\n
https:\/\/hadoop.apache.org\/docs\/stable\/hadoop-project-dist\/hadoop-hdfs\/WebHDFS.html<\/p>\n\n\n\n
How to Install Hadoop on Windows<\/a><\/blockquote>