How to export data from hdfs to mysql?

sagar.gitsolution
Posts: 1
Joined: Thu Aug 24, 2017 3:14 am

How to export data from hdfs to mysql?

Postby sagar.gitsolution » Thu Sep 21, 2017 10:47 am

:~$ sqoop export --options-file database.props --table TestTable --input-fields-terminated-by "," --export-dir /user/orderdetails_where/
Warning: /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 17/09/21 12:27:36 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.11.1 17/09/21 12:27:36 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 17/09/21 12:27:36 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 17/09/21 12:27:36 INFO tool.CodeGenTool: Beginning code generation Thu Sep 21 12:27:37 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 17/09/21 12:27:37 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM TestTable AS t LIMIT 1 17/09/21 12:27:37 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM TestTable AS t LIMIT 1 17/09/21 12:27:37 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce Note: /tmp/sqoop- /compile/b9c0f84e10afb97d2c3e32647b9b3d04/TestTable.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 17/09/21 12:27:38 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-sagar_gitsolutio/compile/b9c0f84e10afb97d2c3e32647b9b3d04/TestTable.jar 17/09/21 12:27:38 INFO mapreduce.ExportJobBase: Beginning export of TestTable 17/09/21 12:27:38 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 17/09/21 12:27:39 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 17/09/21 12:27:39 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 17/09/21 12:27:39 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 17/09/21 12:27:39 INFO client.RMProxy: Connecting to ResourceManager at e2e-27-137.e2enetworks.net.in/101.53.130.137:8032 17/09/21 12:27:41 INFO input.FileInputFormat: Total input paths to process : 4 17/09/21 12:27:41 INFO input.FileInputFormat: Total input paths to process : 4 17/09/21 12:27:41 INFO mapreduce.JobSubmitter: number of splits:3 17/09/21 12:27:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505892097982_0075 17/09/21 12:27:42 INFO impl.YarnClientImpl: Submitted application application_1505892097982_0075 17/09/21 12:27:42 INFO mapreduce.Job: The url to track the job: http://e2e-27-137.e2enetworks.net.in:19 ... 7982_0075/ 17/09/21 12:27:42 INFO mapreduce.Job: Running job: job_1505892097982_0075 17/09/21 12:27:49 INFO mapreduce.Job: Job job_1505892097982_0075 running in uber mode : false 17/09/21 12:27:49 INFO mapreduce.Job: map 0% reduce 0% 17/09/21 12:27:56 INFO mapreduce.Job: map 100% reduce 0% 17/09/21 12:27:56 INFO mapreduce.Job: Job job_1505892097982_0075 failed with state FAILED due to: Task failed task_1505892097982_0075_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0

17/09/21 12:27:56 INFO mapreduce.Job: Counters: 12 Job Counters Failed map tasks=1 Killed map tasks=2 Launched map tasks=3 Data-local map tasks=3 Total time spent by all maps in occupied slots (ms)=53780 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=13445 Total vcore-milliseconds taken by all map tasks=13445 Total megabyte-milliseconds taken by all map tasks=27535360 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 17/09/21 12:27:56 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/09/21 12:27:56 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 17.1806 seconds (0 bytes/sec) 17/09/21 12:27:56 INFO mapreduce.ExportJobBase: Exported 0 records. 17/09/21 12:27:56 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:439) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

Return to “Big Data & Hadoop”



cron

Disclaimer

Global Association of Risk Professionals, Inc. (GARP®) does not endorse, promote, review or warrant the accuracy of the products or services offered by EduPristine for FRM® related information, nor does it endorse any pass rates claimed by the provider. Further, GARP® is not responsible for any fees or costs paid by the user to EduPristine nor is GARP® responsible for any fees or costs of any person or entity providing any services to EduPristine Study Program. FRM®, GARP® and Global Association of Risk Professionals®, are trademarks owned by the Global Association of Risk Professionals, Inc

CFA Institute does not endorse, promote, or warrant the accuracy or quality of the products or services offered by EduPristine. CFA Institute, CFA®, Claritas® and Chartered Financial Analyst® are trademarks owned by CFA Institute.

Utmost care has been taken to ensure that there is no copyright violation or infringement in any of our content. Still, in case you feel that there is any copyright violation of any kind please send a mail to abuse@edupristine.com and we will rectify it.