forked from caroljmcdonald/mapr-streams-spark
-
Notifications
You must be signed in to change notification settings - Fork 0
UlfAndreasson/mapr-streams-spark
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Install and fire up the Sandbox using the instructions here: http://maprdocs.mapr.com/home/SandboxHadoop/c_sandbox_overview.html. Use an SSH client such as Putty (Windows) or Terminal (Mac) to login. See below for an example: use userid: user01 and password: mapr. For VMWare use: $ ssh user01@ipaddress For Virtualbox use: $ ssh [email protected] -p 2222 You can build this project with Maven using IDEs like Eclipse or NetBeans, and then copy the JAR file to your MapR Sandbox, or you can install Maven on your sandbox and build from the Linux command line. You can use scp to copy your JAR file to the MapR Sandbox: use userid: user01 and password: mapr. For VMWare use: $ scp nameoffile.jar user01@ipaddress:/user/user01/. For Virtualbox use: $ scp -P 2222 nameoffile.jar [email protected]:/user/user01/. Copy the data file using scp to here: /user/user01/data/sms-call.txt - Create the topics to read from and write to in MapR streams: maprcli stream create -path /user/user01/stream -produceperm p -consumeperm p -topicperm p maprcli stream topic create -path /user/user01/stream -topic cdrs -partitions 3 maprcli stream topic create -path /user/user01/stream -topic cdrp -partitions 3 To run the MapR Streams Java producer to produce messages, run the Java producer with the topic and data file arguments: java -cp mapr-streams-spark-1.0.jar:`mapr classpath` com.streamskafka.example.MsgProducer /user/user01/stream:cdrp /user/user01/data/sms-call.txt To Run the Spark Consumer Producer (in separate consoles if you want to run at the same time) run the spark consumer with the topic to read from and write to: spark-submit --class com.sparkkafka.example.SparkKafkaConsumer --master local[2] mapr-streams-spark-1.0.jar localhost:9999 /user/user01/stream:cdrs /user/user01/stream:cdrp To run the MapR Streams Java consumer, run the Java consumer with the topic to read from: java -cp mapr-streams-spark-1.0.jar:`mapr classpath` com.streamskafka.example.MsgConsumer /user/user01/stream:cdrp - For Yarn you should change --master parameter to yarn-client - "--master yarn-client"
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published
Languages
- Scala 53.4%
- Java 46.6%