Create loghub DStream.
Create loghub DStream.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val numReceivers = 2 val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, loghubProject, logStream, loghubGroupName, endpoint, numReceivers accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The number of receivers.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
Create loghub DStream.
Create loghub DStream.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val instanceWorker = "loghub_worker" val lease = "loghub_lease" val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val numReceivers = 2 val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, instanceWorker, lease, loghubProject, logStream, loghubGroupName, endpoint, numReceivers accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of WorkInstance
table, such as "loghub_worker" above.
The name of ShardLease
table, such as "loghub_lease" above.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The number of receivers.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
Create loghub DStream.
Create loghub DStream.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val instanceWorker = "loghub_worker" val lease = "loghub_lease" val loghubProject = "sample-project" val logStream = "sample-logstore" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val numReceivers = 2 val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, loghubProject, logStream, loghubGroupName, endpoint, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val instanceWorker = "loghub_worker" val lease = "loghub_lease" val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, instanceWorker, lease, loghubProject, logStream, loghubGroupName, endpoint, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of WorkInstance
table, such as "loghub_worker" above.
The name of ShardLease
table, such as "loghub_lease" above.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
Create loghub DStream.
Create loghub DStream.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val instanceWorker = "loghub_worker" val lease = "loghub_lease" val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val instanceNameBase = "sample-namebase" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val numReceivers = 2 val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, instanceWorker, lease, loghubProject, logStream, loghubGroupName, instanceNameBase, endpoint, numReceivers accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of WorkInstance
table, such as "loghub_worker" above.
The name of ShardLease
table, such as "loghub_lease" above.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The name base of each loghub instance.
The endpoint of loghub
The number of receivers.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
(Since version 1.0.5) No need to provide "loghubInstanceNameBase" argument
Create loghub DStream.
Create loghub DStream.
val dbHost = "localhost" val dbPort = "3306" val db = "sample-db" val dbUser = "tom" var pwd = "123456" val instanceWorker = "loghub_worker" val lease = "loghub_lease" val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val instanceNameBase = "sample-namebase" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, dbHost, dbPort, db, dbUser, pwd, instanceWorker, lease, loghubProject, logStream, loghubGroupName, instanceNameBase, endpoint, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The host of Mysql needed by loghub.
The port of Mysql needed by loghub.
The name of mysql database.
The username of mysql.
The password of mysql.
The name of WorkInstance
table, such as "loghub_worker" above.
The name of ShardLease
table, such as "loghub_lease" above.
The name of LogService
project
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The name base of each loghub instance.
The endpoint of loghub.
The Aliyun AccessKeyId.
The Aliyun AccessKeySecret.
The storage level.
(Since version 1.0.5) No need to provide "loghubInstanceNameBase" argument
Various utility classes for working with Aliyun LogService.
Prepare Work: