Quantcast
Channel: Paolo Salvatori's Blog
Viewing all 70 articles
Browse latest View live

Released listener for queues and subscriptions in the Service Bus Explorer

$
0
0

I just released an improved version of the Service Bus Explorer 2.1 and 2.2.

The zip file contains:

  • The source code for the Service Bus Explorer 2.2.1.1. This version of the tool uses the Microsoft.ServiceBus.dll 2.2.1.1 that is compatible with the current version of the Windows Azure Service Bus, but not with the Service Bus 1.1, that is, the current version of the on-premises version of the Service Bus.
  • The Service Bus Explorer 2.1. This version can be used with the Service Bus 1.1. The Service Bus Explorer 2.1 uses the Microsoft.ServiceBus.dll client library which is compatible with the Service Bus for Windows Server 1.1 RTM version, but not with the 1.1 Beta version or the Service Bus for Windows Server 1.0. For this reason, for those of you that are still using the Service Bus for Windows Server version 1.0, I included the old version (1.8) of the Service Bus Explorer in a zip file called 1.8 which in turn is contained in the zip file of the current version. The old version of the Service Bus Explorer uses the Microsoft.ServiceBus.dll 1.8 which is compatible with the Service Bus for Windows Server. For those of you that are instead using the Service Bus for Windows Server 1.1 Beta, you can download the Service Bus Explorer 2.0 from my SkyDrive.
  • The Service Bus Explorer 1.8. This version can be used with the Service Bus 1.0

This version introduces the following updates for both the 2.1 and 2.2 version:

  • Introduced a new command to start one or more listeners for a given queue or subscription. Right click the queue or subscription and select, respectively, Create Queue Listerner or Create Subscription Listener from the context menu as shown in the following picture.

  • In the listener dialog you can set the following properties:
    1. MaxConcurrentCalls: gets or sets the maximum number of concurrent calls to the callback the message pump should initiate.
    2. Refresh Interval (sec): sets a refresh interval in seconds for queue or subscription message count information.
    3. Graph: enable or disable the graph.
    4. AutoComplete: gets or sets a value that indicates whether the message-pump should call Complete(Guid) or Complete(Guid) on messages after the callback has completed processing
    5. Tracking: enable or disable message tracking. When enabled, you can use the Messages tab to access messages.
    6. Logging: enable or disable logging.
    7. Verbose: enable or disable verbose mode when logging is enabled.

  • Press the Start button to start the listener.

  • The Start button turns into the Stop button. 
  • Press the Stop button to stop the listener. 
  • Press the Close the listener dialog. If the listener is open, this operation closes the listener.
  • Press the Clear button to clear tracked messages and log content.
  • Press the Messages tab to access messages. 
  • You can click s message row to access its payload, custom and system properties.

  • Click the the magnifying glass button to define a filter expression for received messages using a SQL Expression (e.g. sys.Size > 300 and sys.Label=’Service Bus Explorer’ and City=’Pisa’). For more information, see SqlFilter.SqlExpression Property.

  • Double click a row or click Repair and Resubmit Message from the context menu to open a message in the a separate dialog. This functionality allows to clone and send the selected messages to a the same or alternative queue or topic in the Service Bus namespace. If you want to edit the payload, system properties or user-defined properties, you have to select, edit and send messages one at a time.

  • Minor changes:
    • Fixed ListView control visualization when the vertical bar is visible
    • Increased ReaderQuotas when reading WCF messages. This adds support for large messages.

Service Bus Explorer 2.3 and 2.1 improved version now available

$
0
0

I just released an improved version of the Service Bus Explorer 2.1 and a new version (2.3) of the tool based on the Microsoft.ServiceBus.dll 2.3.2.0.

The zip file contains:

  • The source code for the Service Bus Explorer 2.3.2.0. This version of the tool uses the Microsoft.ServiceBus.dll 2.3.2.0 that is compatible with the current version of the Windows Azure Service Bus, but not with the Service Bus 1.1, that is, the current version of the on-premises version of the Service Bus.
  • The Service Bus Explorer 2.1.3.0. This version can be used with the Service Bus 1.1. The Service Bus Explorer 2.1 uses the Microsoft.ServiceBus.dll client library which is compatible with the Service Bus for Windows Server 1.1 RTM version, but not with the 1.1 Beta version or the Service Bus for Windows Server 1.0. For this reason, for those of you that are still using the Service Bus for Windows Server version 1.0, I included the old version (1.8) of the Service Bus Explorer in a zip file called 1.8 which in turn is contained in the zip file of the current version. The old version of the Service Bus Explorer uses the Microsoft.ServiceBus.dll 1.8 which is compatible with the Service Bus for Windows Server. For those of you that are instead using the Service Bus for Windows Server 1.1 Beta, you can download the Service Bus Explorer 2.0 from my SkyDrive.
  • The Service Bus Explorer 1.8. This version can be used with the Service Bus 1.0

This version introduces the following features:

  • Improved support for AMQP transport protocol in the Connect Form. For example, when using the Service Bus Explorer to connect to an on-premises namespace, if you select AMQP as Transport Type in the Connect Form, the tool automatically changes the value of the TransportType and RuntimePort parameters:
    • NetMessaging: RuntimePort=9354;TransportType=NetMessaging
    • AMQP: RuntimePort=5671;TransportType=Amqp
  • Added support for the stsEndpoint parameter in the connection string for cloud namespaces. This feature was specifically requested by the Service Bus team.
  • The tool can now use the AMQP transport protocol to read messages from queues, subscriptions and deadletter queues.
  • Added full support to namespace-level SAS connection strings, both in the configuration file and Connect Form.
  • Changed Receive label into Receive and Delete in the ReceiveModeForm to make it clear that the receive operation deletes messages from the underlying queue, subscription or deadletter queue.
  • Fixed a bug when reading messages from the deadletter queue of a queue or subscription.
  • Fixed a bug in the SelectEntityForm: replaced TreeView.TopNode with TreeView.Nodes[0]
  • Implemented support for the new ForwardDeadLetteredMessagesTo property of the QueueDescription and SubscriptionDescription classes.

  • Implemented batching support in the listener for queues and subscriptions. You can now specify a value for the PrefetchCount property used by the MessageReceiver object used by the listener to prefetch multiple messages from a queue or subscription. This greatly improves the overall performance of the listener. Now you can also specify the value for the Mode property of the the MessageReceiver object used by the listener.

To enable or disable this feature, you can use the showMessageCount setting in the configuration file, or use the new Show Message Count checkbox in the Options Form as shown in the picture below.

  • As shown in the figure below, numbers in the property list of queues, topics, subscriptions, and notification hubs are now formatted to make it easier to read their value.

  • Replaced the logo to reflect the recent change of name from Windows Azure to Microsoft Azure.

How to create a Service Bus Namespace and an Event Hub using a PowerShell script

$
0
0

This week I received the same two questions twice from two different customers: how can I create a Service Bus namespace from a PowerShell script? Moreover, after I created the Service Bus namespace, how can I create an event hub from a PowerShell script? The answer to the first question is easy: to create a new Azure Service Bus namespace you can use the New-AzureSBNamespace PowerShell cmdlet. The answer to the second question is also straightforward: to create a new Event Hub you can use the CreateEventHub method of the NamespaceManager class. The second question should be posed in a different way: how can I use the classes of the Microsoft.ServiceBus.dll assembly inside a PowerShell script? The following post shows a PowerShell script that you can use to create a Service Bus namespace, an Event Hub and a Consumer Group for the newly created Event Hub.

<#
    .SYNOPSIS
    This script can be used to provision a namespace and an event hub.

    .DESCRIPTION
    This script can be used to provision a namespace and an event hub.
    In particular, the script allows to specify the following parameters:
    -- Event Hub path
    -- Event hub message retention in days
    -- Event Hub partition count
    -- Event Hub user metadata
    -- Service Bus Namespace
    -- Azure Datacenter location

    .PARAMETER  Path
    Specifies the full path of the event hub.

    .PARAMETER  PartitionCount
    Specifies the current number of shards on the event hub.

    .PARAMETER  MessageRetentionInDays
    Specifies the number of days to retain the events for this event hub.

    .PARAMETER  UserMetadata
    Specifies the user metadata for the event hub.

    .PARAMETER  ConsumerGroupName
    Specifies the name of a custom consumer group.

    .PARAMETER  ConsumerGroupUserMetadata
    Specifies the user metadata for the custom consumer group.

    .PARAMETER  Namespace
    Specifies the name of the Service Bus namespace.

    .PARAMETER  CreateACSNamespace
    Specifies whether  to create an ACS namespace associated to the Service Bus namespace.

    .PARAMETER  Location
    Specifies the location to create the namespace in. The default value is "West Europe".

    Valid values:
    -- East Asia
    -- East US
    -- North Central US
    -- North Europe
    -- West Europe
    -- West US

#>

[CmdletBinding(PositionalBinding=$True)]
Param(
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Path,                                  # required    needs to be alphanumeric    
    [Int]$PartitionCount = 16,                      # optional    default to 16
    [Int]$MessageRetentionInDays = 7,               # optional    default to 7
    [String]$UserMetadata = $null,                  # optional    default to $null
    [String]$ConsumerGroupName = "MyConsumerGroup", # optional    default to "MyConsumerGroup"
    [String]$ConsumerGroupUserMetadata = $null,     # optional    default to $null
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Namespace,                             # required    needs to be alphanumeric
    [Bool]$CreateACSNamespace = $False,             # optional    default to $false
    [String]$Location = "West Europe"               # optional    default to "West Europe"
    )

# Set the output level to verbose and make the script stop on error
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"

# WARNING: Make sure to reference the latest version of the \Microsoft.ServiceBus.dll
Write-Output "Adding the [Microsoft.ServiceBus.dll] assembly to the script..."
Add-Type -Path "C:\Projects\Azure\ServiceBusExplorerNew\ServiceBusExplorer2.4Git\bin\Debug\Microsoft.ServiceBus.dll"
Write-Output "The [Microsoft.ServiceBus.dll] assembly has been successfully added to the script."

# Mark the start time of the script execution
$startTime = Get-Date

# Create Azure Service Bus namespace
$CurrentNamespace = Get-AzureSBNamespace -Name $Namespace


# Check if the namespace already exists or needs to be created
if ($CurrentNamespace)
{
    Write-Output "The namespace [$Namespace] already exists in the [$($CurrentNamespace.Region)] region."
}
else
{
    Write-Host "The [$Namespace] namespace does not exist."
    Write-Output "Creating the [$Namespace] namespace in the [$Location] region..."
    New-AzureSBNamespace -Name $Namespace -Location $Location -CreateACSNamespace $CreateACSNamespace -NamespaceType Messaging
    $CurrentNamespace = Get-AzureSBNamespace -Name $Namespace
    Write-Host "The [$Namespace] namespace in the [$Location] region has been successfully created."
}

# Create the NamespaceManager object to create the event hub
Write-Host "Creating a NamespaceManager object for the [$Namespace] namespace..."
$NamespaceManager = [Microsoft.ServiceBus.NamespaceManager]::CreateFromConnectionString($CurrentNamespace.ConnectionString);
Write-Host "NamespaceManager object for the [$Namespace] namespace has been successfully created."

# Check if the event hub already exists
if ($NamespaceManager.EventHubExists($Path))
{
    Write-Output "The [$Path] event hub already exists in the [$Namespace] namespace."
}
else
{
    Write-Output "Creating the [$Path] event hub in the [$Namespace] namespace: PartitionCount=[$PartitionCount] MessageRetentionInDays=[$MessageRetentionInDays]..."
    $EventHubDescription = New-Object -TypeName Microsoft.ServiceBus.Messaging.EventHubDescription -ArgumentList $Path
    $EventHubDescription.PartitionCount = $PartitionCount
    $EventHubDescription.MessageRetentionInDays = $MessageRetentionInDays
    $EventHubDescription.UserMetadata = $UserMetadata
    $NamespaceManager.CreateEventHub($EventHubDescription);
    Write-Host "The [$Path] event hub in the [$Namespace] namespace has been successfully created."
}

# Create the consumer group if not exists
Write-Output "Creating the consumer group [$ConsumerGroupName] for the [$Path] event hub..."
$ConsumerGroupDescription = New-Object -TypeName Microsoft.ServiceBus.Messaging.ConsumerGroupDescription -ArgumentList $Path, $ConsumerGroupName
$ConsumerGroupDescription.UserMetadata = $ConsumerGroupUserMetadata
$NamespaceManager.CreateConsumerGroupIfNotExists($ConsumerGroupDescription);
Write-Host "The consumer group [$ConsumerGroupName] for the [$Path] event hub has been successfully created."

# Mark the finish time of the script execution
$finishTime = Get-Date

# Output the time consumed in seconds
$TotalTime = ($finishTime - $startTime).TotalSeconds
Write-Output "The script completed in $TotalTime seconds."

Note 1: as highlighted in the comments, make sure to load the latest version of the Microsoft.ServiceBus.dll assembly in the script.

Note 2: you can follow the same approach to create topics, queues, subscriptions and notification hubs instead of event hubs.

The picture below shows the result of the execution of the script with the following parameters:

powershell .\CreateEventHub.ps1 -Path ioteventhub -PartitionCount 32 -MessageRetentionInDays 3 -UserMetadata ‘This event hub is used by the devices of the IoT solution’ -ConsumerGroupName iotconsumergroup -ConsumerGroupUserMetadata ‘This consumer group is used by the IoT solution’ -Namespace iotsolution -Location ‘West Europe’

CommandPrompt

I’d like to remark that this kind of scripts should be:

  • parametric to be reusable
  • included in the source control system (Visual Studio Online or GitHub) along with the solution and versioned as the application code
  • executed as part of the Continuous Delivery process used to automatically deploy the solution to a QA environment when a new version of the application is built

In fact, you probably heard the term DevOps more and more times. The term remarks the need to integrate development and operations tasks to build software solutions efficiently over time. This goal can be achieved using a process in which you develop an application, deploy it, learn from production usage of it, change it in response to what you learn, and repeat the cycle quickly and reliably.

To achieve this result, you have to build a development and deployment cycle that is repeatable, reliable, predictable, and has low cycle time. In particular, to create the staging and production environment for an Azure application and to deploy the solution in an automated way and integrate this task in the Continuous integration (CI) and Continuous delivery (CD) processes, you should use parametric PowerShell scripts.    

Note: Continuous integration (CI) means that whenever a developer checks in code to the source repository, a build is automatically triggered. Continuous delivery (CD) takes this one-step further: after a build and automated unit tests are successful, you automatically deploy the application to an environment where you can do more in-depth testing.

You can download the script along with the PowerShell scripts to create queues, topics and Subscription on MSDN Code Gallery: Service Bus PowerShell Scripts

How to create Service Bus queues, topics and subscriptions using a PowerShell script

$
0
0

In my previous post How to create a Service Bus Namespace and an Event Hub using a PowerShell script I provided a PowerShell script to create a Service Bus namespace, an event hub and a custom consumer group. In this post I provide 3 new scripts that you can use to automate the creation of queues, topics and subscriptions:

CreateQueue.ps1

This script can be used to create a subscription for an existing queue.

<#
    .SYNOPSIS
    This script can be used to provision a namespace and queue.

    .DESCRIPTION
    This script can be used to provision a namespace and a queue.

    .PARAMETER  Path
    Specifies the full path of the queue.

    .PARAMETER  AutoDeleteOnIdle
    Specifies after how many minutes the queue is automatically deleted. The minimum duration is 5 minutes.

    .PARAMETER  DefaultMessageTimeToLive
    Specifies default message time to live value in minutes. This is the duration after which the message expires,
    starting from when the message is sent to Service Bus. This is the default value used when TimeToLive is not set on a message itself.
    Messages older than their TimeToLive value will expire and no longer be retained in the message store.
    Subscribers will be unable to receive expired messages.A message can have a lower TimeToLive value than that specified here,
    but by default TimeToLive is set to MaxValue. Therefore, this property becomes the default time to live value applied to messages.

    .PARAMETER  DuplicateDetectionHistoryTimeWindow
    Specifies the duration of the duplicate detection history in minutes. The default value is 10 minutes.

    .PARAMETER  EnableBatchedOperations
    Specifies whether server-side batched operations are enabled.

    .PARAMETER  EnableDeadLetteringOnMessageExpiration
    Specifies whether this queue has dead letter support when a message expires.

    .PARAMETER  EnableExpress
    Specifies whether to enable the queue to be partitioned across multiple message brokers.
    An express queue holds a message in memory temporarily before writing it to persistent storage.

    .PARAMETER  EnablePartitioning
    Specifies whether the queue to be partitioned across multiple message brokers is enabled.

    .PARAMETER  ForwardDeadLetteredMessagesTo
    Specifies the path to the recipient to which the dead lettered message is forwarded.

    .PARAMETER  ForwardTo
    Specifies the path to the recipient to which the message is forwarded.

    .PARAMETER  IsAnonymousAccessible
    Specifies whether the message is anonymous accessible.

    .PARAMETER  LockDuration
    Specifies the duration of a peek lock in seconds; that is, the amount of time that the message is locked for other receivers.
    The maximum value for LockDuration is 5 minutes; the default value is 1 minute.

    .PARAMETER  MaxDeliveryCount
    Specifies the maximum delivery count. A message is automatically deadlettered after this number of deliveries.

    .PARAMETER  MaxSizeInMegabytes
    Specifies the maximum size of the queue in megabytes, which is the size of memory allocated for the queue.

    .PARAMETER  RequiresDuplicateDetection
    Specifies whether the queue requires duplicate detection.

    .PARAMETER  RequiresSession
    Specifies whether the queue supports the concept of session.

    .PARAMETER  SupportOrdering
    Specifies whether the queue supports ordering.

    .PARAMETER  UserMetadata
    Specifies the user metadata.

    .PARAMETER  Namespace
    Specifies the name of the Service Bus namespace.

    .PARAMETER  CreateACSNamespace
    Specifies whether  to create an ACS namespace associated to the Service Bus namespace.

    .PARAMETER  Location
    Specifies the location to create the namespace in. The default value is "West Europe".

    Valid values:
    -- East Asia
    -- East US
    -- Central US
    -- North Central US
    -- North Europe
    -- West Europe
    -- West US

    .NOTES
    Author     : Paolo Salvatori
    Twitter    : @babosbird
    Blog       : http://blogs.msdn.com/b/paolos/
#>

[CmdletBinding(PositionalBinding=$True)]
Param(
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Path,                                           # required    needs to be alphanumeric    
    [Int]$AutoDeleteOnIdle = -1,                             # optional    default to -1
    [Int]$DefaultMessageTimeToLive = -1,                     # optional    default to -1
    [Int]$DuplicateDetectionHistoryTimeWindow = 10,          # optional    default to 10
    [Bool]$EnableBatchedOperations = $True,                  # optional    default to true
    [Bool]$EnableDeadLetteringOnMessageExpiration = $False,  # optional    default to false
    [Bool]$EnableExpress = $False,                           # optional    default to false
    [Bool]$EnablePartitioning = $False,                      # optional    default to false
    [String]$ForwardDeadLetteredMessagesTo = $Null,          # optional    default to null
    [String]$ForwardTo = $Null,                              # optional    default to null
    [Bool]$IsAnonymousAccessible = $False,                   # optional    default to false
    [Int]$LockDuration = 30,                                 # optional    default to 30
    [Int]$MaxDeliveryCount = 10,                             # optional    default to 10
    [Int]$MaxSizeInMegabytes = 1024,                         # optional    default to 1024
    [Bool]$RequiresDuplicateDetection = $False,              # optional    default to false
    [Bool]$RequiresSession = $False,                         # optional    default to false
    [Bool]$SupportOrdering = $True,                          # optional    default to true
    [String]$UserMetadata = $Null,                           # optional    default to null
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Namespace,                                      # required    needs to be alphanumeric
    [Bool]$CreateACSNamespace = $False,                      # optional    default to $false
    [String]$Location = "West Europe"                        # optional    default to "West Europe"
    )

# Set the output level to verbose and make the script stop on error
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"

# WARNING: Make sure to reference the latest version of the \Microsoft.ServiceBus.dll
Write-Output "Adding the [Microsoft.ServiceBus.dll] assembly to the script..."
Add-Type -Path "C:\Projects\Azure\ServiceBusExplorerNew\ServiceBusExplorer2.4Git\bin\Debug\Microsoft.ServiceBus.dll"
Write-Output "The [Microsoft.ServiceBus.dll] assembly has been successfully added to the script."

# Mark the start time of the script execution
$startTime = Get-Date

# Create Azure Service Bus namespace
$CurrentNamespace = Get-AzureSBNamespace -Name $Namespace


# Check if the namespace already exists or needs to be created
if ($CurrentNamespace)
{
    Write-Output "The namespace [$Namespace] already exists in the [$($CurrentNamespace.Region)] region."
}
else
{
    Write-Host "The [$Namespace] namespace does not exist."
    Write-Output "Creating the [$Namespace] namespace in the [$Location] region..."
    New-AzureSBNamespace -Name $Namespace -Location $Location -CreateACSNamespace $CreateACSNamespace -NamespaceType Messaging
    $CurrentNamespace = Get-AzureSBNamespace -Name $Namespace
    Write-Host "The [$Namespace] namespace in the [$Location] region has been successfully created."
}

# Create the NamespaceManager object to create the queue
Write-Host "Creating a NamespaceManager object for the [$Namespace] namespace..."
$NamespaceManager = [Microsoft.ServiceBus.NamespaceManager]::CreateFromConnectionString($CurrentNamespace.ConnectionString);
Write-Host "NamespaceManager object for the [$Namespace] namespace has been successfully created."

# Check if the queue already exists
if ($NamespaceManager.QueueExists($Path))
{
    Write-Output "The [$Path] queue already exists in the [$Namespace] namespace."
}
else
{
    Write-Output "Creating the [$Path] queue in the [$Namespace] namespace..."
    $QueueDescription = New-Object -TypeName Microsoft.ServiceBus.Messaging.QueueDescription -ArgumentList $Path
    if ($AutoDeleteOnIdle -ge 5)
    {
        $QueueDescription.AutoDeleteOnIdle = [System.TimeSpan]::FromMinutes($AutoDeleteOnIdle)
    }
    if ($DefaultMessageTimeToLive -gt 0)
    {
        $QueueDescription.DefaultMessageTimeToLive = [System.TimeSpan]::FromMinutes($DefaultMessageTimeToLive)
    }
    if ($DuplicateDetectionHistoryTimeWindow -gt 0)
    {
        $QueueDescription.DuplicateDetectionHistoryTimeWindow = [System.TimeSpan]::FromMinutes($DuplicateDetectionHistoryTimeWindow)
    }
    $QueueDescription.EnableBatchedOperations = $EnableBatchedOperations
    $QueueDescription.EnableDeadLetteringOnMessageExpiration = $EnableDeadLetteringOnMessageExpiration
    $QueueDescription.EnableExpress = $EnableExpress
    $QueueDescription.EnablePartitioning = $EnablePartitioning
    $QueueDescription.ForwardDeadLetteredMessagesTo = $ForwardDeadLetteredMessagesTo
    $QueueDescription.ForwardTo = $ForwardTo
    $QueueDescription.IsAnonymousAccessible = $IsAnonymousAccessible
    if ($LockDuration -gt 0)
    {
        $QueueDescription.LockDuration = [System.TimeSpan]::FromSeconds($LockDuration)
    }
    $QueueDescription.MaxDeliveryCount = $MaxDeliveryCount
    $QueueDescription.MaxSizeInMegabytes = $MaxSizeInMegabytes
    $QueueDescription.RequiresDuplicateDetection = $RequiresDuplicateDetection
    $QueueDescription.RequiresSession = $RequiresSession
    if ($EnablePartitioning)
    {
        $QueueDescription.SupportOrdering = $False
    }
    else
    {
        $QueueDescription.SupportOrdering = $SupportOrdering
    }
    $QueueDescription.UserMetadata = $UserMetadata
    $NamespaceManager.CreateQueue($QueueDescription);
    Write-Host "The [$Path] queue in the [$Namespace] namespace has been successfully created."
}

# Mark the finish time of the script execution
$finishTime = Get-Date

# Output the time consumed in seconds
$TotalTime = ($finishTime - $startTime).TotalSeconds
Write-Output "The script completed in $TotalTime seconds."

The picture below shows the result of the execution of the script with the following parameters:

powershell .\CreateQueue.ps1 -Path iotqueue -Namespace iotsolution -DefaultMessageTimeToLive 5 -EnableExpress $True -EnablePartitioning $True -MaxSizeInMegabytes 3072 -UserMetadata ‘This queue is used by the IoT solution’    

CreateQueue

 

CreateTopic.ps1

This script can be used to provision a namespace and topic.

<#
    .SYNOPSIS
    This script can be used to provision a namespace and topic.

    .DESCRIPTION
    This script can be used to provision a namespace and a topic.

    .PARAMETER  Path
    Specifies the full path of the topic.

    .PARAMETER  AutoDeleteOnIdle
    Specifies after how many minutes the topic is automatically deleted. The minimum duration is 5 minutes.

    .PARAMETER  DefaultMessageTimeToLive
    Specifies default message time to live value in minutes. This is the duration after which the message expires,
    starting from when the message is sent to Service Bus. This is the default value used when TimeToLive is not set on a message itself.
    Messages older than their TimeToLive value will expire and no longer be retained in the message store.
    Subscribers will be unable to receive expired messages.A message can have a lower TimeToLive value than that specified here,
    but by default TimeToLive is set to MaxValue. Therefore, this property becomes the default time to live value applied to messages.

    .PARAMETER  DuplicateDetectionHistoryTimeWindow
    Specifies the duration of the duplicate detection history in minutes. The default value is 10 minutes.

    .PARAMETER  EnableBatchedOperations
    Specifies whether server-side batched operations are enabled.

    .PARAMETER  EnableExpress
    Specifies whether to enable the topic to be partitioned across multiple message brokers.
    An express topic holds a message in memory temporarily before writing it to persistent storage.

    .PARAMETER  EnableFilteringMessagesBeforePublishing
    Specifies whether messages should be filtered before publishing.

    .PARAMETER  EnablePartitioning
    Specifies whether the topic to be partitioned across multiple message brokers is enabled.

    .PARAMETER  IsAnonymousAccessible
    Specifies whether the message is anonymous accessible.

    .PARAMETER  MaxSizeInMegabytes
    Specifies the maximum size of the topic in megabytes, which is the size of memory allocated for the topic.

    .PARAMETER  RequiresDuplicateDetection
    Specifies whether the topic requires duplicate detection.

    .PARAMETER  SupportOrdering
    Specifies whether the topic supports ordering.

    .PARAMETER  UserMetadata
    Specifies the user metadata.

    .PARAMETER  Namespace
    Specifies the name of the Service Bus namespace.

    .PARAMETER  CreateACSNamespace
    Specifies whether  to create an ACS namespace associated to the Service Bus namespace.

    .PARAMETER  Location
    Specifies the location to create the namespace in. The default value is "West Europe".

    Valid values:
    -- East Asia
    -- East US
    -- Central US
    -- North Central US
    -- North Europe
    -- West Europe
    -- West US

    .NOTES
    Author     : Paolo Salvatori
    Twitter    : @babosbird
    Blog       : http://blogs.msdn.com/b/paolos/
#>

[CmdletBinding(PositionalBinding=$True)]
Param(
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Path,                                           # required    needs to be alphanumeric    
    [Int]$AutoDeleteOnIdle = -1,                             # optional    default to -1
    [Int]$DefaultMessageTimeToLive = -1,                     # optional    default to -1
    [Int]$DuplicateDetectionHistoryTimeWindow = 10,          # optional    default to 10
    [Bool]$EnableBatchedOperations = $True,                  # optional    default to true
    [Bool]$EnableFilteringMessagesBeforePublishing = $False, # optional    default to false
    [Bool]$EnableExpress = $False,                           # optional    default to false
    [Bool]$EnablePartitioning = $False,                      # optional    default to false
    [Bool]$IsAnonymousAccessible = $False,                   # optional    default to false
    [Int]$MaxSizeInMegabytes = 1024,                         # optional    default to 1024
    [Bool]$RequiresDuplicateDetection = $False,              # optional    default to false
    [Bool]$SupportOrdering = $True,                          # optional    default to true
    [String]$UserMetadata = $Null,                           # optional    default to null
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Namespace,                                      # required    needs to be alphanumeric
    [Bool]$CreateACSNamespace = $False,                      # optional    default to $false
    [String]$Location = "West Europe"                        # optional    default to "West Europe"
    )

# Set the output level to verbose and make the script stop on error
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"

# WARNING: Make sure to reference the latest version of the \Microsoft.ServiceBus.dll
Write-Output "Adding the [Microsoft.ServiceBus.dll] assembly to the script..."
Add-Type -Path "C:\Projects\Azure\ServiceBusExplorerNew\ServiceBusExplorer2.4Git\bin\Debug\Microsoft.ServiceBus.dll"
Write-Output "The [Microsoft.ServiceBus.dll] assembly has been successfully added to the script."

# Mark the start time of the script execution
$startTime = Get-Date

# Create Azure Service Bus namespace
$CurrentNamespace = Get-AzureSBNamespace -Name $Namespace


# Check if the namespace already exists or needs to be created
if ($CurrentNamespace)
{
    Write-Output "The namespace [$Namespace] already exists in the [$($CurrentNamespace.Region)] region."
}
else
{
    Write-Host "The [$Namespace] namespace does not exist."
    Write-Output "Creating the [$Namespace] namespace in the [$Location] region..."
    New-AzureSBNamespace -Name $Namespace -Location $Location -CreateACSNamespace $CreateACSNamespace -NamespaceType Messaging
    $CurrentNamespace = Get-AzureSBNamespace -Name $Namespace
    Write-Host "The [$Namespace] namespace in the [$Location] region has been successfully created."
}

# Create the NamespaceManager object to create the topic
Write-Host "Creating a NamespaceManager object for the [$Namespace] namespace..."
$NamespaceManager = [Microsoft.ServiceBus.NamespaceManager]::CreateFromConnectionString($CurrentNamespace.ConnectionString);
Write-Host "NamespaceManager object for the [$Namespace] namespace has been successfully created."

# Check if the topic already exists
if ($NamespaceManager.TopicExists($Path))
{
    Write-Output "The [$Path] topic already exists in the [$Namespace] namespace."
}
else
{
    Write-Output "Creating the [$Path] topic in the [$Namespace] namespace..."
    $TopicDescription = New-Object -TypeName Microsoft.ServiceBus.Messaging.TopicDescription -ArgumentList $Path
    if ($AutoDeleteOnIdle -ge 5)
    {
        $TopicDescription.AutoDeleteOnIdle = [System.TimeSpan]::FromMinutes($AutoDeleteOnIdle)
    }
    if ($DefaultMessageTimeToLive -gt 0)
    {
        $TopicDescription.DefaultMessageTimeToLive = [System.TimeSpan]::FromMinutes($DefaultMessageTimeToLive)
    }
    if ($DuplicateDetectionHistoryTimeWindow -gt 0)
    {
        $TopicDescription.DuplicateDetectionHistoryTimeWindow = [System.TimeSpan]::FromMinutes($DuplicateDetectionHistoryTimeWindow)
    }
    $TopicDescription.EnableBatchedOperations = $EnableBatchedOperations
    $TopicDescription.EnableExpress = $EnableExpress
    $TopicDescription.EnableFilteringMessagesBeforePublishing = $EnableFilteringMessagesBeforePublishing
    $TopicDescription.EnablePartitioning = $EnablePartitioning
    $TopicDescription.IsAnonymousAccessible = $IsAnonymousAccessible
    $TopicDescription.MaxSizeInMegabytes = $MaxSizeInMegabytes
    $TopicDescription.RequiresDuplicateDetection = $RequiresDuplicateDetection
    if ($EnablePartitioning)
    {
        $TopicDescription.SupportOrdering = $False
    }
    else
    {
        $TopicDescription.SupportOrdering = $SupportOrdering
    }
    $TopicDescription.UserMetadata = $UserMetadata
    $NamespaceManager.CreateTopic($TopicDescription);
    Write-Host "The [$Path] topic in the [$Namespace] namespace has been successfully created."
}

# Mark the finish time of the script execution
$finishTime = Get-Date

# Output the time consumed in seconds
$TotalTime = ($finishTime - $startTime).TotalSeconds
Write-Output "The script completed in $TotalTime seconds."

The picture below shows the result of the execution of the script with the following parameters:

powershell .\CreateSubscription.ps1 -TopicPath stocks -Name ORCL -Namespace stockmarket -DefaultMessageTimeToLive 1 -SqlFilter ‘StockTicker=”ORCL”’ -SqlRuleAction ‘SET sys.Label=”Stocks”; SET Priority=”Medium”’ -UserMetadata ‘This subscription is used to process messages for the ORCL stock ticker’

CreateSubscription

You can download these scripts along with the PowerShell script to create event hubs on MSDN Code Gallery: Service Bus PowerShell Scripts.

Service Bus Explorer 2.5 now available!

$
0
0

I just released an improved version of the Service Bus Explorer tool based on the Microsoft.ServiceBus.dll 2.5.3.0.

The zip file contains:

  • The source code for the Service Bus Explorer 2.5.2.0. This version of the tool uses the Microsoft.ServiceBus.dll 2.5.3.0 that is compatible with the current version of the Windows Azure Service Bus, but not with the Service Bus 1.1, that is, the current version of the on-premises version of the Service Bus.
  • The Service Bus Explorer 2.1.3.0. This version can be used with the Service Bus 1.1. The Service Bus Explorer 2.1 uses the Microsoft.ServiceBus.dll client library which is compatible with the Service Bus for Windows Server 1.1 RTM version. You can download the source code of the Service Bus Explorer 2.1.3 from my OneDrive.
  • The Service Bus Explorer 1.8. This version can be used with the Service Bus 1.0

This version introduces the following updates:

  • Updated Microsoft.ServiceBus.dll to version 2.5.3.0.
  • Added support for the EnableExpress property of queues and topics:

  • The following bug has been fixed:
    1. Go to a topic with two or more enabled subscriptions. All the subscription icons show enabled state.
    2. Right click one subscription and click ‘Disable Subscription’, icon shows disabled state.
    3. Go to parent ‘Subscriptions’ node and ‘Refresh’: the disabled subscription shows an *enabled* icon even though it is disabled.
  • The logging mechanism in the PartitionListernerControl has been highly improved. Now logging stops as soon as the user clicks the Stop or Close button in the Consumer Group and Partition Listener dialog.

  • The code of the Consumer Group and Partition Listener has been dramatically improved:
  • The tool now uses the EventProcessorFactory<T> : IEventProcessorFactory class for creating instances of the EventProcessor : IEventProcessor class that are used to process events from event hub partitions.
  • The tool provides a custom checkpoint manager class: EventProcessorCheckpointManager : ICheckpointManager. In order to use the client-side checkpoint mechanism introduced by the tool, make sure to check the Checkpoint control in the Consumer Group and Partition Listener dialog as shown in the picture below.

  • The application uses the checkpoint manager to perform checkpoints in the in-process memory and persists checkpoints to a JSON file at the exit. At the start, the application reads checkpoints from the EventHubPartitionCheckpoints.json file located in the same directory of the tool. To remove all checkpoints, it’s sufficient to delete the file. You can also open the file and selectively delete checkpoints for specific event hub partitions.
[
  {
    "namespace": "eventhubs",
    "eventHub": "basicsampleeventhub",
    "leases": {
      "0": {
        "PartitionId": "0",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "244712",
        "SequenceNumber": 1699
      },
      "1": {
        "PartitionId": "1",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "127416",
        "SequenceNumber": 885
      },
      "7": {
        "PartitionId": "7",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "335176",
        "SequenceNumber": 2327
      },
      "2": {
        "PartitionId": "2",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "86152",
        "SequenceNumber": 599
      },
      "4": {
        "PartitionId": "4",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "238264",
        "SequenceNumber": 1654
      },
      "3": {
        "PartitionId": "3",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "-1",
        "SequenceNumber": 0
      },
      "5": {
        "PartitionId": "5",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "-1",
        "SequenceNumber": 0
      },
      "6": {
        "PartitionId": "6",
        "Owner": "ServiceBusExplorer",
        "Token": null,
        "Epoch": 0,
        "Offset": "-1",
        "SequenceNumber": 0
      }
    }
  }
]
  • This version of the tool introduces the possibility to send events directly to a specific event hub partition using the EventHubSender class. 
  • A bug affecting the dimension of the controls in the Messages and Deadletter Messages dialogs for queues and subscriptions has been fixed. To send messages to a specific event hub partition, right click a partition and choose Send Events as shown in the picture below.

  • When you send messages to a specific partition, the EventData.PartitionKey property is ignored. For this reason, the Update PartitionKey and No PartitionKey checkboxes are not available when sending messages to a given event hub partition.

  • When a queue or subscription is partitioned, now the tool invokes the MessageReceiver.Peek and MessageReceiver.Receive methods respectively, to peek and read messages one by one from the current queue, subscription or deadletter queue, instead of using the MessageReceiver.PeekBatch or MessageReceiver.ReceiveBatch methods. In fact, when the entity is partitioned, the batch methods read messages only from a single partition, hence they don’t return all the messages from the 16 partitions or fragments that form the entity.
  • Fxed the value shown by the Max Size in Gigabytes slider when the queue or topic is partitioned. In fact, when the queue or subscription is partitioned, the QueueDescription.MaxSizeInMegabytes and TopicDescription.MaxSizeInMegabytes property contains the size you specified when you created the entity (1024;2048;3072;4096;5120) multiplied by 16, that is, the number of fragments that compose the partitioned entity.
  • Fixed a problem that happened when updating the value of the ForwardTo and ForwardDeadLetteredMessagesTo properties of a subscription.
  • This version of the Service Bus Explorer introduces the possibility to save individual or multiple messages to a file in JSON format. See below for details.
  • In the Messages or Deadletter tab of a queue or subscription you can save a single message to a JSON file: right click the message to open the context menu and click Save Selected Message as shown in the picture below.

      This opens the Save File As dialog as shown in the picture below.

    The JSON file contains the body and properties of the selected BrokeredMessage object. Properties are in camel case.

    {"body": {"deviceid": 84,
        "value": 68},
      "contentType": null,
      "correlationId": null,
      "deliveryCount": 1,
      "enqueuedSequenceNumber": 1,
      "enqueuedTimeUtc": "2014-12-03T16:39:48.4958476Z",
      "expiresAtUtc": "9999-12-31T23:59:59.9999999",
      "forcePersistence": false,
      "isBodyConsumed": false,
      "isLargeMessage": false,
      "label": "Service Bus Explorer",
      "lockedUntilUtc": null,
      "lockToken": null,
      "messageId": "5cbc08aa-dca5-434b-bc27-afc8ba98307f",
      "partitionKey": "18",
      "properties": {"deviceId": 84,
        "value": 68,
        "time": 635532215878943345,
        "city": "Milan",
        "country": "Italy"},
      "replyTo": null,
      "replyToSessionId": null,
      "scheduledEnqueueTimeUtc": "0001-01-01T00:00:00",
      "sequenceNumber": 281474976710657,
      "sessionId": null,
      "size": 229,
      "state": 0,
      "timeToLive": "10675199.02:48:05.4775807",
      "to": null,
      "viaPartitionKey": null
    }
  • As an alternative, you can double click a message in the Messages or Deadletter tab for queue or subscription to open the Repair and Resubmit Message dialog and then click the Save button as shown in the picture below.

  • In the Messages or Deadletter tab of a queue or subscription you can save multiple messages to a JSON file: select multiple messages and right click one of them to open the context menu and then click Save Selected Messages as shown in the picture below.

      This opens the Save File As dialog as shown in the picture below.

    The JSON file contains an array of items. Each item contains the body and properties of one of the selected BrokeredMessage objects. Properties are in camel case.

    [
      {"body": {"deviceid": 36,
          "value": 93},
        "contentType": null,
        "correlationId": null,
        "deliveryCount": 1,
        "enqueuedSequenceNumber": 1,
        "enqueuedTimeUtc": "2014-12-03T16:39:48.7794889Z",
        "expiresAtUtc": "9999-12-31T23:59:59.9999999",
        "forcePersistence": false,
        "isBodyConsumed": false,
        "isLargeMessage": false,
        "label": "Service Bus Explorer",
        "lockedUntilUtc": null,
        "lockToken": null,
        "messageId": "197c5f6b-635c-437e-b419-5b346b626c35",
        "partitionKey": "0",
        "properties": {"deviceId": 36,
          "value": 93,
          "time": 635532215878943345,
          "city": "Milan",
          "country": "Italy"},
        "replyTo": null,
        "replyToSessionId": null,
        "scheduledEnqueueTimeUtc": "0001-01-01T00:00:00",
        "sequenceNumber": 4785074604081153,
        "sessionId": null,
        "size": 228,
        "state": 0,
        "timeToLive": "10675199.02:48:05.4775807",
        "to": null,
        "viaPartitionKey": null
      },
      {"body": {"deviceid": 85,
          "value": 29},
        "contentType": null,
        "correlationId": null,
        "deliveryCount": 1,
        "enqueuedSequenceNumber": 2,
        "enqueuedTimeUtc": "2014-12-03T16:39:56.5925297Z",
        "expiresAtUtc": "9999-12-31T23:59:59.9999999",
        "forcePersistence": false,
        "isBodyConsumed": false,
        "isLargeMessage": false,
        "label": "Service Bus Explorer",
        "lockedUntilUtc": null,
        "lockToken": null,
        "messageId": "fa9482b5-6dff-4268-9a5c-3d9b08bb358c",
        "partitionKey": "942",
        "properties": {"deviceId": 85,
          "value": 29,
          "time": 635532215962583503,
          "city": "Milan",
          "country": "Italy"},
        "replyTo": null,
        "replyToSessionId": null,
        "scheduledEnqueueTimeUtc": "0001-01-01T00:00:00",
        "sequenceNumber": 562949953421314,
        "sessionId": null,
        "size": 230,
        "state": 0,
        "timeToLive": "10675199.02:48:05.4775807",
        "to": null,
        "viaPartitionKey": null
      },
      {"body": {"deviceid": 84,
          "value": 68},
        "contentType": null,
        "correlationId": null,
        "deliveryCount": 1,
        "enqueuedSequenceNumber": 1,
        "enqueuedTimeUtc": "2014-12-03T16:39:48.4958476Z",
        "expiresAtUtc": "9999-12-31T23:59:59.9999999",
        "forcePersistence": false,
        "isBodyConsumed": false,
        "isLargeMessage": false,
        "label": "Service Bus Explorer",
        "lockedUntilUtc": null,
        "lockToken": null,
        "messageId": "5cbc08aa-dca5-434b-bc27-afc8ba98307f",
        "partitionKey": "18",
        "properties": {"deviceId": 84,
          "value": 68,
          "time": 635532215878943345,
          "city": "Milan",
          "country": "Italy"},
        "replyTo": null,
        "replyToSessionId": null,
        "scheduledEnqueueTimeUtc": "0001-01-01T00:00:00",
        "sequenceNumber": 281474976710657,
        "sessionId": null,
        "size": 229,
        "state": 0,
        "timeToLive": "10675199.02:48:05.4775807",
        "to": null,
        "viaPartitionKey": null
      }
    ]
  • In the Consumer Group or Partition Listener dialog you can save a single event to a JSON file: right click the event under the Events tab to open the context menu and click Save Selected Event as shown in the picture below.

      This opens the Save File As dialog as shown in the picture below.

    The JSON file contains the body and properties of the selected EventData object. Properties are in camel case.

    {
      "body": {
        "deviceid": 6,
        "name": "device006",
        "value": 50,
        "location": "Milan"
      },
      "enqueuedTimeUtc": "2014-11-06T17:18:01.057Z",
      "offset": "208",
      "partitionKey": "device006",
      "properties": {
        "id": 6,
        "name": "device006",
        "location": "Milan",
        "value": 50
      },
      "sequenceNumber": 1,
      "systemProperties": {
        "Publisher": "device006",
        "PartitionKey": "device006",
        "EnqueuedTimeUtc": "2014-11-06T17:18:01.057Z",
        "SequenceNumber": 1,
        "Offset": "208"
      }
    } 
  • As an alternative, you can double click an event in the Consumer Group or Partition Listener dialog to open the View Event Data dialog and then click the Save button as shown in the picture below.

  • In the Consumer Group or Partition Listener dialog you can save multiple events to a JSON file: select multiple events and right click one of them to open the context menu and then click Save Selected Events as shown in the picture below. 

      This opens the Save File As dialog as shown in the picture below.

    The JSON file contains an array of items. Each item contains the body and properties of one of the selected EventData objects. Properties are in camel case. 

    [
      {
        "body": {
          "deviceid": 6,
          "name": "device006",
          "value": 26,
          "location": "Milan"
        },
        "enqueuedTimeUtc": "2014-11-06T17:18:03.267Z",
        "offset": "624",
        "partitionKey": "device006",
        "properties": {
          "id": 6,
          "name": "device006",
          "location": "Milan",
          "value": 26
        },
        "sequenceNumber": 3,
        "systemProperties": {
          "Publisher": "device006",
          "PartitionKey": "device006",
          "EnqueuedTimeUtc": "2014-11-06T17:18:03.267Z",
          "SequenceNumber": 3,
          "Offset": "624"
        }
      },
      {
        "body": {
          "deviceid": 6,
          "name": "device006",
          "value": 23,
          "location": "Milan"
        },
        "enqueuedTimeUtc": "2014-11-06T17:18:02.173Z",
        "offset": "416",
        "partitionKey": "device006",
        "properties": {
          "id": 6,
          "name": "device006",
          "location": "Milan",
          "value": 23
        },
        "sequenceNumber": 2,
        "systemProperties": {
          "Publisher": "device006",
          "PartitionKey": "device006",
          "EnqueuedTimeUtc": "2014-11-06T17:18:02.173Z",
          "SequenceNumber": 2,
          "Offset": "416"
        }
      },
      {
        "body": {
          "deviceid": 6,
          "name": "device006",
          "value": 50,
          "location": "Milan"
        },
        "enqueuedTimeUtc": "2014-11-06T17:18:01.057Z",
        "offset": "208",
        "partitionKey": "device006",
        "properties": {
          "id": 6,
          "name": "device006",
          "location": "Milan",
          "value": 50
        },
        "sequenceNumber": 1,
        "systemProperties": {
          "Publisher": "device006",
          "PartitionKey": "device006",
          "EnqueuedTimeUtc": "2014-11-06T17:18:01.057Z",
          "SequenceNumber": 1,
          "Offset": "208"
        }
      }
    ] 

Service Bus Explorer 2.6 now available!

$
0
0

I just released an improved version of the Service Bus Explorer tool based on the Microsoft.ServiceBus.dll 2.6.1.0. You can find the code here:

The zip file contains:

  • The source code for the Service Bus Explorer 2.6.1.0. This version of the tool uses the Microsoft.ServiceBus.dll 2.5.3.0 that is compatible with the current version of the Windows Azure Service Bus, but not with the Service Bus 1.1, that is, the current version of the on-premises version of the Service Bus.
  • The Service Bus Explorer 2.1.3.0. This version can be used with the Service Bus 1.1. The Service Bus Explorer 2.1 uses the Microsoft.ServiceBus.dll client library which is compatible with the Service Bus for Windows Server 1.1 RTM version. You can download the source code of the Service Bus Explorer 2.1.3 from my OneDrive.

This version introduces the following updates:

  • The tool now uses the Microsoft.ServiceBus.dll v.2.6.1.0.
  • Completely refreshed support for dynamic relay services and added full support for persistent relay services. For more information on persistent relay services, see How to handle Service Bus Relay Services in a multi-tenant environment.
  • You can select dynamic and persistent relay services in the main treeview and view their properties in the main panel.

  • You can create, delete, update persistent relay services. In particular, you can define the relay type or binding, the transport security and client authorization characteristics of the persistent relay service in the Description tab of the HandleRelayControl.

Relay service definition

  • You can create, review, update, delete the authorization rules alias shared access policies at the entity level for persistent relay services n the Authorization Rules tab of the HandleRelayControl.

Relay service authorization rules

  • You can query the metrics of both persistent and dynamic relay services in the Metrics tab of the HandleRelayControl. See point 3 in the picture below. For more information on this subject, see Service Bus Entity Metrics REST APIs.

Metric rule definition


Metrics data and charts

  • You can test both dynamic and persistent relay services in SDI and MDI mode.

  • Added support to import/export persistent relay services from/to an XML file.
  • When the saveMessageToFile setting in the configuration file is set to true, the message content of the Test Relay form is saved to file on exit.

HandlePartitionControl

Consumer Group / Partition Listener

  • The Consumer Group / Partition Listener control added the possibility to start receiving events from a specific point in time by defining a value for the EventHubReceiver.StartingDateTimeUtc property. Note: you have to specify date and time in UTC format, not in the local date and time format.

Consumer Group / Partition Listener: Listener Tab

Consumer Group / Partition Listener: EventsTab

Consumer Group / Partition Listener

  • Fixed visualization of event data properties in the Consumer Group / Partition Listener control (PartitionListenerControl).
  • Greatly improved message tracking in the Consumer Group / Partition Listener control (PartitionListenerControl).
  • Fixed and extended Clear funtionality in the Consumer Group / Partition Listener control (PartitionListenerControl).
  • Added the All item to Metrics. When All is selected, the tool will retrieve all the metrics for the selected entity. See point 1 in the picture below.
  • Added the possibility to delete a single metric query by pressing the delete button at the end of the row. See point 2 in the picture below.

  • No chart is shown if a metric doesn’t return any data.
  • When no time range is explicitly specified in a metric rule, the tool retrieves metric data of the last 7 days.
  • Added Metrics support for the Event Hubs, Consumer Groups, Notification Hubs and Relays.

Event Hub: metric rule definition

Event Hub: metric data and charts


Consumer Group: metric rule definition

Consumer Group:  metric data and charts

Notification Hub: metric rule definition

Notification Hub: metric data and charts

Relay: metric rule definition

 

Relay: metric data and charts

  • If you right click the namespace node in treeview and select Open Metrics in SDI or MDI mode, you can access a dialog where you can select metrics of different entities. For example, this option allows to compare the throughput of an event hub with the throughput of one of its consumer groups.

Namespace: metric rule definition

Namespace: metric data and charts

  • Bug fixed by the developer community on GitHub (thanks guys!):

How to store Event Hub events to Azure SQL Database

$
0
0

Introduction

This sample shows how to use the EventProcessorHost to retrieve events from an Event Hub and store them in a batch mode to an Azure SQL Database. The solution demonstrates how the use the following techniques:

  • Send events to an Event Hub using both AMQP and HTTPS transport protocols.
  • Create an entity level shared access policy with only the Send claim. This key will be used to create SAS tokens, one for each publisher endpoint. 
  • Issue a SAS token to secure individual publisher endpoints.
  • Use a SAS token to authenticate at a publisher endpoint level.
  • Use the EventProcessorHost to retrieve and process events from an event hub.
  • Perform structured and sematic logging using a custom EventSource class and ETW Logs introduced by Azure SDK 2.5.
  • Use a stored procedure with a Table-Valued Parameter to store multiple events in a batch mode to a table on an Azure SQL database.

You can download the code from MSDN Code Gallery.

Scenario

This solution simulates an Internet of Things (IoT) scenario where thousands of devices send events (e.g. sensor readings) to a backend system via a message broker. The backend system retrieves events from the messaging infrastrcure and store them to a persistent repository in a scalable manner.

Architecture

The sample is structured as follows:

  • A Windows Forms application can be used to create an event hub and an entity level shared access policy with only the Send access right. 
  • The same application can be used to simulate a configurable amount of devices that send readings into the event hub. Each device uses a separate publisher endpoint to send data to the underlying event hub and a separate SAS token to authenticate with the Service Bus namespace.
  • An event hub is used to ingest device events.
  • A worker role with multiple instances uses an EventProcessorHost to read and process messages from the partitions of the event hub.
  • The custom EventProcessor class uses a Table-Valued Parameter to insert events into a table of a SQL Database in a batch mode by invoking a stored procedure.
  • The worker role uses a custom EventSourceclass and the ETW log support introduced by the Azure SDK 2.5 to write log data to table storage.
  • The stored procedure uses the MERGE statement to implement an UPSERT mechanism.

The following picture shows the architecture of the solution:

 

References

Event Hubs

Table-Valued Parameters

ETW Logs

Visual Studio Solution

The Visual Studio solution includes the following projects:
 
  • CreateIoTDbWithMerge.sql: this script can be used to create the SQL Server database used to store device events.
  • Entities: this library contains the Payload class. This class defines the structure and content of the EventData message body.
  • EventProcessorHostWorkerRole: this library defines the worker role used to handle the events from the event hub.
  • Helpers: this library defines the TraceEventSource class used by the worker role to create ETW logs at runtime.
  • Sender: this Windows Forms application can be used to create the Event Hub used by the sample and simulate a configurable amount of devices sending events to the event hub.
  • StoreEventsToAzureSqlDatabase: this Azure project defines the cloud service hosting the worker role used to handle the events from the event hub.

NOTE: To reduce the size of tha zip file, I deleted the NuGet packages. To repair the solution, make sure to right click the solution and select Enable NuGet Package Restore as shown in the picture below. For more information on this topic, see the following post.

Solution

This section briefly describes the individual components of the solution.

SQL Azure Database

Run the CreateIoTDbWithMerge.sql script to create the database used by the solution. In particular, the script create the following artifacts:

  • The EventTableType used-defined table type.
  • The Events table used to store events.
  • The sp_InsertEvents stored procedure used to store events. The stored procedure receives a single EventTableType parameter and uses the MERGE statement to implement an UPSERT mechanism. This mechanism is used to implement idempotency: if an row already exists in the table with the a given EventId, the store procedure updates its columns, otherwise a new record is created.
-- Drop sp_InsertEvents stored procedure 
IF OBJECT_ID('sp_InsertEvents') > 0 DROP PROCEDURE [dbo].[sp_InsertEvents]
GO

-- Drop Events table 
IF OBJECT_ID('Events') > 0 DROP TABLE [dbo].[Events]
GO

-- Drop EventTableType used-defined table type 
DROP TYPE [dbo].[EventTableType]
GO

-- Create EventTableType used-defined table type 
CREATE TYPE [dbo].[EventTableType] AS TABLE(
    [EventId] [int] NOT NULL,
    [DeviceId] [int] NULL,
    [Value] [int] NULL,
    [Timestamp] [datetime2](7) NULL
)
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

-- Create Events table 
CREATE TABLE [dbo].[Events](
    [EventId] [int] NOT NULL,
    [DeviceId] [int] NOT NULL,
    [Value] [int] NOT NULL,
    [Timestamp] [datetime2](7) NULL,
PRIMARY KEY CLUSTERED
(
    [EventId] ASC
)WITH (PAD_INDEX = OFF,
       STATISTICS_NORECOMPUTE = OFF,
       IGNORE_DUP_KEY = OFF,
       ALLOW_ROW_LOCKS = ON,
       ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

-- Create sp_InsertEvents stored procedure 
CREATE PROCEDURE [dbo].[sp_InsertEvents]
    @Events dbo.EventTableType READONLY
AS
BEGIN
    MERGE INTO dbo.[Events] AS A
    USING (
        SELECT * FROM @Events
    ) B ON (A.EventId = B.EventId)
    WHEN MATCHED THEN
        UPDATE SET A.DeviceId = B.DeviceId,
                   A.Value = B.Value,
                   A.Timestamp = B.Timestamp
    WHEN NOT MATCHED THEN
        INSERT ([EventId], [DeviceId], [Value], [Timestamp])
        VALUES(B.[EventId], B.[DeviceId], B.[Value], B.[Timestamp]);
END
GO

Entities

The following table contains the code of the Payload class. This class is used to define the body of the messages sent to the event hub. Note that the properties of the class are decorated with the JsonPropertyAttribute. In fact, both the client application and worker role use Json.Net to serialize and deserialize the message content in JSON.
#region Using Directives 
using System;
using Newtonsoft.Json;
#endregion 

namespace Microsoft.AzureCat.Samples.Entities
{
    [Serializable]
    public class Payload
    {
        /// <summary>
        /// Gets or sets the device id.
        /// </summary> 
        [JsonProperty(PropertyName = "eventId", Order = 1)]
        public int EventId { get; set; }

        /// <summary>
        /// Gets or sets the device id.
        /// </summary> 
        [JsonProperty(PropertyName = "deviceId", Order = 2)]
        public int DeviceId { get; set; }

        /// <summary>
        /// Gets or sets the device value.
        /// </summary> 
        [JsonProperty(PropertyName = "value", Order = 3)]
        public int Value { get; set; }

        /// <summary>
        /// Gets or sets the event timestamp.
        /// </summary> 
        [JsonProperty(PropertyName = "timestamp", Order = 4)]
        public DateTime Timestamp { get; set; }
    }
}

Helpers

This library defines the TraceEventSource class used by the worker role to trace events to ETW logs. The WAD agent running on the worker role instances will read data out of local ETW logs and persist this data to a couple of storage tables (WASDiagnosticTable and WADEventProcessorTable) in the storage account configured for Windows Azure Diagnostics.
#region Using Directives
using System;
using System.Diagnostics.Tracing;
using System.Runtime.CompilerServices;
using Microsoft.WindowsAzure.ServiceRuntime;

#endregion

namespace Microsoft.AzureCat.Samples.Helpers
{
    [EventSource(Name = "TraceEventSource")]
    public sealed class TraceEventSource : EventSource
    {
        #region Internal Enums
        public class Keywords
        {
            public const EventKeywords EventHub = (EventKeywords)1;
            public const EventKeywords DataBase = (EventKeywords)2;
            public const EventKeywords Diagnostic = (EventKeywords)4;
            public const EventKeywords Performance = (EventKeywords)8;
        }
        #endregion

        #region Public Static Properties
        public static readonly TraceEventSource Log = new TraceEventSource();
        #endregion

        #region Private Methods
        [Event(1,
       Message = "TraceIn",
       Keywords = Keywords.Diagnostic,
       Level = EventLevel.Verbose)]
        private void TraceIn(string application,
                             string instance,
                             Guid activityId,
                             string description,
                             string source,
                             string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance))
            {
                return;
            }
            WriteEvent(1, application, instance, activityId, description, source, method);
        }

        [Event(2,
               Message = "TraceOut",
               Keywords = Keywords.Diagnostic,
               Level = EventLevel.Verbose)]
        private void TraceOut(string application,
                              string instance,
                              Guid activityId,
                              string description,
                              string source,
                              string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance))
            {
                return;
            }
            WriteEvent(2, application, instance, activityId, description, source, method);
        }

        [Event(3,
               Message = "TraceApi",
               Keywords = Keywords.Diagnostic,
               Level = EventLevel.Informational)]
        private void TraceExec(string application,
                               string instance,
                               Guid activityId,
                               double elapsed,
                               string description,
                               string source,
                               string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(description))
            {
                return;
            }
            WriteEvent(3, application, instance, activityId, elapsed, description, source, method);
        }

        [Event(4,
               Message = "TraceInfo",
               Keywords = Keywords.Diagnostic,
               Level = EventLevel.Informational)]
        private void TraceInfo(string application,
                               string instance,
                               string description,
                               string source,
                               string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(description))
            {
                return;
            }
            WriteEvent(4, application, instance, description, source, method);
        }

        [Event(5,
               Message = "TraceError",
               Keywords = Keywords.Diagnostic,
               Level = EventLevel.Error)]
        private void TraceError(string application,
                                string instance,
                                Guid activityId,
                                string exception,
                                string innerException,
                                string source,
                                string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(exception))
            {
                return;
            }
            WriteEvent(5, application, instance, activityId, exception, string.IsNullOrWhiteSpace(innerException) ? string.Empty : innerException, source, method);
        }

        [Event(6,
               Message = "OpenPartition",
               Keywords = Keywords.EventHub,
               Level = EventLevel.Informational)]
        private void OpenPartition(string application,
                                   string instance,
                                   string eventHub,
                                   string consumerGroup,
                                   string partitionId,
                                   string source,
                                   string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            WriteEvent(6, application, instance, eventHub, consumerGroup, partitionId, source, method);
        }

        [Event(7,
               Message = "ClosePartition",
               Keywords = Keywords.EventHub,
               Level = EventLevel.Informational)]
        private void ClosePartition(string application,
                                    string instance,
                                    string eventHub,
                                    string consumerGroup,
                                    string partitionId,
                                    string reason,
                                    string source,
                                    string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            WriteEvent(7, application, instance, eventHub, consumerGroup, partitionId, reason, source, method);
        }

        [Event(8,
               Message = "ProcessEvents",
               Keywords = Keywords.EventHub,
               Level = EventLevel.Informational)]
        private void ProcessEvents(string application,
                                   string instance,
                                   string eventHub,
                                   string consumerGroup,
                                   string partitionId,
                                   int messageCount,
                                   string source,
                                   string method)
        {
            if (string.IsNullOrWhiteSpace(application) ||
                string.IsNullOrWhiteSpace(instance) ||
                string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            if (IsEnabled())
            {
                WriteEvent(8, application, instance, eventHub, consumerGroup, partitionId, messageCount, source, method);
            }
        }
        #endregion

        #region Public Methods
        [NonEvent]
        public void TraceApi(TimeSpan elapsed,
                             string description,
                             [CallerFilePath] string source = "",
                             [CallerMemberName] string method = "")
        {
            if (IsEnabled())
            {
                TraceExec(RoleEnvironment.CurrentRoleInstance.Role.Name,
                          RoleEnvironment.CurrentRoleInstance.Id,
                          Guid.NewGuid(),
                          elapsed.TotalMilliseconds,
                          description,
                          source,
                          method);
            }
        }

        [NonEvent]
        public void TraceApi(Guid activityId,
                             TimeSpan elapsed,
                             string description,
                             [CallerFilePath] string source = "",
                             [CallerMemberName] string method = "")
        {
            if (IsEnabled())
            {
                TraceExec(RoleEnvironment.CurrentRoleInstance.Role.Name,
                          RoleEnvironment.CurrentRoleInstance.Id,
                          activityId,
                          elapsed.TotalMilliseconds,
                          description,
                          source,
                          method);
            }
        }

        [NonEvent]
        public void TraceIn([CallerFilePath] string source = "",
                            [CallerMemberName] string method = "")
        {
            if (IsEnabled())
            {
                TraceIn(RoleEnvironment.CurrentRoleInstance.Role.Name,
                        RoleEnvironment.CurrentRoleInstance.Id,
                        Guid.NewGuid(),
                        string.Empty,
                        source,
                        method);
            }
        }

        [NonEvent]
        public void TraceOut([CallerFilePath] string source = "",
                             [CallerMemberName] string method = "")
        {
            if (IsEnabled())
            {
                TraceOut(RoleEnvironment.CurrentRoleInstance.Role.Name,
                         RoleEnvironment.CurrentRoleInstance.Id,
                         Guid.NewGuid(),
                         string.Empty,
                         source,
                         method);
            }
        }

        [NonEvent]
        public void TraceInfo(string description,
                              [CallerFilePath] string source = "",
                              [CallerMemberName] string method = "")
        {
            if (IsEnabled())
            {
                TraceInfo(RoleEnvironment.CurrentRoleInstance.Role.Name,
                          RoleEnvironment.CurrentRoleInstance.Id,
                          description,
                          source,
                          method);
            }
        }

        [NonEvent]
        public void TraceError(string exception,
                               string innerException,
                               [CallerFilePath] string source = "",
                               [CallerMemberName] string method = "")
        {
            if (string.IsNullOrWhiteSpace(exception))
            {
                return;
            }
            if (IsEnabled())
            {
                TraceError(RoleEnvironment.CurrentRoleInstance.Role.Name,
                           RoleEnvironment.CurrentRoleInstance.Id,
                           Guid.NewGuid(),
                           exception,
                           string.IsNullOrWhiteSpace(innerException) ? string.Empty : innerException,
                           source,
                           method);
            }
        }

       [NonEvent]
        public void OpenPartition(string eventHub,
                                  string consumerGroup,
                                  string partitionId,
                                  [CallerFilePath] string source = "",
                                  [CallerMemberName] string method = "")
        {
            if (string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            if (IsEnabled())
            {
                OpenPartition(RoleEnvironment.CurrentRoleInstance.Role.Name,
                              RoleEnvironment.CurrentRoleInstance.Id,
                              eventHub,
                              consumerGroup,
                              partitionId,
                              source,
                              method);
            }
        }

        [NonEvent]
       public void ClosePartition(string eventHub,
                                  string consumerGroup,
                                  string partitionId,
                                  string reason,
                                  [CallerFilePath] string source = "",
                                  [CallerMemberName] string method = "")
        {
            if (string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            if (IsEnabled())
            {
                ClosePartition(RoleEnvironment.CurrentRoleInstance.Role.Name,
                               RoleEnvironment.CurrentRoleInstance.Id,
                               eventHub,
                               consumerGroup,
                               partitionId,
                               reason,
                               source,
                               method);
            }
        }

        [NonEvent]
        public void ProcessEvents(string eventHub,
                                  string consumerGroup,
                                  string partitionId,
                                  int messageCount,
                                  [CallerFilePath] string source = "",
                                  [CallerMemberName] string method = "")
        {
            if (string.IsNullOrWhiteSpace(eventHub) ||
                string.IsNullOrWhiteSpace(consumerGroup) ||
                string.IsNullOrWhiteSpace(partitionId))
            {
                return;
            }
            if (IsEnabled())
            {
                ProcessEvents(RoleEnvironment.CurrentRoleInstance.Role.Name,
                              RoleEnvironment.CurrentRoleInstance.Id,
                              eventHub,
                              consumerGroup,
                              partitionId,
                              messageCount,
                              source,
                              method);
            }
        }

        #endregion
    }
}
EventProcessorHostWorkerRole
The following table contains the code of the WorkerRole class. The class reads the following settings from the service configuration file and then create an instance of the EventProcessorHost class:

 

  • Connectionstring of the Azure SQL Database where to store events.
  • Connectionstring of the Service Bus namespace hsoting the event hub used by the solution.
  • Connectionstring of the storage account used by the EventProcessorHost.
  • The name of the event hub.
  • The name of the consumer group used by the worker role to read events from the event hub.
#region Using Directives
using System;
using System.Linq;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.ServiceBus.Messaging;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.AzureCat.Samples.Helpers;
#endregion

namespace Microsoft.AzureCat.Samples.EventProcessorHostWorkerRole
{
    public class WorkerRole : RoleEntryPoint
    {
        #region Private Constants
        //*******************************
        // Messages & Formats
        //*******************************
        private const string RoleEnvironmentSettingFormat = "Configuration Setting [{0}] = [{1}].";
        private const string RoleEnvironmentConfigurationSettingChangedFormat = "The setting [{0}] is changed: new value = [{1}].";
        private const string RoleEnvironmentConfigurationSettingChangingFormat = "The setting [{0}] is changing: old value = [{1}].";
        private const string RoleEnvironmentTopologyChangedFormat = "The  topology for the [{0}] role is changed.";
        private const string RoleEnvironmentTopologyChangingFormat = "The  topology for the [{0}] role is changing.";
        private const string RoleInstanceCountFormat = "[Role {0}] instance count = [{1}].";
        private const string RoleInstanceEndpointCountFormat = "[Role {0}] instance endpoints count = [{1}].";
        private const string RoleInstanceEndpointFormat = "[Role {0}] instance endpoint [{1}]: protocol = [{2}] address = [{3}] port = [{4}].";
        private const string RoleInstanceIdFormat = "[Role {0}] instance Id = [{1}].";
        private const string RoleInstanceStatusFormat = "[Role {0}] instance Id = [{1}] Status = [{2}].";
        private const string Unknown = "Unknown";
        private const string RegisteringEventProcessor = "Registering Event Processor [EventProcessor]... ";
        private const string EventProcessorRegistered = "Event Processor [EventProcessor] successfully registered. ";

        //*******************************
        // Settings
        //*******************************
        private const string SqlDatabaseConnectionStringSetting = "SqlDatabaseConnectionString";
        private const string StorageAccountConnectionStringSetting = "StorageAccountConnectionString";
        private const string ServiceBusConnectionStringSetting = "ServiceBusConnectionString";
        private const string EventHubNameSetting = "EventHubName";
        private const string ConsumerGroupNameSetting = "ConsumerGroupName";
        #endregion

        #region Private Fields
        private string eventHubName;
        private string consumerGroupName;
        private string sqlDatabaseConnectionString;
        private string storageAccountConnectionString;
        private string serviceBusConnectionString;
        private EventProcessorHost eventProcessorHost;
        #endregion

        #region Public Methods
        public override void Run()
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                while (true)
                {
                    Thread.Sleep(10000);
                }
                // ReSharper disable once FunctionNeverReturns
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        public override bool OnStart()
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Set Default values for the ServicePointManager
                ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 120;
                ServicePointManager.UseNagleAlgorithm = false;
                ServicePointManager.Expect100Continue = false;

                // Setting RoleEnvironment event handlers
                RoleEnvironment.Changed += RoleEnvironment_Changed;
                RoleEnvironment.Changing += RoleEnvironment_Changing;
                RoleEnvironment.StatusCheck += RoleEnvironment_StatusCheck;
                RoleEnvironment.Stopping += RoleEnvironment_Stopping;

                // Read Configuration Settings
                ReadConfigurationSettings();

                // Start Event Processor
                StartEventProcessorAsync().Wait();

                // Run base.OnStart method
                return base.OnStart();
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
                return false;
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }
        #endregion

        #region Private Methods

        private void ReadConfigurationSettings()
        {
            // Read sql database connectionstring setting  
            sqlDatabaseConnectionString = CloudConfigurationHelper.GetSetting(SqlDatabaseConnectionStringSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                        SqlDatabaseConnectionStringSetting,
                                                        sqlDatabaseConnectionString));

            // Read storage account connectionstring setting  
            storageAccountConnectionString = CloudConfigurationHelper.GetSetting(StorageAccountConnectionStringSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                        StorageAccountConnectionStringSetting,
                                                        storageAccountConnectionString));

            // Read service bus connectionstring setting  
            serviceBusConnectionString = CloudConfigurationHelper.GetSetting(ServiceBusConnectionStringSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                        ServiceBusConnectionStringSetting,
                                                        serviceBusConnectionString));

            // Read event hub name setting
            eventHubName = CloudConfigurationHelper.GetSetting(EventHubNameSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                        ServiceBusConnectionStringSetting,
                                                        serviceBusConnectionString));

            // Read event consumer group name setting
            consumerGroupName = CloudConfigurationHelper.GetSetting(ConsumerGroupNameSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                        ServiceBusConnectionStringSetting,
                                                        serviceBusConnectionString));
        }

        private async Task StartEventProcessorAsync()
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();
                var eventHubClient = EventHubClient.CreateFromConnectionString(serviceBusConnectionString, eventHubName);

                // Get the default Consumer Group
                eventProcessorHost = new EventProcessorHost(RoleEnvironment.CurrentRoleInstance.Id,
                                                            eventHubClient.Path.ToLower(),
                                                            consumerGroupName.ToLower(),
                                                            serviceBusConnectionString,
                                                            storageAccountConnectionString)
                {
                    PartitionManagerOptions = new PartitionManagerOptions
                    {
                        AcquireInterval = TimeSpan.FromSeconds(10), // Default is 10 seconds
                        RenewInterval = TimeSpan.FromSeconds(10), // Default is 10 seconds
                        LeaseInterval = TimeSpan.FromSeconds(30) // Default value is 30 seconds
                    }
                };
                TraceEventSource.Log.TraceInfo(RegisteringEventProcessor);
                var eventProcessorOptions = new EventProcessorOptions
                {
                    InvokeProcessorAfterReceiveTimeout = true,
                    MaxBatchSize = 100,
                    PrefetchCount = 100,
                    ReceiveTimeOut = TimeSpan.FromSeconds(30),
                };
                eventProcessorOptions.ExceptionReceived += eventProcessorOptions_ExceptionReceived;
                await eventProcessorHost.RegisterEventProcessorFactoryAsync(new EventProcessorFactory<EventProcessor>(sqlDatabaseConnectionString),
                                                                            eventProcessorOptions);
                TraceEventSource.Log.TraceInfo(EventProcessorRegistered);
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        void eventProcessorOptions_ExceptionReceived(object sender, ExceptionReceivedEventArgs e)
        {
            if (e == null || e.Exception == null)
            {
                return;
            }

            // Trace Exception
            TraceEventSource.Log.TraceError(e.Exception.Message,
                                            e.Exception.InnerException != null ?
                                            e.Exception.InnerException.Message :
                                            string.Empty);
        }

        /// <summary>
        /// Occurs after a change to the service configuration is applied to the running instances of a role.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">Represents the arguments for the Changed event, which occurs after a configuration change has been applied to a role instance.</param>
        private static void RoleEnvironment_Changed(object sender, RoleEnvironmentChangedEventArgs e)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Get the list of configuration setting changes
                var settingChanges = e.Changes.OfType<RoleEnvironmentConfigurationSettingChange>();

                foreach (var settingChange in settingChanges)
                {
                    var value = CloudConfigurationHelper.GetSetting(settingChange.ConfigurationSettingName);
                    TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentConfigurationSettingChangedFormat,
                                                                            settingChange.ConfigurationSettingName ?? string.Empty,
                                                                            string.IsNullOrEmpty(value) ? string.Empty : value));
                }

                // Get the list of configuration changes
                var topologyChanges = e.Changes.OfType<RoleEnvironmentTopologyChange>();

                foreach (var roleName in topologyChanges.Select(topologyChange => topologyChange.RoleName))
                {
                    TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentTopologyChangedFormat,string.IsNullOrEmpty(roleName) ? Unknown : roleName));
                    if (string.IsNullOrEmpty(roleName))
                    {
                        continue;
                    }
                    var role = RoleEnvironment.Roles[roleName];
                    if (role == null)
                    {
                        continue;
                    }
                    TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceCountFormat, roleName, role.Instances.Count));
                    foreach (var roleInstance in role.Instances)
                    {
                        TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceIdFormat, roleName, roleInstance.Id));
                        TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceEndpointCountFormat, roleName, roleInstance.InstanceEndpoints.Count));
                        foreach (var instanceEndpoint in roleInstance.InstanceEndpoints)
                        {
                            TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceEndpointFormat,
                                                                            roleName,
                                                                            instanceEndpoint.Key,
                                                                            instanceEndpoint.Value.Protocol,
                                                                            instanceEndpoint.Value.IPEndpoint.Address,
                                                                            instanceEndpoint.Value.IPEndpoint.Port));
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        /// <summary>
        /// Occurs before a change to the service configuration is applied to the running instances of a role.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">presents the arguments for the Changing event, which occurs before a configuration change is applied to a role instance. </param>
        private static void RoleEnvironment_Changing(object sender, RoleEnvironmentChangingEventArgs e)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Get the list of configuration setting changes
                var settingChanges = e.Changes.OfType<RoleEnvironmentConfigurationSettingChange>();

                foreach (var settingChange in settingChanges)
                {
                    var value = CloudConfigurationHelper.GetSetting(settingChange.ConfigurationSettingName);
                    TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentConfigurationSettingChangingFormat,
                                                                    settingChange.ConfigurationSettingName,
                                                                    string.IsNullOrEmpty(value) ? string.Empty : value));
                }

                // Get the list of configuration changes
                var topologyChanges = e.Changes.OfType<RoleEnvironmentTopologyChange>();

                foreach (var roleName in topologyChanges.Select(topologyChange => topologyChange.RoleName))
                {
                    TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentTopologyChangingFormat,
                                                                    string.IsNullOrEmpty(roleName) ? Unknown : roleName));
                    if (string.IsNullOrEmpty(roleName))
                    {
                        continue;
                    }
                    var role = RoleEnvironment.Roles[roleName];
                    if (role == null)
                    {
                        continue;
                    }
                    TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceCountFormat, roleName, role.Instances.Count));
                    foreach (var roleInstance in role.Instances)
                    {
                        TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceIdFormat, roleName, roleInstance.Id));
                        TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceEndpointCountFormat, roleName, roleInstance.InstanceEndpoints.Count));
                        foreach (var instanceEndpoint in roleInstance.InstanceEndpoints)
                        {
                            TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceEndpointFormat,
                                                                            roleName,
                                                                            instanceEndpoint.Key,
                                                                            instanceEndpoint.Value.Protocol,
                                                                            instanceEndpoint.Value.IPEndpoint.Address,
                                                                            instanceEndpoint.Value.IPEndpoint.Port));
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        /// <summary>
        /// Occurs at a regular interval to indicate the status of a role instance.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">Represents the arguments for the StatusCheck event, which occurs at a regular interval to indicate the status of a role instance.</param>
        private static void RoleEnvironment_StatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Write Role Instance Status
                TraceEventSource.Log.TraceInfo(string.Format(RoleInstanceStatusFormat,
                                                                RoleEnvironment.CurrentRoleInstance.Role.Name,
                                                                RoleEnvironment.CurrentRoleInstance.Id,
                                                                e.Status));
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        /// <summary>
        /// Occurs when a role instance is about to be stopped.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">Represents the arguments for the Stopping event, which occurs when a role instance is being stopped. </param>
        private static void RoleEnvironment_Stopping(object sender, RoleEnvironmentStoppingEventArgs e)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                   ex.InnerException != null ?
                                                   ex.InnerException.Message :
                                                   string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }
        #endregion
    }
}
The following table contains the code of the EventProcessor class. In particular, the ProcessEventsAsync method writes events to the Azure SQL Database in a batch mode by invoking the sp_InsertEvents stored procedure with a DataTable object as input parameter.
#region Using Directives
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.ServiceBus.Messaging;
using Newtonsoft.Json;
using Microsoft.AzureCat.Samples.Entities;
using Microsoft.AzureCat.Samples.Helpers;
#endregion

namespace Microsoft.AzureCat.Samples.EventProcessorHostWorkerRole
{
    public class EventProcessor : IEventProcessor
    {
        #region Private Constants
        //*******************************
        // Messages & Formats
        //*******************************
        private const string RoleEnvironmentSettingFormat = "Configuration Setting [{0}] = [{1}].";

        //*******************************
        // Settings
        //*******************************
        private const string SqlDatabaseConnectionStringSetting = "SqlDatabaseConnectionString";

        //*******************************
        // Columns & Commands & Parameters
        //*******************************
        private const string EventId = "EventId";
        private const string DeviceId = "DeviceId";
        private const string Value = "Value";
        private const string Timestamp = "Timestamp";
        private const string InsertEventsStoreProcedure = "sp_InsertEvents";
        private const string EventsParameter = "@Events";
        private const string TableType = "EventTableType";
        #endregion

        #region Private Fields
        private readonly string sqlDatabaseConnectionString;
        #endregion

        #region Public Constructors

        public EventProcessor()
        {
            // Read sql database connectionstring setting  
            sqlDatabaseConnectionString = CloudConfigurationHelper.GetSetting(SqlDatabaseConnectionStringSetting);
            TraceEventSource.Log.TraceInfo(string.Format(RoleEnvironmentSettingFormat,
                                                         SqlDatabaseConnectionStringSetting,
                                                         sqlDatabaseConnectionString));
        }

        public EventProcessor(string sqlDatabaseConnectionString)
        {
            this.sqlDatabaseConnectionString = sqlDatabaseConnectionString;
        }
        #endregion

        #region IEventProcessor Methods
        public Task OpenAsync(PartitionContext context)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Trace Open Partition
                TraceEventSource.Log.OpenPartition(context.EventHubPath,
                                                   context.ConsumerGroupName,
                                                   context.Lease.PartitionId);
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                ex.InnerException != null ?
                                                ex.InnerException.Message :
                                                string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
            return Task.FromResult<object>(null);
        }

        public async Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> events)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                if (events == null)
                {
                    return;
                }
                var eventDataList = events as IList<EventData> ?? events.ToList();

                // Trace Process Events
                TraceEventSource.Log.ProcessEvents(context.EventHubPath,
                                                   context.ConsumerGroupName,
                                                   context.Lease.PartitionId,
                                                   eventDataList.Count);

                using (var sqlConnection = new SqlConnection(sqlDatabaseConnectionString))
                {
                    await sqlConnection.OpenAsync();

                    var table = new DataTable();

                    // Add columns to the table
                    table.Columns.Add(EventId, typeof(int));
                    table.Columns.Add(DeviceId, typeof(int));
                    table.Columns.Add(Value, typeof(int));
                    table.Columns.Add(Timestamp, typeof(DateTime));

                    // Add rows to the table
                    foreach (var eventData in eventDataList)
                    {
                        // Note: EventData is disposable
                        using (eventData)
                        {
                            var data = DeserializeEventData(eventData);
                            table.Rows.Add(data.EventId, data.DeviceId, data.Value, data.Timestamp);
                        }
                    }

                    // Create command
                    var sqlCommand = new SqlCommand(InsertEventsStoreProcedure, sqlConnection)
                    {
                        CommandType = CommandType.StoredProcedure
                    };

                    // Add table-valued parameter
                    sqlCommand.Parameters.Add(new SqlParameter
                        {
                            ParameterName = EventsParameter,
                            SqlDbType = SqlDbType.Structured,
                            TypeName = TableType,
                            Value = table,
                        });

                    // Execute the query
                    await sqlCommand.ExecuteNonQueryAsync();
                }
                await context.CheckpointAsync();
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                ex.InnerException != null ?
                                                ex.InnerException.Message :
                                                string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }

        public async Task CloseAsync(PartitionContext context, CloseReason reason)
        {
            try
            {
                // TraceIn
                TraceEventSource.Log.TraceIn();

                // Trace Open Partition
                TraceEventSource.Log.ClosePartition(context.EventHubPath,
                                                    context.ConsumerGroupName,
                                                    context.Lease.PartitionId,
                                                    reason.ToString());

                if (reason == CloseReason.Shutdown)
                {
                    await context.CheckpointAsync();
                }
            }
            catch (Exception ex)
            {
                // Trace Exception
                TraceEventSource.Log.TraceError(ex.Message,
                                                ex.InnerException != null ?
                                                ex.InnerException.Message :
                                                string.Empty);
            }
            finally
            {
                // TraceOut
                TraceEventSource.Log.TraceOut();
            }
        }
        #endregion

        #region Private Static Methods
        private static Payload DeserializeEventData(EventData eventData)
        {
            return JsonConvert.DeserializeObject<Payload>(Encoding.UTF8.GetString(eventData.GetBytes()));
        }
        #endregion
    }
}

StoreEventsToAzureSqlDatabase

This project defines the cloud service hosting the worker role. The following table contains the service definition file of the cloud service.
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="StoreEventsToAzureSqlDatabase"
                   xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"
                   schemaVersion="2014-06.2.4">
  <WorkerRole name="EventProcessorHostWorkerRole" vmsize="Small">
    <ConfigurationSettings>
      <Setting name="SqlDatabaseConnectionString" />
      <Setting name="StorageAccountConnectionString" />
      <Setting name="ServiceBusConnectionString" />
      <Setting name="EventHubName" />
      <Setting name="ConsumerGroupName" />
    </ConfigurationSettings>
    <Imports>
      <Import moduleName="RemoteAccess" />
      <Import moduleName="RemoteForwarder" />
    </Imports>
  </WorkerRole>
</ServiceDefinition>
The following table contains the service definition file of the cloud service. Make sure to substitute the placeholders with the expected information before deploying the could service to Azure.
<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="StoreEventsToAzureSqlDatabase"
                      xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration"
                      osFamily="4"
                      osVersion="*"
                      schemaVersion="2014-06.2.4">
  <Role name="EventProcessorHostWorkerRole">
    <Instances count="2" />
    <ConfigurationSettings>
      <Setting name="SqlDatabaseConnectionString" value="[AZURE SQL DATABASE CONNECTION STRING]" />
      <Setting name="StorageAccountConnectionString" value="[STORAGE ACCOUNT CONNECTION STRING]" />
      <Setting name="ServiceBusConnectionString" value="[SERVICE BUS CONNECTION STRING]" />
      <Setting name="EventHubName" value="[EVENT HUB NAME]" />
      <Setting name="ConsumerGroupName" value="[CONSUMER GROUP NAME]" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="..." />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="true" />
    </ConfigurationSettings>
    <Certificates>
      <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption"
                   thumbprint="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
                   thumbprintAlgorithm="sha1" />
    </Certificates>
  </Role>
</ServiceConfiguration>

To configure diagnostics configuration and in particular ETW Logs for the worker role, you can proceed as follows: right click on the role and selec Properties. On the Configuration property page make sure Enable Diagnostics is checked and click the Configure… button. On the Diagnostics configuration dialog go to the ETW Logs tab and select Enable transfer of ETW logs. Add the appropriate event sources to transfer by specifying the event source name and clicking on the Add Event Source button. In the sample, make sure to select the TraceEventSource as shown in the picture below.

Then, right click the new row and select Configure event storage… menu item. In the Storage Configuration dialog, specify the name of the default storage table and, optionally, specify the name of the target table for each EventId defined in the EventSource class. In this sample, all diagnostic messages generated by the TraceEventSource class are configured to be traced to the WADDiagnosticTable (the prefix WAD is automatically added by Windows Azure Diagnostics), while the logs generated by the EventProcessor class are stored in a separate table called WADEventProcessorTable.

Once added, configure additional properties like the storage location for the logs, the log level, any keyword filters and transfer frequency.

Sender

This application can be used to provision the event hub used by the application and simulate a configurable amount of devices.

The following table shows the configuration file of the application. Make sure to substitute the placeholders with the expected information before running the application.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <appSettings>
    <add key="namespace" value="[SERVICE BUS NAMESPACE]"/>
    <add key="keyName" value="[NAMESPACE LEVEL SAS KEY NAME]"/>
    <add key="keyValue" value="[NAMESPACE LEVEL SAS KEY VALUE]"/>
    <add key="eventHub" value="[EVENT HUB NAME]"/>
    <add key="partitionCount" value="16"/>
    <add key="retentionDays" value="7"/>
    <add key="location" value="Milan"/>
    <add key="deviceCount" value="10"/>
    <add key="eventInterval" value="1"/>
    <add key="minValue" value="20"/>
    <add key="maxValue" value="50"/>
  </appSettings>
  <system.serviceModel>
    <extensions>
      <!-- In this extension section we are introducing all known service bus extensions. User can remove the ones they don't need. -->
      <behaviorExtensions>
        <add name="connectionStatusBehavior"
          type="Microsoft.ServiceBus.Configuration.ConnectionStatusElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="transportClientEndpointBehavior"
          type="Microsoft.ServiceBus.Configuration.TransportClientEndpointBehaviorElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="serviceRegistrySettings"
          type="Microsoft.ServiceBus.Configuration.ServiceRegistrySettingsElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      </behaviorExtensions>
      <bindingElementExtensions>
        <add name="netMessagingTransport"
          type="Microsoft.ServiceBus.Messaging.Configuration.NetMessagingTransportExtensionElement, Microsoft.ServiceBus,  Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="tcpRelayTransport"
          type="Microsoft.ServiceBus.Configuration.TcpRelayTransportElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="httpRelayTransport"
          type="Microsoft.ServiceBus.Configuration.HttpRelayTransportElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="httpsRelayTransport"
          type="Microsoft.ServiceBus.Configuration.HttpsRelayTransportElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="onewayRelayTransport"
          type="Microsoft.ServiceBus.Configuration.RelayedOnewayTransportElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      </bindingElementExtensions>
      <bindingExtensions>
        <add name="basicHttpRelayBinding"
          type="Microsoft.ServiceBus.Configuration.BasicHttpRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="webHttpRelayBinding"
          type="Microsoft.ServiceBus.Configuration.WebHttpRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="ws2007HttpRelayBinding"
          type="Microsoft.ServiceBus.Configuration.WS2007HttpRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="netTcpRelayBinding"
          type="Microsoft.ServiceBus.Configuration.NetTcpRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="netOnewayRelayBinding"
          type="Microsoft.ServiceBus.Configuration.NetOnewayRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="netEventRelayBinding"
          type="Microsoft.ServiceBus.Configuration.NetEventRelayBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        <add name="netMessagingBinding"
          type="Microsoft.ServiceBus.Messaging.Configuration.NetMessagingBindingCollectionElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      </bindingExtensions>
    </extensions>
  </system.serviceModel>
</configuration>
The following table contains the code of the MainForm class. Spend a few minutes to analyze the code of the btnStart_Click method. This method check creates the event hub if it doesn’t exist and creates the SendKey used to create SAS tokens for individual devices. Then the code creates a separate Task for each device. Each Task start sending events using the selected transport (AMQP or HTTPS).
#region Using Directives
using System;
using System.Configuration;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Globalization;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Windows.Forms;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Microsoft.AzureCat.Samples.Entities;
#endregion

namespace Microsoft.AzureCat.Samples.Sender
{
    public partial class MainForm : Form
    {
        #region Private Constants
        //***************************
        // Formats
        //***************************
        private const string DateFormat = "<{0,2:00}:{1,2:00}:{2,2:00}> {3}";
        private const string ExceptionFormat = "Exception: {0}";
        private const string InnerExceptionFormat = "InnerException: {0}";
        private const string LogFileNameFormat = "WADTablesCleaner {0}.txt";

        //***************************
        // Constants
        //***************************
        private const string SaveAsTitle = "Save Log As";
        private const string SaveAsExtension = "txt";
        private const string SaveAsFilter = "Text Documents (*.txt)|*.txt";
        private const string Start = "Start";
        private const string Stop = "Stop";
        private const string SenderSharedAccessKey = "SenderSharedAccessKey";
        private const string DeviceId = "id";
        private const string DeviceName = "name";
        private const string DeviceLocation = "location";
        private const string Value = "value";

        //***************************
        // Configuration Parameters
        //***************************
        private const string NamespaceParameter = "namespace";
        private const string KeyNameParameter = "keyName";
        private const string KeyValueParameter = "keyValue";
        private const string EventHubParameter = "eventHub";
        private const string LocationParameter = "location";
        private const string PartitionCountParameter = "partitionCount";
        private const string RetentionDaysParameter = "retentionDays";
        private const string DeviceCountParameter = "deviceCount";
        private const string EventIntervalParameter = "eventInterval";
        private const string MinValueParameter = "minValue";
        private const string MaxValueParameter = "maxValue";
        private const string ApiVersion = "&api-version=2014-05";

        //***************************
        // Configuration Parameters
        //***************************
        private const string DefaultEventHubName = "SampleEventHub";
        private const int DefaultDeviceNumber = 10;
        private const int DefaultMinValue = 20;
        private const int DefaultMaxValue = 50;
        private const int DefaultEventIntervalInSeconds = 1;


        //***************************
        // Messages
        //***************************
        private const string NamespaceCannonBeNull = "The Service Bus namespace cannot be null.";
        private const string EventHubNameCannonBeNull = "The event hub name cannot be null.";
        private const string KeyNameCannonBeNull = "The senderKey name cannot be null.";
        private const string KeyValueCannonBeNull = "The senderKey value cannot be null.";
        private const string EventHubCreatedOrRetrieved = "Event hub [{0}] created or retrieved.";
        private const string MessagingFactoryCreated = "Device[{0,3:000}]. MessagingFactory created.";
        private const string SasToken = "Device[{0,3:000}]. SAS Token created.";
        private const string EventHubClientCreated = "Device[{0,3:000}]. EventHubClient created: Path=[{1}].";
        private const string HttpClientCreated = "Device[{0,3:000}]. HttpClient created: BaseAddress=[{1}].";
        private const string EventSent = "Device[{0,3:000}]. Message sent. PartitionKey=[{1}] Value=[{2}]";
        private const string SendFailed = "Device[{0,3:000}]. Message send failed: [{1}]";
        #endregion

        #region Private Fields
        private CancellationTokenSource cancellationTokenSource;
        private int eventId;
        #endregion

        #region Public Constructor
        /// <summary>
        /// Initializes a new instance of the MainForm class.
        /// </summary>
        public MainForm()
        {
            InitializeComponent();
            ConfigureComponent();
            ReadConfiguration();
        }
        #endregion

        #region Public Methods

        public void ConfigureComponent()
        {
            txtNamespace.AutoSize = false;
            txtNamespace.Size = new Size(txtNamespace.Size.Width, 24);
            txtKeyName.AutoSize = false;
            txtKeyName.Size = new Size(txtKeyName.Size.Width, 24);
            txtKeyValue.AutoSize = false;
            txtKeyValue.Size = new Size(txtKeyValue.Size.Width, 24);
            txtEventHub.AutoSize = false;
            txtEventHub.Size = new Size(txtEventHub.Size.Width, 24);
        }

        public void HandleException(Exception ex)
        {
            if (ex == null || string.IsNullOrEmpty(ex.Message))
            {
                return;
            }
            WriteToLog(string.Format(CultureInfo.CurrentCulture, ExceptionFormat, ex.Message));
            if (ex.InnerException != null && !string.IsNullOrEmpty(ex.InnerException.Message))
            {
                WriteToLog(string.Format(CultureInfo.CurrentCulture, InnerExceptionFormat, ex.InnerException.Message));
            }
        }
        #endregion

        #region Private Methods
        public static bool IsJson(string item)
        {
            if (item == null)
            {
                throw new ArgumentException("The item argument cannot be null.");
            }
            try
            {
                var obj = JToken.Parse(item);
                return obj != null;
            }
            catch (Exception)
            {
                return false;
            }
        }

        public string IndentJson(string json)
        {
            if (string.IsNullOrWhiteSpace(json))
            {
                return null;
            }
            dynamic parsedJson = JsonConvert.DeserializeObject(json);
            return JsonConvert.SerializeObject(parsedJson, Formatting.Indented);
        }

        private void ReadConfiguration()
        {
            try
            {
                txtNamespace.Text = ConfigurationManager.AppSettings[NamespaceParameter];
                txtKeyName.Text = ConfigurationManager.AppSettings[KeyNameParameter];
                txtKeyValue.Text = ConfigurationManager.AppSettings[KeyValueParameter];
                txtEventHub.Text = ConfigurationManager.AppSettings[EventHubParameter] ?? DefaultEventHubName;
                var eventHubDescription = new EventHubDescription(txtEventHub.Text);
                int value;
                var setting = ConfigurationManager.AppSettings[PartitionCountParameter];
                txtPartitionCount.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       eventHubDescription.PartitionCount.ToString(CultureInfo.InvariantCulture);
                setting = ConfigurationManager.AppSettings[RetentionDaysParameter];
                txtMessageRetentionInDays.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       eventHubDescription.MessageRetentionInDays.ToString(CultureInfo.InvariantCulture);
                txtLocation.Text = ConfigurationManager.AppSettings[LocationParameter];
                setting = ConfigurationManager.AppSettings[DeviceCountParameter];
                txtDeviceCount.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       DefaultDeviceNumber.ToString(CultureInfo.InvariantCulture);
                setting = ConfigurationManager.AppSettings[EventIntervalParameter];
                txtEventIntervalInSeconds.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       DefaultEventIntervalInSeconds.ToString(CultureInfo.InvariantCulture);
                setting = ConfigurationManager.AppSettings[MinValueParameter];
                txtMinValue.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       DefaultMinValue.ToString(CultureInfo.InvariantCulture);
                setting = ConfigurationManager.AppSettings[MaxValueParameter];
                txtMaxValue.Text = int.TryParse(setting, out value) ?
                                       value.ToString(CultureInfo.InvariantCulture) :
                                       DefaultMaxValue.ToString(CultureInfo.InvariantCulture);
            }
            catch (Exception ex)
            {
                HandleException(ex);
            }
        }

        private void WriteToLog(string message)
        {
            if (InvokeRequired)
            {
                Invoke(new Action<string>(InternalWriteToLog), new object[] { message });
            }
            else
            {
                InternalWriteToLog(message);
            }
        }

        private void InternalWriteToLog(string message)
        {
            lock (this)
            {
                if (string.IsNullOrEmpty(message))
                {
                    return;
                }
                var lines = message.Split('\n');
                var now = DateTime.Now;
                var space = new string(' ', 19);

                for (var i = 0; i < lines.Length; i++)
                {
                    if (i == 0)
                    {
                        var line = string.Format(DateFormat,
                                                 now.Hour,
                                                 now.Minute,
                                                 now.Second,
                                                 lines[i]);
                        lstLog.Items.Add(line);
                    }
                    else
                    {
                        lstLog.Items.Add(space + lines[i]);
                    }
                }
                lstLog.SelectedIndex = lstLog.Items.Count - 1;
                lstLog.SelectedIndex = -1;
            }
        }

        #endregion

        #region Event Handlers

        private void exitToolStripMenuItem_Click(object sender, EventArgs e)
        {
            Close();
        }

        private void clearLogToolStripMenuItem_Click(object sender, EventArgs e)
        {
            lstLog.Items.Clear();
        }

        /// <summary>
        /// Saves the log to a text file
        /// </summary>
        /// <param name="sender">MainForm object</param>
        /// <param name="e">System.EventArgs parameter</param>
        private void saveLogToolStripMenuItem_Click(object sender, EventArgs e)
        {
            try
            {
                if (lstLog.Items.Count <= 0)
                {
                    return;
                }
                saveFileDialog.Title = SaveAsTitle;
                saveFileDialog.DefaultExt = SaveAsExtension;
                saveFileDialog.Filter = SaveAsFilter;
                saveFileDialog.FileName = string.Format(LogFileNameFormat, DateTime.Now.ToString(CultureInfo.CurrentUICulture).Replace('/', '-').Replace(':', '-'));
                if (saveFileDialog.ShowDialog() != DialogResult.OK ||
                    string.IsNullOrEmpty(saveFileDialog.FileName))
                {
                    return;
                }
                using (var writer = new StreamWriter(saveFileDialog.FileName))
                {
                    foreach (var t in lstLog.Items)
                    {
                        writer.WriteLine(t as string);
                    }
                }
            }
            catch (Exception ex)
            {
                HandleException(ex);
            }
        }

        private void logWindowToolStripMenuItem_Click(object sender, EventArgs e)
        {
            splitContainer.Panel2Collapsed = !((ToolStripMenuItem)sender).Checked;
        }

        private void aboutToolStripMenuItem_Click(object sender, EventArgs e)
        {
            var form = new AboutForm();
            form.ShowDialog();
        }

        private void lstLog_Leave(object sender, EventArgs e)
        {
            lstLog.SelectedIndex = -1;
        }

        private void button_MouseEnter(object sender, EventArgs e)
        {
            var control = sender as Control;
            if (control != null)
            {
                control.ForeColor = Color.White;
            }
        }

        private void button_MouseLeave(object sender, EventArgs e)
        {
            var control = sender as Control;
            if (control != null)
            {
                control.ForeColor = SystemColors.ControlText;
            }
        }

        private void MainForm_Paint(object sender, PaintEventArgs e)
        {
            var width = (mainHeaderPanel.Size.Width - 48)/2;
            var halfWidth = (width - 16)/2;

            txtNamespace.Size = new Size(width, txtNamespace.Size.Height);
            txtKeyName.Size = new Size(width, txtKeyName.Size.Height);
            txtKeyValue.Size = new Size(width, txtKeyValue.Size.Height);
            txtEventHub.Size = new Size(width, txtEventHub.Size.Height);
            txtLocation.Size = new Size(width, txtLocation.Size.Height);
            txtPartitionCount.Size = new Size(halfWidth, txtPartitionCount.Size.Height);
            txtMessageRetentionInDays.Size = new Size(halfWidth, txtMessageRetentionInDays.Size.Height);
            txtDeviceCount.Size = new Size(halfWidth, txtDeviceCount.Size.Height);
            txtEventIntervalInSeconds.Size = new Size(halfWidth, txtEventIntervalInSeconds.Size.Height);
            txtMinValue.Size = new Size(halfWidth, txtMinValue.Size.Height);
            txtMinValue.Size = new Size(halfWidth, txtMinValue.Size.Height);

            txtEventHub.Location = new Point(32 + width, txtEventHub.Location.Y);
            txtKeyValue.Location = new Point(32 + width, txtKeyValue.Location.Y);
            txtLocation.Location = new Point(32 + width, txtLocation.Location.Y);
            txtMessageRetentionInDays.Location = new Point(32 + halfWidth, txtMessageRetentionInDays.Location.Y);
            txtEventIntervalInSeconds.Location = new Point(32 + halfWidth, txtEventIntervalInSeconds.Location.Y);
            txtMinValue.Location = new Point(32 + width, txtMinValue.Location.Y);
            txtMaxValue.Location = new Point(48 + width + halfWidth, txtMaxValue.Location.Y);

            lblEventHub.Location = new Point(32 + width, lblEventHub.Location.Y);
            lblKeyValue.Location = new Point(32 + width, lblKeyValue.Location.Y);
            lblLocation.Location = new Point(32 + width, lblLocation.Location.Y);
            lblMessageRetentionInDays.Location = new Point(32 + halfWidth, lblMessageRetentionInDays.Location.Y);
            lblEventIntervalInSeconds.Location = new Point(32 + halfWidth, lblEventIntervalInSeconds.Location.Y);
            lblMinValue.Location = new Point(32 + width, lblMinValue.Location.Y);
            lblMaxValue.Location = new Point(48 + width + halfWidth, lblMaxValue.Location.Y);
            radioButtonHttps.Location = new Point(32 + halfWidth, radioButtonAmqp.Location.Y);
        }

        private void MainForm_Shown(object sender, EventArgs e)
        {
            txtNamespace.SelectionLength = 0;
        }

        private async void btnStart_Click(object sender, EventArgs e)
        {
            try
            {
                if (string.Compare(btnStart.Text, Start, StringComparison.OrdinalIgnoreCase) == 0)
                {
                    // Change button text
                    btnStart.Text = Stop;

                    // Validate parameters
                    if (!ValidateParameters())
                    {
                        return;
                    }

                    // Create namespace manager
                    var namespaceUri = ServiceBusEnvironment.CreateServiceUri("sb", txtNamespace.Text, string.Empty);
                    var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(txtKeyName.Text, txtKeyValue.Text);
                    var namespaceManager = new NamespaceManager(namespaceUri, tokenProvider);

                    // Check if the event hub already exists, if not, create the event hub.
                    var eventHubDescription = await namespaceManager.EventHubExistsAsync(txtEventHub.Text) ?
                                              await namespaceManager.GetEventHubAsync(txtEventHub.Text) :
                                              await namespaceManager.CreateEventHubAsync(new EventHubDescription(txtEventHub.Text)
                                              {
                                                  PartitionCount = txtPartitionCount.IntegerValue,
                                                  MessageRetentionInDays = txtMessageRetentionInDays.IntegerValue
                                              });
                    WriteToLog(string.Format(EventHubCreatedOrRetrieved, txtEventHub.Text));

                    // Check if the SAS authorization rule used by devices to send events to the event hub already exists, if not, create the rule.
                    var authorizationRule = eventHubDescription.
                                            Authorization.
                                            FirstOrDefault(r => string.Compare(r.KeyName,
                                                                                SenderSharedAccessKey,
                                                                                StringComparison.InvariantCultureIgnoreCase)
                                                                                == 0) as SharedAccessAuthorizationRule;
                    if (authorizationRule == null)
                    {
                        authorizationRule = new SharedAccessAuthorizationRule(SenderSharedAccessKey,
                                                                                 SharedAccessAuthorizationRule.GenerateRandomKey(),
                                                                                 new[]
                                                                                 {
                                                                                     AccessRights.Send
                                                                                 });
                        eventHubDescription.Authorization.Add(authorizationRule);
                        await namespaceManager.UpdateEventHubAsync(eventHubDescription);
                    }

                    cancellationTokenSource = new CancellationTokenSource();
                    var serviceBusNamespace = txtNamespace.Text;
                    var eventHubName = txtEventHub.Text;
                    var senderKey = authorizationRule.PrimaryKey;
                    var location = txtLocation.Text;
                    var eventInterval = txtEventIntervalInSeconds.IntegerValue * 1000;
                    var minValue = txtMinValue.IntegerValue;
                    var maxValue = txtMaxValue.IntegerValue;
                    var cancellationToken = cancellationTokenSource.Token;

                    // Create one task for each device
                    for (var i = 1; i <= txtDeviceCount.IntegerValue; i++)
                    {
                        var deviceId = i;
                        #pragma warning disable 4014
                        #pragma warning disable 4014
                        Task.Run(async () =>
                        #pragma warning restore 4014
                        {
                            var deviceName = string.Format("device{0:000}", deviceId);
                            var random = new Random((int)DateTime.Now.Ticks);

                            if (radioButtonAmqp.Checked)
                            {
                                // The token has the following format:
                                // SharedAccessSignature sr={URI}&sig={HMAC_SHA256_SIGNATURE}&se={EXPIRATION_TIME}&skn={KEY_NAME}
                                var token = CreateSasTokenForAmqpSender(SenderSharedAccessKey,
                                                                        senderKey,
                                                                        serviceBusNamespace,
                                                                        eventHubName,
                                                                        deviceName,
                                                                        TimeSpan.FromDays(1));
                                WriteToLog(string.Format(SasToken, deviceId));

                                var messagingFactory = MessagingFactory.Create(ServiceBusEnvironment.CreateServiceUri("sb", serviceBusNamespace, ""), new MessagingFactorySettings
                                {
                                    TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(token),
                                    TransportType = TransportType.Amqp
                                });
                                WriteToLog(string.Format(MessagingFactoryCreated, deviceId));

                                // Each device uses a different publisher endpoint: [EventHub]/publishers/[PublisherName]
                                var eventHubClient = messagingFactory.CreateEventHubClient(String.Format("{0}/publishers/{1}", eventHubName, deviceName));
                                WriteToLog(string.Format(EventHubClientCreated, deviceId, eventHubClient.Path));

                                while (!cancellationToken.IsCancellationRequested)
                                {
                                    // Create random value
                                    var value = random.Next(minValue, maxValue + 1);

                                    // Create EventData object with the payload serialized in JSON format 
                                    using (var eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(new Payload
                                    {
                                        EventId = eventId++,
                                        DeviceId = deviceId,
                                        Value = value,
                                        Timestamp = DateTime.UtcNow
                                    })))
                                    {
                                        PartitionKey = deviceName
                                    })
                                    {
                                        // Create custom properties
                                        eventData.Properties.Add(DeviceId, deviceId);
                                        eventData.Properties.Add(DeviceName, deviceName);
                                        eventData.Properties.Add(DeviceLocation, location);
                                        eventData.Properties.Add(Value, value);

                                        // Send the event to the event hub
                                        await eventHubClient.SendAsync(eventData);
                                        WriteToLog(string.Format(EventSent, deviceId, deviceName, value));
                                    }

                                    // Wait for the event time interval
                                    Thread.Sleep(eventInterval);
                                }
                            }
                            else
                            {
                                // The token has the following format:
                                // SharedAccessSignature sr={URI}&sig={HMAC_SHA256_SIGNATURE}&se={EXPIRATION_TIME}&skn={KEY_NAME}
                                var token = CreateSasTokenForHttpsSender(SenderSharedAccessKey,
                                                                         senderKey,
                                                                         serviceBusNamespace,
                                                                         eventHubName,
                                                                         deviceName,
                                                                         TimeSpan.FromDays(1));
                                WriteToLog(string.Format(SasToken, deviceId));

                                // Create HttpClient object used to send events to the event hub.
                                var httpClient = new HttpClient
                                {
                                    BaseAddress = new Uri(String.Format("https://{0}.servicebus.windows.net/{1}/publishers/{2}",
                                                                        serviceBusNamespace,
                                                                        eventHubName,
                                                                        deviceName).ToLower())
                                };
                                httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", token);
                                httpClient.DefaultRequestHeaders.Add("ContentType", "application/json;type=entry;charset=utf-8");
                                WriteToLog(string.Format(HttpClientCreated, deviceId, httpClient.BaseAddress));

                                while (!cancellationToken.IsCancellationRequested)
                                {
                                    // Create random value
                                    var value = random.Next(minValue, maxValue + 1);

                                    // Create HttpContent
                                    var postContent = new ByteArrayContent(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(new Payload
                                    {
                                        EventId = eventId++,
                                        DeviceId = deviceId,
                                        Value = value,
                                        Timestamp = DateTime.UtcNow
                                    })));

                                    // Create custom properties

                                    postContent.Headers.Add(DeviceId, deviceId.ToString(CultureInfo.InvariantCulture));
                                    postContent.Headers.Add(DeviceName, deviceName);
                                    //postContent.Headers.Add(DeviceLocation, location);
                                    postContent.Headers.Add(Value, value.ToString(CultureInfo.InvariantCulture));

                                    try
                                    {
                                        var response = await httpClient.PostAsync(httpClient.BaseAddress + "/messages" + "?timeout=60" + ApiVersion, postContent, cancellationToken);
                                        response.EnsureSuccessStatusCode();
                                        WriteToLog(string.Format(EventSent, deviceId, deviceName, value));
                                    }
                                    catch (HttpRequestException ex)
                                    {
                                        WriteToLog(string.Format(SendFailed, deviceId, ex.Message));
                                    }
                                }
                            }
                        },
                        cancellationToken).ContinueWith(t =>
#pragma warning restore 4014
                        #pragma warning restore 4014
                        {
                            if (t.IsFaulted && t.Exception != null)
                            {
                                HandleException(t.Exception);
                            }
                        }, cancellationToken);
                    }

                }
                else
                {
                    // Change button text
                    btnStart.Text = Start;
                    if (cancellationTokenSource != null)
                    {
                        cancellationTokenSource.Cancel();
                    }
                }
            }
            catch (Exception ex)
            {
                HandleException(ex);
            }
        }

        private bool ValidateParameters()
        {
            if (string.IsNullOrWhiteSpace(txtNamespace.Text))
            {
                WriteToLog(NamespaceCannonBeNull);
                return false;
            }
            if (string.IsNullOrWhiteSpace(txtEventHub.Text))
            {
                WriteToLog(EventHubNameCannonBeNull);
                return false;
            }
            if (string.IsNullOrWhiteSpace(txtKeyName.Text))
            {
                WriteToLog(KeyNameCannonBeNull);
                return false;
            }
            if (string.IsNullOrWhiteSpace(txtKeyValue.Text))
            {
                WriteToLog(KeyValueCannonBeNull);
                return false;
            }
            return true;
        }

        public static string CreateSasTokenForAmqpSender(string senderKeyName,
                                                         string senderKey,
                                                         string serviceNamespace,
                                                         string hubName,
                                                         string publisherName,
                                                         TimeSpan tokenTimeToLive)
        {
            // This is the format of the publisher endpoint. Each device uses a different publisher endpoint.
            // sb://<NAMESPACE>.servicebus.windows.net/<EVENT_HUB_NAME>/publishers/<PUBLISHER_NAME>. 
            var serviceUri = ServiceBusEnvironment.CreateServiceUri("sb",
                                                                    serviceNamespace,
                                                                    String.Format("{0}/publishers/{1}",
                                                                                   hubName,
                                                                                   publisherName))
                .ToString()
                .Trim('/');
            // SharedAccessSignature sr=<URL-encoded-resourceURI>&sig=<URL-encoded-signature-string>&se=<expiry-time-in-ISO-8061-format. >&skn=<senderKeyName>
            return SharedAccessSignatureTokenProvider.GetSharedAccessSignature(senderKeyName, senderKey, serviceUri, tokenTimeToLive);
        }

        // Create a SAS token for a specified scope. SAS tokens are described in http://msdn.microsoft.com/en-us/library/windowsazure/dn170477.aspx.
        private static string CreateSasTokenForHttpsSender(string senderKeyName,
                                                           string senderKey,
                                                           string serviceNamespace,
                                                           string hubName,
                                                           string publisherName,
                                                           TimeSpan tokenTimeToLive)
        {
            // Set token lifetime. When supplying a device with a token, you might want to use a longer expiration time.
            var origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
            var difference = DateTime.Now.ToUniversalTime() - origin;
            var tokenExpirationTime = Convert.ToUInt32(difference.TotalSeconds) + tokenTimeToLive.Seconds;

            // https://<NAMESPACE>.servicebus.windows.net/<EVENT_HUB_NAME>/publishers/<PUBLISHER_NAME>. 
            var uri = ServiceBusEnvironment.CreateServiceUri("https", serviceNamespace, String.Format("{0}/publishers/{1}", hubName, publisherName))
                .ToString()
                .Trim('/');
            var stringToSign = HttpUtility.UrlEncode(uri) + "\n" + tokenExpirationTime;
            var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(senderKey));

            var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

            // SharedAccessSignature sr=<URL-encoded-resourceURI>&sig=<URL-encoded-signature-string>&se=<expiry-time-in-ISO-8061-format. >&skn=<senderKeyName>
            var token = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}",
            HttpUtility.UrlEncode(uri), HttpUtility.UrlEncode(signature), tokenExpirationTime, senderKeyName);
            return token;
        }
        #endregion
    }
}

How to implement a partitioned SendBatch method for Azure Service Bus entities

$
0
0

Introduction

When a developer tries to use the SendBatch or SendBatchAsync methods exposed by the MessageSender, QueueClient, TopicClient, and EventHubClient classes contained in the Microsoft.ServiceBus.dll, and the batch size is greater than the maximum allowed size for a BrokeredMessage or an EventHub object (at the time of writing, the limit is 256 KB), the method call throws a MessageSizeExceededException. This library contains synchronous and asynchronous extension methods for the MessageSender, QueueClient,TopicClient, and EventHubClient classes that allow to send a batch which size is greater than the maximum allowed size for a batch. In particular, the implementation partitions the batch into one or multiple batches, each smaller than the maximum size allowed, and sends them in a loop, to respect the chronological order of the messages contained in the original batch. The code can be found here.

GitHub

The solution is also available on GitHub.

Solution

The ServiceBusExtensions library contains 4 classes:

  • MessageSenderExtensions: this class exposes the SendPartitionedBatch and SendPartitionedBatchAsync extension methods for the MessageSender class.
  • QueueClientExtensions: this class exposes the SendPartitionedBatch andSendPartitionedBatchAsync extension methods for the QueueClient class.
  • TopicClientExtensions: this class exposes the SendPartitionedBatch andSendPartitionedBatchAsync extension methods for the TopicClient class.
  • EventHubClientExtensions: this class exposes the SendPartitionedBatch andSendPartitionedBatchAsync extension methods for the EventHubClient class.

The TesterClient project contains a Console Application that can be used to test the library

ServiceBusExtensions Library

The following table contains the code of the QueueClientExtensions class. The code for the MessageSenderExtensions and TopicClientExtensions classes is very similar, so I will omit it for simplicity.

#region Using Directives
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using Microsoft.ServiceBus.Messaging;
using System.Threading.Tasks;
#endregion

namespace Microsoft.AzureCat.ServiceBusExtensions
{
  /// <summary>
  /// This class contains extensions methods for the QueueClient class
  /// </summary>
  public static class QueueClientClientExtensions
  {
    #region Private Constants
    //*******************************
    // Formats
    //*******************************
    private const string BrokeredMessageListCannotBeNullOrEmpty = "The brokeredMessageEnumerable parameter cannot be null or empty.";
    private const string SendPartitionedBatchFormat = "[QueueClient.SendPartitionedBatch] Batch Sent: BatchSizeInBytes=[{0}] MessageCount=[{1}]";
    private const string SendPartitionedBatchAsyncFormat = "[QueueClient.SendPartitionedBatchAsync] Batch Sent: BatchSizeInBytes=[{0}] MessageCount=[{1}]";
    #endregion

    #region Public Methods
    /// <summary>
    /// Sends a set of brokered messages (for batch processing).
    /// If the batch size is greater than the maximum batch size,
    /// the method partitions the original batch into multiple batches,
    /// each smaller in size than the maximum batch size.
    /// </summary>
    /// <param name="queueClient">The current QueueClient object.</param>
    /// <param name="brokeredMessageEnumerable">The collection of brokered messages to send.</param>
    /// <param name="trace">true to cause a message to be written; otherwise, false.</param>
    /// <returns>The asynchronous operation.</returns>
    public async static Task SendPartitionedBatchAsync(this QueueClient queueClient, IEnumerable<BrokeredMessage> brokeredMessageEnumerable, bool trace = false)
    {
      var brokeredMessageList = brokeredMessageEnumerable as IList<BrokeredMessage> ?? brokeredMessageEnumerable.ToList();
      if (brokeredMessageEnumerable == null || !brokeredMessageList.Any())
      {
        throw new ArgumentNullException(BrokeredMessageListCannotBeNullOrEmpty);
      }

      var batchList = new List<BrokeredMessage>();
      long batchSize = 0;

      foreach (var brokeredMessage in brokeredMessageList)
      {
        if ((batchSize + brokeredMessage.Size) > Constants.MaxBathSizeInBytes)
        {
          // Send current batch
          await queueClient.SendBatchAsync(batchList);
          Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchAsyncFormat, batchSize, batchList.Count));

          // Initialize a new batch
          batchList = new List<BrokeredMessage> { brokeredMessage };
          batchSize = brokeredMessage.Size;
        }
        else
        {
          // Add the BrokeredMessage to the current batch
          batchList.Add(brokeredMessage);
          batchSize += brokeredMessage.Size;
        }
      }
      // The final batch is sent outside of the loop
      await queueClient.SendBatchAsync(batchList);
      Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchAsyncFormat, batchSize, batchList.Count));
    }

    /// <summary>
    /// Sends a set of brokered messages (for batch processing).
    /// If the batch size is greater than the maximum batch size,
    /// the method partitions the original batch into multiple batches,
    /// each smaller in size than the maximum batch size.
    /// </summary>
    /// <param name="queueClient">The current QueueClient object.</param>
    /// <param name="brokeredMessageEnumerable">The collection of brokered messages to send.</param>
    /// <param name="trace">true to cause a message to be written; otherwise, false.</param>
    public static void SendPartitionedBatch(this QueueClient queueClient, IEnumerable<BrokeredMessage> brokeredMessageEnumerable, bool trace = false)
    {
      var brokeredMessageList = brokeredMessageEnumerable as IList<BrokeredMessage> ?? brokeredMessageEnumerable.ToList();
      if (brokeredMessageEnumerable == null || !brokeredMessageList.Any())
      {
        throw new ArgumentNullException(BrokeredMessageListCannotBeNullOrEmpty);
      }

      var batchList = new List<BrokeredMessage>();
      long batchSize = 0;

      foreach (var brokeredMessage in brokeredMessageList)
      {
        if ((batchSize + brokeredMessage.Size) > Constants.MaxBathSizeInBytes)
        {
          // Send current batch
          queueClient.SendBatch(batchList);
          Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchFormat, batchSize, batchList.Count));

          // Initialize a new batch
          batchList = new List<BrokeredMessage> { brokeredMessage };
          batchSize = brokeredMessage.Size;
        }
        else
        {
          // Add the BrokeredMessage to the current batch
          batchList.Add(brokeredMessage);
          batchSize += brokeredMessage.Size;
        }
      }
      // The final batch is sent outside of the loop
      queueClient.SendBatch(batchList);
      Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchFormat, batchSize, batchList.Count));
    }
    #endregion
  }
}

The following table contains the code of the EventHubClientExtensions class. Note: all the event data sent in a batch using the EventHubClient.SendBatch or EventHubClient.SendBatchAsync methods need to have the same PartitionKey. In fact, when using one of these methods, all the event data contained in the batch are insersted in the same partition of the event hub by the Event Hub message broker. Hence, they need to share the same value in the PartitionKey property as the latter is used to determine to which partition to send event data.

#region Copyright
//=======================================================================================
// Microsoft Azure Customer Advisory Team
//
// This sample is supplemental to the technical guidance published on the community
// blog at http://blogs.msdn.com/b/paolos/.
//
// Author: Paolo Salvatori
//=======================================================================================
// Copyright © 2015 Microsoft Corporation. All rights reserved.
//
// THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER
// EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. YOU BEAR THE RISK OF USING IT.
//=======================================================================================
#endregion

#region Using Directives
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using Microsoft.ServiceBus.Messaging;
using System.Threading.Tasks;
#endregion

namespace Microsoft.AzureCat.ServiceBusExtensions
{
  /// <summary>
  /// This class contains extensions methods for the EventHubClient class
  /// </summary>
  public static class EventHubClientExtensions
  {
    #region Private Constants
    //*******************************
    // Formats
    //*******************************
    private const string EventDataListCannotBeNullOrEmpty = "The eventDataEnumerable parameter cannot be null or empty.";
    private const string SendPartitionedBatchFormat = "[EventHubClient.SendPartitionedBatch] Batch Sent: BatchSizeInBytes=[{0}] MessageCount=[{1}]";
    private const string SendPartitionedBatchAsyncFormat = "[EventHubClient.SendPartitionedBatchAsync] Batch Sent: BatchSizeInBytes=[{0}] MessageCount=[{1}]";
    #endregion

    #region Public Methods
    /// <summary>
    /// Asynchronously sends a batch of event data to the same partition.
    /// All the event data in the batch need to have the same value in the Partitionkey property.
    /// If the batch size is greater than the maximum batch size,
    /// the method partitions the original batch into multiple batches,
    /// each smaller in size than the maximum batch size.
    /// </summary>
    /// <param name="eventHubClient">The current EventHubClient object.</param>
    /// <param name="eventDataEnumerable">An IEnumerable object containing event data instances.</param>
    /// <param name="trace">true to cause a message to be written; otherwise, false.</param>
    /// <returns>The asynchronous operation.</returns>
    public async static Task SendPartitionedBatchAsync(this EventHubClient eventHubClient, IEnumerable<EventData> eventDataEnumerable, bool trace = false)
    {
      var eventDataList = eventDataEnumerable as IList<EventData> ?? eventDataEnumerable.ToList();
      if (eventDataEnumerable == null || !eventDataList.Any())
      {
        throw new ArgumentNullException(EventDataListCannotBeNullOrEmpty);
      }

      var batchList = new List<EventData>();
      long batchSize = 0;

      foreach (var eventData in eventDataList)
      {
        if ((batchSize + eventData.SerializedSizeInBytes) > Constants.MaxBathSizeInBytes)
        {
          // Send current batch
          await eventHubClient.SendBatchAsync(batchList);
          Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchAsyncFormat, batchSize, batchList.Count));

          // Initialize a new batch
          batchList = new List<EventData> { eventData };
          batchSize = eventData.SerializedSizeInBytes;
        }
        else
        {
          // Add the EventData to the current batch
          batchList.Add(eventData);
          batchSize += eventData.SerializedSizeInBytes;
        }
      }
      // The final batch is sent outside of the loop
      await eventHubClient.SendBatchAsync(batchList);
      Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchAsyncFormat, batchSize, batchList.Count));
    }

    /// <summary>
    /// Asynchronously sends a batch of event data to the same partition.
    /// All the event data in the batch need to have the same value in the Partitionkey property.
    /// If the batch size is greater than the maximum batch size,
    /// the method partitions the original batch into multiple batches,
    /// each smaller in size than the maximum batch size.
    /// </summary>
    /// <param name="eventHubClient">The current EventHubClient object.</param>
    /// <param name="eventDataEnumerable">An IEnumerable object containing event data instances.</param>
    /// <param name="trace">true to cause a message to be written; otherwise, false.</param>
    public static void SendPartitionedBatch(this EventHubClient eventHubClient, IEnumerable<EventData> eventDataEnumerable, bool trace = false)
    {
      var eventDataList = eventDataEnumerable as IList<EventData> ?? eventDataEnumerable.ToList();
      if (eventDataEnumerable == null || !eventDataList.Any())
      {
        throw new ArgumentNullException(EventDataListCannotBeNullOrEmpty);
      }

      var batchList = new List<EventData>();
      long batchSize = 0;

      foreach (var eventData in eventDataList)
      {
        if ((batchSize + eventData.SerializedSizeInBytes) > Constants.MaxBathSizeInBytes)
        {
          // Send current batch
          eventHubClient.SendBatch(batchList);
          Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchFormat, batchSize, batchList.Count));

          // Initialize a new batch
          batchList = new List<EventData> { eventData };
          batchSize = eventData.SerializedSizeInBytes;
        }
        else
        {
          // Add the EventData to the current batch
          batchList.Add(eventData);
          batchSize += eventData.SerializedSizeInBytes;
        }
      }
      // The final batch is sent outside of the loop
      eventHubClient.SendBatch(batchList);
      Trace.WriteLineIf(trace, string.Format(SendPartitionedBatchFormat, batchSize, batchList.Count));
    }
    #endregion
  }
}

TestClient

The following picture shows the TestClient console application that can be used to test each extension method defined by the ServiceBusExtensions library.

TestClient

In the appSettings section of the configuration file you can define the following settings:

  • connectionString: the Service Bus namespace connectionstring.
  • messageSizeInBytes: the size of individual BrokeredMessage and EventData messages.
  • messageCountInBatch: the number of messages in a batch.

Upon start, the client application creates the following entities in the target Service Bus namespace, if they don’t already exist.

  • batchtestqueue: this this is the queue used to test the extensions method contained in the QueueClientExtensions and MessageSenderExtensions classes.
  • batchtesttopic: this is the topic used to test the extensions method contained in the TopicClientExtensions class.
  • batcheventhub: this is the event hub used to test the extensions method contained in the EventHubClientExtensions class

Then, the user can use the menu shown by the application to select one of the tests. Each test tries to use the original SendBatchAsync method exposed by each of the essageSender, QueueClient,TopicClient, and EventHubClient classes. If the batch size is greater than the maximum allowed size, the method call will throw a MessageSizeExceededException. The SendPartitionedBatchAsync method instead will split the original batch into one or multiple batches, each smaller than the maximum allowed size, and will send them in the proper order to the target entity.

For your convenience, the following tables includes the code of the TestClient console application.

#region Using Directives
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading.Tasks;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;

#endregion

namespace Microsoft.AzureCat.ServiceBusExtensions.TestClient
{
  /// <summary>
  /// This class can be used to test the extensions methods defines in the ServiceBusExtensions library.
  /// </summary>
  public class Program
  {
    #region Private Constants
    //***************************
    // Configuration Parameters
    //***************************
    private const string ConnectionString = "connectionString";
    private const string MessageSizeInBytes = "messageSizeInBytes";
    private const string MessageCountInBatch = "messageCountInBatch";

    //***************************
    // Entities
    //***************************
    private const string QueueName = "batchtestqueue";
    private const string TopicName = "batchtesttopic";
    private const string SubscriptionName = "auditing";
    private const string EventHubName = "batchtesteventhub";

    //***************************
    // Default Values
    //***************************
    private const int DefaultMessageCountInBatch = 100;
    private const int DefaultMessageSizeInBytes = 16384;

    //***************************
    // Formats
    //***************************
    private const string ParameterFormat = "{0}: [{1}]";
    private const string PressKeyToExit = "Press a key to exit.";
    private const string MenuChoiceFormat = "Select a numeric key between 1 and {0}";
    private const string ConnectionStringCannotBeNull = "The Service Bus connection string has not been defined in the configuration file.";
    private const string QueueCreatedFormat = "Queue [{0}] successfully created.";
    private const string TopicCreatedFormat = "Topic [{0}] successfully created.";
    private const string SubscriptionCreatedFormat = "Subscription [{0}] successfully created.";
    private const string EventHubCreatedFormat = "Event Hub [{0}] successfully created.";
    private const string QueueAlreadyExistsFormat = "Queue [{0}] already exists.";
    private const string TopicAlreadyExistsFormat = "Topic [{0}] already exists.";
    private const string SubscriptionAlreadyExistsFormat = "Subscription [{0}] already exists.";
    private const string EventHubAlreadyExistsFormat = "Event Hub [{0}] already exists.";
    private const string CallingMessageSenderSendBatchAsync = "Calling MessageSender.SendBatchAsync...";
    private const string MessageSenderSendBatchAsyncCalled = "MessageSender.SendBatchAsync called.";
    private const string CallingMessageSenderSendPartitionedBatchAsync = "Calling MessageSender.SendPartitionedBatchAsync...";
    private const string MessageSenderSendPartitionedBatchAsyncCalled = "MessageSender.SendPartitionedBatchAsync called.";
    private const string CallingQueueClientSendBatchAsync = "Calling QueueClient.SendBatchAsync...";
    private const string QueueClientSendBatchAsyncCalled = "QueueClient.SendBatchAsync called.";
    private const string CallingQueueClientSendPartitionedBatchAsync = "Calling QueueClient.SendPartitionedBatchAsync...";
    private const string QueueClientSendPartitionedBatchAsyncCalled = "QueueClient.SendPartitionedBatchAsync called.";
    private const string CallingTopicClientSendBatchAsync = "Calling TopicClient.SendBatchAsync...";
    private const string TopicClientSendBatchAsyncCalled = "TopicClient.SendBatchAsync called.";
    private const string CallingTopicClientSendPartitionedBatchAsync = "Calling TopicClient.SendPartitionedBatchAsync...";
    private const string TopicClientSendPartitionedBatchAsyncCalled = "TopicClient.SendPartitionedBatchAsync called.";
    private const string CallingEventHubClientSendBatchAsync = "Calling EventHubClient.SendBatchAsync...";
    private const string EventHubClientSendBatchAsyncCalled = "EventHubClient.SendBatchAsync called.";
    private const string CallingEventHubClientSendPartitionedBatchAsync = "Calling EventHubClient.SendPartitionedBatchAsync...";
    private const string EventHubClientSendPartitionedBatchAsyncCalled = "EventHubClient.SendPartitionedBatchAsync called.";

    //***************************
    // Menu Items
    //***************************
    private const string MessageSenderSendPartitionedBatchAsyncTest = "MessageSender.SendPartitionedBatchAsync Test";
    private const string QueueClientSendPartitionedBatchAsyncTest = "QueueClient.SendPartitionedBatchAsync Test";
    private const string TopicClientSendPartitionedBatchAsyncTest = "TopicClient.SendPartitionedBatchAsync Test";
    private const string EventHubClientSendPartitionedBatchAsyncTest = "EventHubClient.SendPartitionedBatchAsync Test";
    private const string Exit = "Exit";
    #endregion

    #region Private Static Fields
    private static string connectionString;
    private static int messageSizeInBytes;
    private static int messageCountInBatch;
    private static MessagingFactory messagingFactory;
    private static readonly List<string> menuItemList = new List<string>
    {
      MessageSenderSendPartitionedBatchAsyncTest ,
      QueueClientSendPartitionedBatchAsyncTest,
      TopicClientSendPartitionedBatchAsyncTest,
      EventHubClientSendPartitionedBatchAsyncTest,
      Exit
    };
    #endregion

    #region Main Method
    public static void Main()
    {
      try
      {
        if (ReadConfiguration() && CreateEntitiesAsync().Result)
        {
          // Add ConsoleTraceListener
          Trace.Listeners.Add(new ConsoleTraceListener());

          // Create MessagingFactory object
          messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);

          int key;
          while ((key = ShowMenu()) != menuItemList.Count - 1)
          {
            switch (menuItemList[key])
            {
              case MessageSenderSendPartitionedBatchAsyncTest:
                // Test MessageSender.SendPartitionedBatchAsync method
                MessageSenderTest().Wait();
                break;
              case QueueClientSendPartitionedBatchAsyncTest:
                // Test QueueClient.SendPartitionedBatchAsync method
                QueueClientTest().Wait();
                break;
              case TopicClientSendPartitionedBatchAsyncTest:
                // Test TopicClient.SendPartitionedBatchAsync method
                TopicClientTest().Wait();
                break;
              case EventHubClientSendPartitionedBatchAsyncTest:
                // Test EventHubClient.SendPartitionedBatchAsync method
                EventHubClientTest().Wait();
                break;
              case Exit:
                break;
            }
          }
        }
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
      PrintMessage(PressKeyToExit);
      Console.ReadLine();
    }
    #endregion

    #region Private Static Methods
    private async static Task MessageSenderTest()
    {
      // Create MessageSender object
      var messageSender = await messagingFactory.CreateMessageSenderAsync(QueueName);

      // Test MessageSender.SendBatchAsync: if the batch size is greater than the max batch size
      // the method throws a  MessageSizeExceededException
      try
      {
        PrintMessage(CallingMessageSenderSendBatchAsync);
        await messageSender.SendBatchAsync(CreateBrokeredMessageBatch());
        PrintMessage(MessageSenderSendBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }

      try
      {
        // Send the batch using the SendPartitionedBatchAsync method
        PrintMessage(CallingMessageSenderSendPartitionedBatchAsync);
        await messageSender.SendPartitionedBatchAsync(CreateBrokeredMessageBatch(), true);
        PrintMessage(MessageSenderSendPartitionedBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
    }

    private async static Task QueueClientTest()
    {
      // Create QueueClient object
      var queueClient = messagingFactory.CreateQueueClient(QueueName);

      // Test QueueClient.SendBatchAsync: if the batch size is greater than the max batch size
      // the method throws a  MessageSizeExceededException
      try
      {
        PrintMessage(CallingQueueClientSendBatchAsync);
        await queueClient.SendBatchAsync(CreateBrokeredMessageBatch());
        PrintMessage(QueueClientSendBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }

      try
      {
        // Send the batch using the SendPartitionedBatchAsync method
        PrintMessage(CallingQueueClientSendPartitionedBatchAsync);
        await queueClient.SendPartitionedBatchAsync(CreateBrokeredMessageBatch(), true);
        PrintMessage(QueueClientSendPartitionedBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
    }

    private async static Task TopicClientTest()
    {
      // Create TopicClient object
      var topicClient = messagingFactory.CreateTopicClient(TopicName);

      // Test TopicClient.SendBatchAsync: if the batch size is greater than the max batch size
      // the method throws a  MessageSizeExceededException
      try
      {
        PrintMessage(CallingTopicClientSendBatchAsync);
        await topicClient.SendBatchAsync(CreateBrokeredMessageBatch());
        PrintMessage(TopicClientSendBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }

      try
      {
        // Send the batch using the SendPartitionedBatchAsync method
        PrintMessage(CallingTopicClientSendPartitionedBatchAsync);
        await topicClient.SendPartitionedBatchAsync(CreateBrokeredMessageBatch(), true);
        PrintMessage(TopicClientSendPartitionedBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
    }

    private async static Task EventHubClientTest()
    {
      // Create EventHubClient object
      var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, EventHubName);

      // Test EventHubClient.SendBatchAsync: if the batch size is greater than the max batch size
      // the method throws a  MessageSizeExceededException
      try
      {
        PrintMessage(CallingEventHubClientSendBatchAsync);
        await eventHubClient.SendBatchAsync(CreateEventDataBatch());
        PrintMessage(EventHubClientSendBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }

      try
      {
        // Send the batch using the SendPartitionedBatchAsync method
        PrintMessage(CallingEventHubClientSendPartitionedBatchAsync);
        await eventHubClient.SendPartitionedBatchAsync(CreateEventDataBatch(), true);
        PrintMessage(EventHubClientSendPartitionedBatchAsyncCalled);
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
    }

    private static IEnumerable<BrokeredMessage> CreateBrokeredMessageBatch()
    {
      var messageList = new List<BrokeredMessage>();
      for (var i = 0; i < messageCountInBatch; i++)
      {
        messageList.Add(new BrokeredMessage(Encoding.UTF8.GetBytes(new string('A', messageSizeInBytes))));
      }
      return messageList;
    }

    private static IEnumerable<EventData> CreateEventDataBatch()
    {
      var messageList = new List<EventData>();
      for (var i = 0; i < messageCountInBatch; i++)
      {
        // Note: the partition key in this sample is null.
        // it's mandatory that all event data in a batch have the same PartitionKey
        messageList.Add(new EventData(Encoding.UTF8.GetBytes(new string('A', messageSizeInBytes))));
      }
      return messageList;
    }

    private async static Task<bool> CreateEntitiesAsync()
    {
      try
      {
        // Create NamespaceManeger object
        var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

        // Create test queue
        if (!await namespaceManager.QueueExistsAsync(QueueName))
        {
          await namespaceManager.CreateQueueAsync(new QueueDescription(QueueName)
          {
            EnableBatchedOperations = true,
            EnableExpress = true,
            EnablePartitioning = true,
            EnableDeadLetteringOnMessageExpiration = true
          });
          PrintMessage(string.Format(QueueCreatedFormat, QueueName));
        }
        else
        {
          PrintMessage(string.Format(QueueAlreadyExistsFormat, QueueName));
        }

        // Create test topic
        if (!await namespaceManager.TopicExistsAsync(TopicName))
        {
          await namespaceManager.CreateTopicAsync(new TopicDescription(TopicName)
          {
            EnableBatchedOperations = true,
            EnableExpress = true,
            EnablePartitioning = true,
          });
          PrintMessage(string.Format(TopicCreatedFormat, TopicName));
        }
        else
        {
          PrintMessage(string.Format(TopicAlreadyExistsFormat, TopicName));
        }

        // Create test subscription
        if (!await namespaceManager.SubscriptionExistsAsync(TopicName, SubscriptionName))
        {
          await namespaceManager.CreateSubscriptionAsync(new SubscriptionDescription(TopicName, SubscriptionName)
          {
            EnableBatchedOperations = true
          }, new RuleDescription(new TrueFilter()));
          PrintMessage(string.Format(SubscriptionCreatedFormat, SubscriptionName));
        }
        else
        {
          PrintMessage(string.Format(SubscriptionAlreadyExistsFormat, SubscriptionName));
        }

        // Create test event hub
        if (!await namespaceManager.EventHubExistsAsync(EventHubName))
        {
          await namespaceManager.CreateEventHubAsync(new EventHubDescription(EventHubName)
          {
            PartitionCount = 16,
            MessageRetentionInDays = 1
          });
          PrintMessage(string.Format(EventHubCreatedFormat, EventHubName));
        }
        else
        {
          PrintMessage(string.Format(EventHubAlreadyExistsFormat, EventHubName));
        }
      }
      catch (Exception ex)
      {
        PrintException(ex);
        return false;
      }
      return true;
    }

    private static int ShowMenu()
    {
      // Print Menu Header
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("[");
      Console.ForegroundColor = ConsoleColor.Yellow;
      Console.Write("Menu");
      Console.ForegroundColor = ConsoleColor.Green;
      Console.WriteLine("]");
      Console.ResetColor();

      // Print Menu Items
      for (var i = 0; i < menuItemList.Count; i++)
      {
        Console.ForegroundColor = ConsoleColor.Green;
        Console.Write("[");
        Console.ForegroundColor = ConsoleColor.Yellow;
        Console.Write("{0}", i + 1);
        Console.ForegroundColor = ConsoleColor.Green;
        Console.Write("]");
        Console.ResetColor();
        Console.Write(": ");
        Console.WriteLine(menuItemList[i]);
        Console.ResetColor();
      }

      // Select an option
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("[");
      Console.ForegroundColor = ConsoleColor.Yellow;
      Console.Write(MenuChoiceFormat, menuItemList.Count);
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("]");
      Console.ResetColor();
      Console.Write(": ");

      var key = 'a';
      while (key < '1' || key > '9')
      {
        key = Console.ReadKey(true).KeyChar;
      }
      Console.WriteLine();
      return key - '1';
    }

    private static bool ReadConfiguration()
    {
      try
      {
        // Set window size
        Console.SetWindowSize(120, 40);

        // Read connectionString setting
        connectionString = ConfigurationManager.AppSettings[ConnectionString];
        if (string.IsNullOrWhiteSpace(connectionString))
        {
          throw new ArgumentException(ConnectionStringCannotBeNull);
        }
        PrintMessage(string.Format(ParameterFormat, ConnectionString, connectionString));

        // Read messageSizeInBytes setting
        int value;
        var setting = ConfigurationManager.AppSettings[MessageSizeInBytes];
        messageSizeInBytes = int.TryParse(setting, out value) ?
                value :
                DefaultMessageSizeInBytes;
        PrintMessage(string.Format(ParameterFormat, MessageSizeInBytes, messageSizeInBytes));

        // Read messageCountInBatch setting
        setting = ConfigurationManager.AppSettings[MessageCountInBatch];
        messageCountInBatch = int.TryParse(setting, out value) ?
                value :
                DefaultMessageCountInBatch;
        PrintMessage(string.Format(ParameterFormat, MessageCountInBatch, messageCountInBatch));
        return true;
      }
      catch (Exception ex)
      {
        PrintException(ex);
      }
      return false;
    }

    private static void PrintMessage(string message, [CallerMemberName] string memberName = "")
    {
      if (string.IsNullOrWhiteSpace(message) || string.IsNullOrWhiteSpace(memberName))
      {
        return;
      }
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("[");
      Console.ForegroundColor = ConsoleColor.Yellow;
      Console.Write(memberName);
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("]");
      Console.ResetColor();
      Console.Write(": ");
      Console.WriteLine(message);
    }

    private static void PrintException(Exception ex,
                       [CallerFilePath] string sourceFilePath = "",
                       [CallerMemberName] string memberName = "",
                       [CallerLineNumber] int sourceLineNumber = 0)
    {
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("[");
      Console.ForegroundColor = ConsoleColor.Yellow;
      string fileName = null;
      if (File.Exists(sourceFilePath))
      {
        var file = new FileInfo(sourceFilePath);
        fileName = file.Name;
      }
      Console.Write(string.IsNullOrWhiteSpace(fileName) ? "Unknown" : fileName);
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write(":");
      Console.ForegroundColor = ConsoleColor.Yellow;
      Console.Write(string.IsNullOrWhiteSpace(memberName) ? "Unknown" : memberName);
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write(":");
      Console.ForegroundColor = ConsoleColor.Yellow;
      Console.Write(sourceLineNumber.ToString(CultureInfo.InvariantCulture));
      Console.ForegroundColor = ConsoleColor.Green;
      Console.Write("]");
      Console.ResetColor();
      Console.Write(": ");
      Console.WriteLine(ex != null && !string.IsNullOrWhiteSpace(ex.Message) ? ex.Message : "An error occurred.");
      var aggregateException = ex as AggregateException;
      if (aggregateException == null)
      {
        return;
      }
      foreach (var exception in aggregateException.InnerExceptions)
      {
        PrintException(exception);
      }
    }
    #endregion
  }
}

How to find absence of signal in a Stream Analytics job

$
0
0

Introduction

Recently, I worked at an IoT solution where the customer had to build a Stream Analytics query to check when no data has arrived from a given device in a configurable time window. To solve this problem, the idea is to correlate the data stream containing real-time events (e.g. sensor readings) with device reference data. In this context, real-time data comes through a data stream input that uses the Event Hub or IoT Hub provider for telemetry data ingestion, whereas the reference data is defined as a blob which contains the metadata of all the devices In the sample below, JSON format is used both by the data stream and reference data, but you can use another format between those supported by Stream Analytics providers.. Now, in order to find all the devices that didn’t send a message within a given time window, reference data input should appear in the Stream Analytics query as the left element of the LEFT JOIN clause. But Stream Analytics does not support Reference Data as the left element in a JOIN clause L So I had to tweak the code to bring Reference Data on the left part. However, at least at the time of this writing, Stream Analytics does not support Reference Data as the left element in a JOIN clause. So I had to tweak the code to bring reference data on the left part. This article describes this technique. Basically, you have to create a multi-step query as follows:

  1. The first step in the query below simply creates a single event every N seconds (60 in my sample).
  2. The second step joins reference data with the result of the first step to turn reference data into a temporal data stream. This way I can bring reference data on the left of a JOIN clause. Please note the use of the ON clause. This is sort of magic trick to correlate the event produced by the first query with the reference data.
  3. The third step calculates the event count for each device in the given time window. Note: the duration of the time window needs to be equal to the duration of the time window used by the first step.
  4. The final SELECT applies a LEFT JOIN to the output of the second step and the output of third step. Since the second step was obtained from reference data, which contains all the devices, and the reference data appears as the left element of a LEFT JOIN, the query will analyze all the devices, not only those that sent at least a message in the current time window.

Test Files

This section contains the test files used on the QUERY tab on the Azure Management Portal to test the query.

DeviceEvents.Json

[
    {
        "deviceId":1,
        "name":"device001",
        "value": 5,
        "status": "active",
        "timestamp": "2015-12-16T10:00:00.0000000Z"
    },
    {
        "deviceId":3,
        "name":"device003",
        "value":23,
        "status": "active",
        "timestamp": "2015-12-16T10:00:05.0000000Z"
    },
    {
        "deviceId":5,
        "name":"device005",
        "value":17,
        "status": "active",
        "timestamp": "2015-12-16T10:00:20.0000000Z"
    },
    {
        "deviceId":1,
        "name":"device001",
        "value":8,
        "status": "active",
        "timestamp": "2015-12-16T10:00:30.0000000Z"
    },
    {
        "deviceId":3,
        "name":"device003",
        "value":22,
        "status": "active",
        "timestamp": "2015-12-16T10:00:35.0000000Z"
    },
    {
        "deviceId":4,
        "name":"device004",
        "value": 18,
        "status": "active",
        "timestamp": "2015-12-16T10:00:36.0000000Z"
    },
    {
        "deviceId":1,
        "name":"device001",
        "value":65,
        "status": "active",
        "timestamp": "2015-12-16T10:01:00.0000000Z"
    },
    {
        "deviceId":3,
        "name":"device003",
        "value":28,
        "status": "active",
        "timestamp": "2015-12-16T10:01:05.0000000Z"
    },
    {
        "deviceId":5,
        "name":"device005",
        "value":3,
        "status": "active",
        "timestamp": "2015-12-16T10:01:15.0000000Z"
    },
    {
        "deviceId":1,
        "name":"device001",
        "value":54,
        "status": "active",
        "timestamp": "2015-12-16T10:01:30.0000000Z"
    },
    {
        "deviceId":3,
        "name":"device003",
        "value":43,
        "status": "active",
        "timestamp": "2015-12-16T10:01:35.0000000Z"
    }
]

Remarks:

  • Please look at the timestamp of the events: events are contained in about 2 minutes time range. Hence, they are segmented in 3 consecutive time windows of 60 seconds each.
  • Device with deviceId = 2 does not produce any events
  • Device with deviceId = 4 produces a single event in the second time window

DeviceReferenceData.json

This file needs to be copied to blob storage and used as reference data. It contains a record for each device.

[
    {
        "deviceId": 1,
        "location": "Milan",
        "building": "A01",
        "minThreshold": 20.0,
        "maxThreshold": 50.0
    },
    {
        "deviceId": 2,
        "location": "Milan",
        "building": "A01",
        "minThreshold": 20.0,
        "maxThreshold": 50.0
    },
    {
        "deviceId": 3,
        "location": "Milan",
        "building": "A01",
        "minThreshold": 20.0,
        "maxThreshold": 50.0
    },
    {
        "deviceId": 4,
        "location": "Milan",
        "building": "A01",
        "minThreshold": 20.0,
        "maxThreshold": 50.0
    },
    {
        "deviceId": 5,
        "location": "Milan",
        "building": "A02",
        "minThreshold": 20.0,
        "maxThreshold": 40.0
    }
]

First Query

In the first query, I calculate how many events have been received from each device in a given time window.

-- Find devices that it's at least 1 minute that don't send a message
WITH
OneEvent AS /* generate one event per period, any event */
(
                SELECT COUNT(*) As eventCount
                FROM DeviceEvents TIMESTAMP BY DeviceEvents.[timestamp]
                GROUP BY TumblingWindow(s, 60)
),
AllDevices AS /* generate one event per deviceId per period */
(
                SELECT ReferenceData.deviceId
                FROM OneEvent JOIN ReferenceData
                ON OneEvent.eventCount - OneEvent.eventCount = ReferenceData.deviceId - ReferenceData.deviceId
),
ActualCounts AS /* compute how many events have been received in the time window from each device */
(
                SELECT deviceId,
                       COUNT(*) AS eventCount
                FROM DeviceEvents TIMESTAMP BY DeviceEvents.[timestamp]
                GROUP BY TumblingWindow(s, 60), deviceId
)
/* left join AllDevices with ActualCounts to find devices with zero events */
SELECT
      AllDevices.deviceId,
      CASE WHEN ActualCounts.eventCount IS NULL THEN 0
           ELSE ActualCounts.eventCount
      END AS eventCount,
      System.Timestamp AS [timestamp]
FROM
AllDevices LEFT JOIN ActualCounts
ON
ActualCounts.deviceId = AllDevices.deviceId
AND DATEDIFF(ms, ActualCounts, AllDevices) = 0

Test First Query

If you test the first query with using DeviceEvents.json and DeviceReferenceData.json as test files, you will get the following results. Please note, that in the figure below results are contained in 3 consecutive time windows that are highlighted in 3 different colors.

image1

Second Query

In the second query, it’s sufficient to add a WHERE clause to project only the devices that didn’t produce any event in the given time window.

-- Find devices that it's at least 1 minute that don't send a message
WITH
OneEvent AS /* generate one event per period, any event */
(
                SELECT COUNT(*) As eventCount
                FROM DeviceEvents TIMESTAMP BY DeviceEvents.[timestamp]
                GROUP BY TumblingWindow(s, 60)
),
AllDevices AS /* generate one event per deviceId per period */
(
                SELECT ReferenceData.deviceId
                FROM OneEvent JOIN ReferenceData
                ON OneEvent.eventCount - OneEvent.eventCount = ReferenceData.deviceId - ReferenceData.deviceId
),
ActualCounts AS /* compute how many events have been received in the time window from each device */
(
                SELECT deviceId,
                       COUNT(*) AS eventCount
                FROM DeviceEvents TIMESTAMP BY DeviceEvents.[timestamp]
                GROUP BY TumblingWindow(s, 60), deviceId
)
/* left join AllDevices with ActualCounts to find devices with zero events */
SELECT
      AllDevices.deviceId,
      CASE WHEN ActualCounts.eventCount IS NULL THEN 0
           ELSE ActualCounts.eventCount
      END AS eventCount,
      System.Timestamp AS [timestamp]
FROM
AllDevices LEFT JOIN ActualCounts
ON
ActualCounts.deviceId = AllDevices.deviceId
AND DATEDIFF(ms, ActualCounts, AllDevices) = 0
WHERE ActualCounts.eventCount IS NULL

Test Second Query

If you test the first query with using DeviceEvents.json and DeviceReferenceData.json as test files, you will get the following results. Please note, that in the figure below results are contained in 3 consecutive time windows that are highlighted in 3 different colors.

Image2

Architecture Design

The following picture shows the architecture design of the demo. You can use my tool, Service Bus Explorer, to send device events to the input Event Hub and receive results from the output Event Hub.

Architecture

Inputs:

  • DeviceEvents: Event Hub Data Stream
  • ReferenceData: Blob Reference Data

 

Outputs:

  • Output: Event Hub Output

Provisoning

You can use the following ARM template and parameters file to create the Stream Analytics job. Note: this script creates only the Stream Analytics job and not the Event Hubs and Storage Account hosting the blob used as reference data. You can extend the ARM template to add the creation of the Event Hubs, storage account and container hosting the blob file. Make sure to copy the DeviceReferenceData.json to the container at the end of the provisioning process.

azuredeploy.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",

    "contentVersion": "1.0.0.0",

    "parameters": {
        "jobName": {
            "type": "string",
            "metadata": {
                "description": "Name of the Stream Analytics job",
                "defaultValue": "IoTDeviceDemo"
            }
        },
        "jobLocation": {
            "type": "string",
            "metadata": {
                "description": "Location of the Stream Analytics job",
                "defaultValue": "West Europe"
            }
        },
         "referenceStorageAccountName": {
            "type": "string",
            "metadata": {
                "description": "Name of the Storage Account containing the reference data blob"
            }
        },
        "referenceStorageAccountKey": {
            "type": "string",
            "metadata": {
                "description": "Key of the Storage Account containing the reference data blob"
            }
        },
        "referenceContainerName": {
            "type": "string",
            "metadata": {
                "description": "Name of the container of the reference data blob",
                "defaultValue": "devices"
            }
        },
        "referenceBlobName": {
            "type": "string",
            "metadata": {
                "description": "Name of the reference data blob",
                "defaultValue": "DeviceReferenceData.json"
            }
        },
        "inputServiceBusNamespace": {
            "type": "string",
            "metadata": {
                "description": "Name of the Service Bus namespace containing the input event hub"
            }
        },
        "inputEventHubName": {
            "type": "string",
            "metadata": {
                "description": "Name of the input event hub"
            }
        },
        "inputEventHubConsumerGroupName": {
            "type": "string",
            "metadata": {
                "description": "Name of the consumer group used by the job to read data out of the input event hub",
                "defaultValue": "$Default"
            }
        },
        "inputEventHubSharedAccessPolicyName": {
            "type": "string",
            "metadata": {
                "description": "Name of the Shared Access Policy of the input event hub"
            }
        },
        "inputEventHubSharedAccessPolicyKey": {
            "type": "string",
            "metadata": {
                "description": "Key of the Shared Access Policy of the input event hub"
            }
        },
        "outputServiceBusNamespace": {
            "type": "string",
            "metadata": {
                "description": "Name of the Service Bus namespace containing the output event hub"
            }
        },
        "outputEventHubName": {
            "type": "string",
            "metadata": {
                "description": "Name of the output event hub"
            }
        },
        "outputEventHubPartitionKey": {
            "type": "string",
            "metadata": {
                "description": "Name of the payload property used as partition key"
            }
        },
        "outputEventHubSharedAccessPolicyName": {
            "type": "string",
            "metadata": {
                "description": "Name of the Shared Access Policy of the output event hub"
            }
        },
        "outputEventHubSharedAccessPolicyKey": {
            "type": "string",
            "metadata": {
                "description": "Key of the Shared Access Policy of the output event hub"
            }
        }
    },
    "resources": [
        {
            "name": "[parameters('jobName')]",
            "type": "Microsoft.StreamAnalytics/streamingjobs",
            "apiVersion": "2015-04-01",
            "location": "[parameters('jobLocation')]",
            "properties": {
                "sku": {
                    "name": "Standard"
                },
                "eventsOutOfOrderPolicy": "Adjust",
                "eventsOutOfOrderMaxDelayInSeconds": 0,
                "eventsLateArrivalMaxDelayInSeconds": 5,
                "dataLocale": "en-US",
                "inputs": [
                    {
                        "name": "DeviceEvents",
                        "type": "Microsoft.StreamAnalytics/streamingjobs/inputs",
                        "properties": {
                            "type": "Stream",
                            "datasource": {
                                "type": "Microsoft.ServiceBus/EventHub",
                                "properties": {
                                    "eventHubName": "[parameters('inputEventHubName')]",
                                    "consumerGroupName": "[parameters('inputEventHubConsumerGroupName')]",
                                    "serviceBusNamespace": "[parameters('inputServiceBusNamespace')]",
                                    "sharedAccessPolicyName": "[parameters('inputEventHubSharedAccessPolicyName')]",
                                    "sharedAccessPolicyKey": "[parameters('inputEventHubSharedAccessPolicyKey')]"
                                }
                            },
                            "serialization": {
                                "type": "Json",
                                "properties": {
                                    "encoding": "UTF8"
                                }
                            }
                        }
                    },
                    {
                        "name": "ReferenceData",
                        "type": "Microsoft.StreamAnalytics/streamingjobs/inputs",
                        "properties": {
                            "type": "Reference",
                            "datasource": {
                                "type": "Microsoft.Storage/Blob",
                                "properties": {
                                    "blobName": "[parameters('referenceBlobName')]",
                                    "storageAccounts": [
                                        {
                                            "accountName": "[parameters('referenceStorageAccountName')]",
                                            "accountKey": "[parameters('referenceStorageAccountKey')]"
                                        }
                                    ],
                                    "container": "[parameters('referenceContainerName')]",
                                    "pathPattern": "[parameters('referenceBlobName')]"
                                }
                            },
                            "serialization": {
                                "type": "Json",
                                "properties": {
                                    "encoding": "UTF8"
                                }
                            }
                        }
                    }
                ],
                "transformation": {
                    "name": "DeviceDemo",
                    "type": "Microsoft.StreamAnalytics/streamingjobs/transformations",
                    "properties": {
                        "streamingUnits": 6,
                        "query": "-- Find devices that it's at least 1 minute that don't send a message\r\nWITH\r\n
OneEvent AS /* generate one event per period, any event */\r\n(\r\n
SELECT COUNT(*) As eventCount\r\n FROM DeviceEvents TIMESTAMP BY
DeviceEvents.[timestamp]\r\n GROUP BY TumblingWindow(s, 60)\r\n),
\r\nAllDevices AS /* generate one event per deviceId per period */\r\n(\r\n
SELECT ReferenceData.deviceId\r\n
FROM OneEvent JOIN ReferenceData\r\n ON OneEvent.eventCount -
OneEvent.eventCount = ReferenceData.deviceId - ReferenceData.deviceId\r\n),
\r\nActualCounts AS /* compute how many events have been received in the time
window from each device */\r\n(\r\n SELECT deviceId,
\r\n COUNT(*) AS eventCount\r\n
FROM DeviceEvents TIMESTAMP BY DeviceEvents.[timestamp]\r\n
GROUP BY TumblingWindow(s, 60), deviceId\r\n)\r\n/* left join AllDevices with
ActualCounts to find devices with zero events */\r\nSELECT\r\n
AllDevices.deviceId,\r\n CASE WHEN ActualCounts.eventCount IS NULL THEN 0
\r\n ELSE ActualCounts.eventCount\r\n END AS eventCount,
\r\n System.Timestamp AS [timestamp]\r\nFROM\r\nAllDevices LEFT JOIN
ActualCounts\r\nON\r\nActualCounts.deviceId = AllDevices.deviceId\r\nAND
DATEDIFF(ms, ActualCounts, AllDevices) = 0\r\nWHERE ActualCounts.eventCount IS NULL"
} }, "outputs": [ { "name": "Output", "type": "Microsoft.StreamAnalytics/streamingjobs/outputs", "properties": { "datasource": { "type": "Microsoft.ServiceBus/EventHub", "properties": { "eventHubName": "[parameters('outputEventHubName')]", "partitionKey": "[parameters('outputEventHubPartitionKey')]", "serviceBusNamespace": "[parameters('outputServiceBusNamespace')]", "sharedAccessPolicyName": "[parameters('outputEventHubSharedAccessPolicyName')]", "sharedAccessPolicyKey": "[parameters('outputEventHubSharedAccessPolicyKey')]" } }, "serialization": { "type": "Json", "properties": { "encoding": "UTF8" } } } } ] } } ] }

azuredeploy-parameters.json

{
    "jobName": {
        "value": "AbsenceOfSignal"
    },
    "jobLocation": {
        "value": "<job-location>"
    },
     "referenceStorageAccountName": {
        "value": "<storage-account-name>"
    },
    "referenceStorageAccountKey": {
        "value": "<storage-account-key>"
    },
    "referenceContainerName": {
        "value": "<container-name>"
    },
    "referenceBlobName": {
        "value": "DeviceReferenceData.json"
    },
    "inputServiceBusNamespace": {
        "value": "<input-event-hub-servicebus-namespace>"
    },
    "inputEventHubName": {
        "value": "<input-event-hub-name>"
    },
    "inputEventHubConsumerGroupName": {
        "value": "<input-event-hub-consumergroup-name>"
    },
    "inputEventHubSharedAccessPolicyName": {
        "value": "<input-event-hub-shared-access-policy-name>"
    },
    "inputEventHubSharedAccessPolicyKey": {
        "value": "<input-event-hub-shared-access-policy-key>"
    },
    "outputServiceBusNamespace": {
        "value": "<output-event-hub-servicebus-namespace>"
    },
    "outputEventHubName": {
        "value": "<output-event-hub-name>"
    },
    "outputEventHubPartitionKey": {
        "value": "deviceId"
    },
    "outputEventHubSharedAccessPolicyName": {
        "value": "<output-event-hub-shared-access-policy-name>"
    },
    "outputEventHubSharedAccessPolicyKey": {
        "value": "<output-event-hub-shared-access-policy-key>"
    }
}

Note: make sure that all the resources used by the solution are located in the same datacenter.

PowerShell Script

Use the following PowerShell Script to create the Stream Analytics job.

# To login to Azure Resource Manager
Login-AzureRmAccount

# Select a default subscription for your current session in case your account has multiple Azure subscriptions
Get-AzureRmSubscription –SubscriptionName “Paolo's Azure Account” | Select-AzureRmSubscription

# Initialize Variables
$resourceGroupName = 'StreamAnalyticsWestEuropeResourceGroup'
$deploymentName = 'StreamAnalyticsIoTDemo'
$location = "West Europe"
$templateFolderName = 'C:\Projects\Azure\StreamAnalytics\TestFiles\ARM\'
$templateFile = $templateFolderName + 'azuredeploy.json'
$templateParameterFile = $templateFolderName + 'azuredeploy-parameters.json'

# Check if the resource group for the Stream Analytics job already exists
$resourceGroup = Get-AzureRmResourceGroup -Name $resourceGroupName `
                                          -ErrorAction Ignore

if (!$resourceGroup)
{
    # Create the resource group if it does not exists
    Write-Host 'Creating' $resourceGroupName 'resource group...'
    $stopWatch = [Diagnostics.Stopwatch]::StartNew()
    New-AzureRmResourceGroup -Name $resourceGroupName `
                             -Location $location `
                             -ErrorAction Stop
    $stopWatch.Stop()
    Write-Host 'The resource group' $resourceGroupName ' has been successfully created in' $stopWatch.Elapsed.TotalSeconds 'seconds'
}
else
{
    Write-Host 'The resource group' $resourceGroupName 'already exists'
}

# Check if a deployment with the specified name already exists
$deployment = Get-AzureRmResourceGroupDeployment -Name $deploymentName `
                                                 -ResourceGroupName $resourceGroupName

if ($deployment)
{
    # Delete the resource group if it already exists
    Write-Host 'The' $deploymentName 'deployment already exists. Deleting the deployment...'
    $stopWatch = [Diagnostics.Stopwatch]::StartNew()
    Remove-AzureRmResourceGroupDeployment -Name $deploymentName `
                                          -ResourceGroupName $resourceGroupName `
                                          -Force `
                                          -ErrorAction Stop
    $stopWatch.Stop()
    Write-Host 'The deployment' $deploymentName ' has been successfully deleted in ' $stopWatch.Elapsed.TotalSeconds 'seconds'
}

# Create the resource group deployment
Write-Host 'Adding deployment' $deploymentName 'to' $resourceGroupName 'resource group...'
$stopWatch = [Diagnostics.Stopwatch]::StartNew()
New-AzureRmResourceGroupDeployment -Name $deploymentName `
                                   -ResourceGroupName $resourceGroupName `
                                   -TemplateFile $templateFile `
                                   -TemplateParameterFile $templateParameterFile `
                                   -ErrorAction Stop
$stopWatch.Stop()
$deployment = Get-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName `
                                                 -Name $deploymentName `
                                                 -ErrorAction Stop
Write-Host 'Operation Complete'
Write-Host '=================='
Write-Host 'Provisioning State :' $deployment[0].ProvisioningState
Write-Host 'Time elapsed :' $stopWatch.Elapsed.TotalSeconds 'seconds'

How to normalize incoming events in a Stream Analytics job

$
0
0

If you want to normalize the incoming events in a Stream Analytics job and transform records with the following format:

Time Id P1 P2 P3
2015-12-16T10:00:01.0000000Z 01 15 30 10
2015-12-16T10:00:01.0000000Z 02 25 40 20
2015-12-16T10:00:01.0000000Z 03 45 18 5
2015-12-16T10:00:01.0000000Z 04 20 23 56


to the following format:

Time Id Type Value
2015-12-16T10:00:01.0000000Z 01 P1 15
2015-12-16T10:00:01.0000000Z 01 P2 30
2015-12-16T10:00:01.0000000Z 01 P3 10
2015-12-16T10:00:01.0000000Z 02 P1 25
2015-12-16T10:00:01.0000000Z 02 P2 40
2015-12-16T10:00:01.0000000Z 02 P3 20
2015-12-16T10:00:01.0000000Z 03 P1 45
2015-12-16T10:00:01.0000000Z 03 P2 18
2015-12-16T10:00:01.0000000Z 03 P3 5
2015-12-16T10:00:01.0000000Z 04 P1 20
2015-12-16T10:00:01.0000000Z 04 P2 23
2015-12-16T10:00:01.0000000Z 04 P3 56


you can use the GetRecordProperties function and CROSS APPLY operator as shown by the following query:

SELECT e.Time, e.Id, record.PropertyName as Type, record.PropertyValue AS Value
FROM Events e
CROSS APPLY GetRecordProperties (e) AS record
WHERE record.PropertyName = 'p3' OR
      record.PropertyName = 'p2' OR
      record.PropertyName = 'p3'

You can use the following input file to test the Stream Analytics query on the Azure Management portal.

[
  {
    "Time": "2015-12-16T10:00:01.0000000Z",
    "Id": "01",
    "P1": 15,
    "P2": 30,
    "P3": 10,
  },
  {
    "Time": "2015-12-16T10:00:01.0000000Z",
    "Id": "02",
    "P1": 25,
    "P2": 40,
    "P3": 20,
  },
  {
    "Time": "2015-12-16T10:00:01.0000000Z",
    "Id": "03",
    "P1": 45,
    "P2": 18,
    "P3": 5,
  },
  {
    "Time": "2015-12-16T10:00:01.0000000Z",
    "Id": "04",
    "P1": 20,
    "P2": 23,
    "P3": 56,
  }
]

Running the query against the test file returns the following dataset:

Time Id Type Value
2015-12-16T10:00:01.0000000Z 01 P1 15
2015-12-16T10:00:01.0000000Z 01 P2 30
2015-12-16T10:00:01.0000000Z 01 P3 10
2015-12-16T10:00:01.0000000Z 02 P1 25
2015-12-16T10:00:01.0000000Z 02 P2 40
2015-12-16T10:00:01.0000000Z 02 P3 20
2015-12-16T10:00:01.0000000Z 03 P1 45
2015-12-16T10:00:01.0000000Z 03 P2 18
2015-12-16T10:00:01.0000000Z 03 P3 5
2015-12-16T10:00:01.0000000Z 04 P1 20
2015-12-16T10:00:01.0000000Z 04 P2 23
2015-12-16T10:00:01.0000000Z 04 P3 56
Viewing all 70 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>