Useful transactional replication stored procedure

The scenario is when your publication settings has the option
immediate_sync configured as false (means if you run the Snapshot Agent, the snapshot files will be created for the new articles only and also a new subscriber to the existing publication, the snapshot files only get created for the new subscription)

This option immediate_sync will prevent to generate all snapshot files again when add a new article in the replication, but it still need schema lock to check all articles modification.

If you have a large number of articles and a bunch of subscribers, you might face the situation you have to add a new article in your transactional replication and after run the snapshot the article is not propagating to the subscribers.

First, check your publication settings.

exec sp_helppublication 'PublicationName'
GO

If you are using the settings immediate_sync and allow_anonymous false you can run the procedure below to solve the issue that your new article didn’t propagate to your subscribers.

exec sp_refreshsubscriptions 'PublicationName'
GO



T-SQL script to get detailed login permissions

This script is very helpful when you need to know all the permissions a user have in detail.

 

SELECT distinct 
@@SERVERNAME as ServerName
, UserName as LoginName
, UserType as LoginType
, DatabaseUserName
, Role
, PermissionType
, PermissionState
, DatabaseName = db_name()
, ObjectName  -- can be an object or the entire database
, ObjectType
, ColumnName
--	, item -- used to check consistencies in each subquery
FROM (
SELECT  
    UserName = princ.name,
    UserType = CASE princ.type
                    WHEN 'S' THEN 'SQL'
                    WHEN 'U' THEN 'Windows'
					WHEN 'G' THEN 'Windows Group'
					WHEN 'R' THEN 'Database Role'
					ELSE princ.type 
                END,  
    DatabaseUserName = princ.name,       
    Role = princ.name,      
    PermissionType = perm.permission_name,       
    PermissionState = perm.state_desc,       
    ObjectType = obj.type_desc,
    ObjectName = coalesce(OBJECT_NAME(perm.major_id), db_name()),
    ColumnName = col.name,
	1 as item
FROM    
    sys.database_principals princ  
LEFT JOIN sys.login_token ulogin on princ.sid = ulogin.sid
LEFT JOIN sys.database_permissions perm ON perm.grantee_principal_id = princ.principal_id
LEFT JOIN sys.columns col ON col.object_id = perm.major_id AND col.column_id = perm.minor_id
LEFT JOIN sys.objects obj ON perm.major_id = obj.object_id
WHERE perm.permission_name <> 'CONNECT' 
UNION
SELECT  
    UserName =  memberprinc.name ,
    UserType = CASE memberprinc.type
                    WHEN 'S' THEN 'SQL'
                    WHEN 'U' THEN 'Windows'
					WHEN 'G' THEN 'Windows Group'
					WHEN 'R' THEN 'Database Role'
					else memberprinc.type 
                END, 
    DatabaseUserName = memberprinc.name,   
    Role = roleprinc.name,      
    PermissionType = perm.permission_name,       
    PermissionState = perm.state_desc,       
    ObjectType = obj.type_desc,
    ObjectName = db_name(),
    ColumnName = col.name,
	2 as item
FROM    
    sys.database_role_members members
INNER JOIN sys.database_principals roleprinc ON roleprinc.principal_id = members.role_principal_id
INNER JOIN sys.database_principals memberprinc ON memberprinc.principal_id = members.member_principal_id
LEFT JOIN sys.login_token ulogin on memberprinc.sid = ulogin.sid
LEFT JOIN sys.database_permissions perm ON perm.grantee_principal_id = roleprinc.principal_id
LEFT JOIN sys.columns col on col.object_id = perm.major_id AND col.column_id = perm.minor_id
LEFT JOIN sys.objects obj ON perm.major_id = obj.object_id
UNION
SELECT  
    UserName = roleprinc.name,
    UserType = 
	CASE roleprinc.type
                    WHEN 'S' THEN 'SQL'
                    WHEN 'U' THEN 'Windows'
					WHEN 'G' THEN 'Windows Group'
					WHEN 'R' THEN 'Database Role'
					ELSE roleprinc.type 
                END,  
    DatabaseUserName = roleprinc.name,
    Role = roleprinc.name,      
    PermissionType = perm.permission_name,       
    PermissionState = perm.state_desc,       
    ObjectType = obj.type_desc,
    ObjectName = coalesce(OBJECT_NAME(perm.major_id), db_name()),
    ColumnName = col.name,
	3 as item
FROM    
    sys.database_principals roleprinc 
LEFT JOIN sys.database_permissions perm ON perm.grantee_principal_id = roleprinc.principal_id
LEFT JOIN sys.columns col on col.object_id = perm.major_id AND col.column_id = perm.minor_id                   
LEFT JOIN sys.objects obj ON obj.object_id = perm.major_id
UNION
SELECT 
    UserName = princ.name collate Latin1_General_CI_AS,
    UserType = CASE princ.type
                    WHEN 'S' THEN 'SQL'
                    WHEN 'U' THEN 'Windows'
					WHEN 'G' THEN 'Windows Group'
					WHEN 'R' THEN 'Database Role'
					ELSE princ.type 
                END ,  
    DatabaseUserName = princ.name collate Latin1_General_CI_AS,
    Role =  CASE 
                WHEN logins.sysadmin = 1 THEN 'sysadmin'
                WHEN logins.securityadmin = 1 THEN 'securityadmin'
                WHEN logins.serveradmin = 1 THEN 'serveradmin'
                WHEN logins.setupadmin = 1 THEN 'setupadmin'
                WHEN logins.processadmin = 1 THEN 'processadmin'
                WHEN logins.diskadmin = 1 THEN 'diskadmin'
                WHEN logins.dbcreator = 1 THEN 'dbcreator'
                WHEN logins.bulkadmin = 1 THEN 'bulkadmin'
                ELSE 'Public' 
            END,
    PermissionType	= perm.permission_name,
    PermissionState = 'GRANT',
    ObjectType		= NULL,
    ObjectName		= princ.default_database_name,
    ColumnName		= NULL,
	4 as item
FROM sys.server_principals princ 
INNER JOIN sys.syslogins logins ON princ.sid = logins.sid 
LEFT JOIN sys.database_permissions perm ON perm.grantee_principal_id = princ.principal_id
WHERE princ.type  <> 'R' AND princ.name NOT LIKE '##%'
) P  
where (Role <> 'Public' or ObjectName = db_name())
ORDER BY
P.DatabaseUserName,
P.ObjectName,
P.ColumnName,
P.PermissionType,
P.PermissionState,
P.ObjectType

Method to update LOB data with minimal log

Use the .WRITE (expression, @Offset,@Length) clause to perform a partial or full update of varchar(max)nvarchar(max), and varbinary(max) data types.

For example, a partial update of a varchar(max) column might delete or modify only the first 200 characters of the column, whereas a full update would delete or modify all the data in the column. .WRITE updates that insert or append new data are minimally logged if the database recovery model is set to bulk-logged or simple.

UPDATE <table_or_view_name>
SET column_name.WRITE (expression, @offset, @length)
FROM <table_source>
WHERE <search_condition>

expression is the value that is copied to column_nameexpression must evaluate to or be able to be implicitly cast to the column_name type. If expression is set to NULL, @Length is ignored, and the value in column_nameis truncated at the specified @Offset.

@Offset is the starting point in the value of column_name at which expression is written. @Offset is a zero-based ordinal position, is bigint, and cannot be a negative number. If @Offset is NULL, the update operation appends expression at the end of the existing column_name value and @Length is ignored. If @Offset is greater than the length of the column_name value, the Database Engine returns an error. If @Offset plus @Length exceeds the end of the underlying value in the column, the deletion occurs up to the last character of the value. If @Offset plus LEN(expression) is greater than the underlying declared size, an error is raised.

@Length is the length of the section in the column, starting from @Offset, that is replaced by expression@Length is bigint and cannot be a negative number. If @Length is NULL, the update operation removes all data from @Offset to the end of the column_name value.

The regular update statement results in overwriting the entire string using full logging and it’s very inefficient when dealing with large value updates.

Note: The update using WRITE method will fail if the value is null.

More information about performance in my tip on MSSQLTips

 

 

Execution runtime information for developers ;)

Developers should use this statistics to get more information about what they just did and pay attention to high physical or logical read values.

SQL Server provides commands that are used to return query statistics like SET STATISTICS IO, TIME.

The IO command return information about physical, logical for tables referenced in the query and also worktables, which means tables in tempdb for the duration of the query and are removed automatically when it has finished the operation.

Using statistics time will show the parse, compile time for the query and the actual execution time after the query results. It’s useful to measure the amount of time a query takes to execute from end to end.

How to install SQL Server SQLPS Powershell module

SQLPS module to Powershell helps to do so many things on SQL Server without SSMS and the main objective is to automate tasks for your environment.

This module isn’t installed by default, so there are a couple of steps before use it. First, install the module and the simple cmdlet install-module SQLPS won’t work.

To install the module you need install a couple of things (if you have SQL Server instance there is already installed).

The installation is quite simple and isn’t necessary to restart the system. There are 3 files necessary and need to install in the sequence below.

With all installed let’s try to import the module on powershell.

2018-08-01_20-36-01

Actually that didn’t work (yet). In the error message shows it’s not allowed to run scripts in my system.

Next step is to change the execution policy to run scripts in the system. Let’s try it using the cmdlets Get-ExecutionPolicy to know the actual value and Set-ExecutionPolicy to change it.  ( more information )

2018-08-01_20-45-23

Ok, second error and this time because I’m trying to change to RemoteSigned and only as administrator I can change it.

2018-08-01_20-54-28

Running as administrator the local machine execution policy has changed and the SQLPS module can be imported and use the cmdlets.

2018-08-01_21-00-16

How update works in transactional replication?

In a post about how update works I showed what happened when run an update with the same values. SQL Server is smart enough to see that and not changing anything and register minimum log.

Another day I saw a comment to avoid updating records when none of the values are changing.

Avoid updating records when none of the values are changing. This still does a write and, if the table is in replication or change tracking, still causes the row to be propagated out to other servers. If you are updating a potentially large number of records, make sure to only update the ones where the new value doesn’t equal the old value.

Let’s see how the update behavior when update 10 millions rows but without any change and see if will be any row propagation to another server.

objectexplorer

My Lab contains two servers and a demo database replicated from SQL01 to SQL02.

It’s configured a transaction replication to send the articles and I’m using a table created with the code:

CREATE TABLE tblMillionsRows (
    id BIGINT NOT NULL IDENTITY PRIMARY KEY,
    largeColumn NVARCHAR(MAX) DEFAULT REPLICATE('TESTE',100),
    smallColumn NVARCHAR(150) DEFAULT 'DEMO',
    dateColumn DATETIME
        DEFAULT GETDATE()
);
GO

Inserting rows:

INSERT INTO dbo.tblMillionsRows (
    largeColumn,
    smallColumn,
    dateColumn
)
DEFAULT VALUES
GO 2000000

It will take a while to insert all rows. The next step is add this table in the replication.

DECLARE @publication    AS sysname;
DECLARE @table AS sysname;
DECLARE @filterclause AS nvarchar(500);
DECLARE @filtername AS nvarchar(386);
DECLARE @schemaowner AS sysname;
SET @publication = N'SQL01_demo_tb01'; 
SET @table = N'tblMillionsRows';
SET @schemaowner = N'dbo';

EXEC sp_addarticle 
	@publication = @publication, 
	@article = @table, 
	@source_object = @table,
	@source_owner = @schemaowner, 
	@schema_option = 0x80030F3,
	@vertical_partition = N'true', 
	@type = N'logbased',
	@filter_clause = @filterclause;

EXEC sp_articlecolumn 
	@publication = @publication, 
	@article = @table;

EXEC sp_startpublication_snapshot 
	@publication = 'SQL01_demo_tb01', 
	@publisher =  'SQL01'   

With all set, let’s do some tests. First updating the heap table and seeing if something is replicated.

UPDATE tblMillionsRows SET smallColumn = N'DEMO' WHERE id < 10000
GO

After the update, I ran the script below to see the transactions and commands available to replicate and nothing changed.

SELECT * FROM distribution.dbo.MSrepl_Commands 
SELECT * FROM distribution.dbo.MSrepl_Transactions 
EXEC sp_browsereplcmds 

But, when I change one row running the update below I could see a new command to replicate in the another server.

UPDATE tblMillionsRows SET smallColumn = N'DEMO1' WHERE id < 10000
GO

replicationcommands

In conclusion, SQL Server won’t replicate updates that don’t change the value. You can see more about updates in my post.

Be careful when doing updates, because if the table is replicated and you change millions rows the transaction replication will create one command for each updated row.

 

Database migration methods

migration-logo-2389341When thinking about migration the most common issue is downtime. There are some methods that require downtime like backup/restore or less downtime using Log Shipping.

You can choose several different methods to migrate your database.  For example, methods that require downtime include:

  • Backup the database, manually copy the backup file to the machine and then restore the database there. This method is simple and the use of compression minimizes the time that is required.
  • Perform a backup to an Azure Blob storage account, and then restore the database from that backup. This method removes the necessity to manually copy the backup file.
  • Detach the database, copy the files to an Azure Blob storage and then attach them to the SQL Server instance. Use this method if you plan to store database files in Azure Blob storage permanently instead of on a hard disk.
  • Log Shipping, use a backup/restore process and the downtime depends on the transaction log file backup copying a small file in the process. This method is simple and is necessary a connection between the machines and permission to copy the backup files.

Methods that not require downtime:

  • AlwaysOn, adding a new secondary replica after replication has completed, you can failover to make the machine the primary replica. For Azure virtual machine use the Add Azure Replica Wizard.
  • Transactional Replication will minimize downtime, but don’t have an AlwaysOn deployment in your SQL Server system.

If you need help to decide which method is better for your scenario, feel free to contact me.

SqlPackage a tool to import/export SQL Server and Azure SQL DB

repair_database-512SqlPackage is a command-line utility that you can use for exporting and importing operations in both on-premises SQL Server databases and in cloud databases. SqlPackage supports the following operations:

  • Extract. Creates a database snapshot DACPAC file from a SQL Server database or from Azure SQL Database.
  • Publish. Updates the schema in a live database to match the schema in a DACPAC
    file. If the database does not exist on the destination server, the publish operation
    creates it.
  • Export. Exports both schema and data from a SQL Server database or from Azure SQL Database into a BACPAC file.
  • Import. Imports the schema and data from a BACPAC into a new database.
  • DeployReport. Creates an XML report that describes the changes that would be made by a publish operation.
  • DriftReport. Creates an XML report of the changes that have been made to a registered database.
  • Script. Creates a Transact-SQL script that you can use to update the schema of a target database to match the schema of a source database.
    Use the /Action: or /a parameter to specify which action to execute.

SqlPackage example to import from bacpac file to Azure
sqlpackage.exe /Action:Import /tsn:tcp:.database.windows.net,1433 /tdn: /tu: /tp: /sf: /p:DatabaseEdition=Premium /p:DatabaseServiceObjective=P4 /p:Storage=File

More about SqlPackage parameters:
https://msdn.microsoft.com/en-us/library/hh550080%28v=vs.103%29.aspx?f=255&MSPPError=-2147217396

Moving the msdb, model, and tempdb databases files

All system databases, except the resource database, can be moved to new locations to help balance I/O load.

To move the msdb, model, and tempdb databases, perform the following steps:

  • For each file to be moved, execute the ALTER DATABASE … MODIFY FILE statement.
  • Stop the instance of SQL Server.
  • Move the files to the new location (this step is not necessary for tempdb, as its files are recreated automatically on startup).
  • Restart the instance of SQL Server.

The process for moving the master database is different from the process for other databases. To move the master database, perform the following steps:

  • Open SQL Server Configuration Manager.
  • In the SQL Server Services node, right-click the instance of SQL Server, click Properties, and then click the Startup Parameters tab.
  • Edit the Startup Parameters values to point to the planned location for the master database data (-d parameter) and log (-l parameter) files.
  • Stop the instance of SQL Server.
  • Move the master.mdf and mastlog.ldf files to the new location.
  • Restart the instance of SQL Server