Category Archives: Uncategorized

Who Says C# Interfaces Can’t Have Implementations?

This post is the seventh installment of the 2017 C# Advent Calendar operated by Matthew Groves. Thanks for letting me participate!

Overview

According to the C# Programming Guide, an interface cannot contain an implementation:

When a class or struct implements an interface, the class or struct must provide an implementation for all of the members that the interface defines. The interface itself provides no functionality that a class or struct can inherit in the way that it can inherit base class functionality.

There is a new feature in the works for C# 8 that will provide default interface implementations. However, I’m known for my impatience, and I don’t want to wait. The fact is, interfaces can have implementations in C# today.

Note: At this point, I’d like to give a shout out to the ASP.NET Core team. I can’t really take credit for this concept, I learned it by digging through the ASP.NET Core source on GitHub.

The Magic of Extension Methods

When I first encountered extension methods, I viewed them as a helpful way to add functionality to third-party classes. Adding additional logic to System.DateTime is helpful, right? However, another powerful use case is to create extensions for your own interfaces. When you write an extension method for an interface, you’re actually providing it with implementation. Sure, the consumer may need an extra using statement at the top of the file, but it’s still an implementation! Taking this approach has several advantages:

  • Extension methods can be safely added to interfaces without risking backward compatibility for consumers (especially if they’re in a separate namespace).
  • The interface becomes easier to implement there are few methods on the interface itself which must be implemented.
  • There is guaranteed consistency in method implementation across all class implementations.

There is, however, one rule you must follow. The extension method must only use members exposed by the interface being extended. While hacks like typecasting back to a particular implementation will work, it risks the purity of the implementation. What happens when the extension is used against a different implementation of the interface? Generally speaking, this rule should be broken only for performance optimizations and should have a fallback that works against any implementation.

The Elephant In The Room

Perhaps the best-known example of extension methods in .NET is LINQ. The vast majority of the API surface of LINQ is extension methods on the IEnumerable<T> and IQueryable<T> interfaces.

LINQ was added in .NET 3.0. If the LINQ team had decided to add the LINQ methods directly to IEnumerable<T>, this would have constituted a breaking change. Every class which implemented IEnumerable<T> would have been broken upon upgrading to .NET 3.0 until the entire suite of LINQ methods was implemented. Just imagine implementing that across every collection class. C# developers everywhere would have chased the LINQ team around with pitchforks.

Instead, the LINQ team implemented their system as extension methods. The result is that .NET 3.0 and LINQ were fully backward compatible with libraries written in .NET 2.0. Additionally, a whole lot of headaches and duplicated code were avoided. And it was done by, effectively, adding implementations to interfaces.

So, Teach Me The Magic

Writing extension methods is actually relatively painless. They’re basically just syntactic sugar layered on top of static methods.

Let’s start with this interface:

namespace MyNamespace
{
    public interface IMyInterface
    {
        IList<int> Values { get; set; }
    }
}

Now, let’s add an extension method that returns the count of all values in the list greater than a threshold.

namespace MyNamespace
{
    public static class MyInterfaceExtensions
    {
        public static int CountGreaterThan(this IMyInterface myInterface, int threshold)
        {
            return Values?.Where(p => p > threshold).Count() ?? 0;
        }
    }
}

The extension method can be consumed like this:

using MyNamespace;

// ...

public void DoSomething()
{
    var myImplementation = new MyInterfaceImplementation();

    // Note that there's no typecast to IMyInterface required
    var countGreaterThanFive = myImplementation.CountGreaterThan(5);    
}

// ...

There are four key pieces to the puzzle:

  1. MyInterfaceExtensions and CountGreaterThan are both public (though they could be internal if you want to use them only within your library).
  2. MyInterfaceExtensions and CountGreaterThan are both static.
  3. The first parameter of CountGreaterThan is the interface and is preceded by the “this” keyword.
  4. The file where DoSomething is declared includes a using statement for the namespace where the extensions are declared (Visual Studio will help by adding this automatically).

Note: There are many different approaches for code organization surrounding these extensions methods. Some teams may prefer them in the same file, others in a separate file in the same folder, and others may want extensions in a separate folder/namespace. Just be sure your team picks a pattern and sticks with it. For teams that choose separate files, including comments on the interface that point to the extension files is a good idea.

Making Mocks Easy, One Extension Method At A Time

My favorite use case is for supporting unit tests, especially when I’m providing lots of method overloads. The reason I point out this specific use case it to show that extension methods can be very useful outside of developing reusable libraries. Almost all development today requires unit testing.

When writing unit tests, it’s often best to test against mocks of interfaces, rather than real implementations. The reasons why are beyond the scope of this post, so just trust me on this. I used to think it was silly until I learned the hard way it wasn’t. Creating mocks can be greatly simplified by adding implementation to interfaces.

For example, imagine an interface named ICartItem which represents an item in an online shopping cart. It needs several different methods to support changing the quantity in the cart.

public interface ICartItem
{
    int Quantity { get; set; }
    void IncrementQuantity();
    void IncrementQuantity(int delta);
    void DecrementQuantity();
}

The Quantity property can be set directly if the user enters a value, but there are also up and down arrows on the UI which tie to the IncrementQuantity and DecrementQuantity methods.

Continuing the example, ICartItem is used by a CartManager class that manages the shopping cart, and CartManager contains lots of business logic that requires unit tests. These tests require mock implementations of ICartItem. Since the interface defines a property and three methods, every mock needs to include four implementations. However, the three methods have a very basic function, and their implementation should be consistent across all classes that implement the interface. Additionally, they can be written to be supported by the Quantity property.

public interface ICartItem
{
    int Quantity { get; set; }
}

public static class CartItemExtensions
{
    public static void IncrementQuantity(this ICartItem cartItem)
    {
        cartItem.IncrementQuantity(1);
    }

    public static void IncrementQuantity(this ICartItem cartItem, int delta)
    {
        cartItem.Quantity += delta;
    }

    public static void DecrementQuantity(this ICartItem cartItem)
    {
        cartItem.IncrementQuantity(-1);
    }
}

public class CartManagerTests
{
    [Fact]
    public void Some_Test()
    {
        // Arrange
    
        // Note: This example uses Moq (https://www.nuget.org/packages/Moq/), your syntax may vary
        var mockCartItem = new Mock<ICartItem>();
        mockCartItem.SetupAllProperties();

        // ...
    }
}

Now every mock of ICartItem is greatly simplified, and they will have support for the IncrementQuantity and DecrementQuantity methods without any specific code added to the tests.

More Advanced, Real-world Examples

There are lots of great examples of this pattern found throughout the ASP.NET Core source code on GitHub. Here is a small selection:

  • ConsoleLoggerFactoryExtensions – Note how each extension implements overloads by forwarding to other extensions with more detailed parameters, until finally a call to ILoggerFactory.AddLogger(ILoggerProvider) is reached.
  • ServiceCollectionServiceExtensions – A veritable avalanche of extensions, which eventually call the most powerful method, IServiceCollection.Add(ServiceDescriptor).
  • ResponseCachingExtensions – In this case, a descendent assembly rather than the original assembly adds an extension to the interface. This helps reduce the inclusion of unnecessary dependencies, users only add the packages they need to the application. Yet, ease-of-use is maintained by extending the original interface.

Summary

That’s how to add implementations to interfaces in a nutshell. It’s a very powerful tool but is especially useful for writing shared libraries as well as for writing code that is easy to unit test. Key use cases to watch out for include:

  • Overloads where simpler methods are just forwarding the requests to methods with more parameters and filling in default values. The simpler methods can be extensions.
  • Helper methods that support more common uses cases by forwarding method calls to more powerful but less frequently used methods.
  • Other methods where the implementation will always be the same for every class that implements the interface, and which are supported by other members of the interface.

Docker Login For Amazon AWS ECR Using Windows Powershell

My recent studies in .Net Core have lead me to the new world of Docker (new for .Net developers, anyway).  The idea of developing low-cost microservices while still working using  my favorite development platform is very exciting.  In the process, I began using the Amazon AWS Docker platform, Elastic Container Services (ECS).

I quickly found that documentation for using ECS from Windows is a bit scarce.  In particular, the Powershell tools are missing a pretty key helper method, get-login.  Calling “aws ecr get-login” on a Linux box delivers you a complete “docker login” command for authenticating to the Elastic Container Registry (ECR).  There is currently no such helper for Windows.  At least, not that I can find, someone correct me if I’m just missing  it.

Instead, I’ve done a bit of digging and found how to authenticate programatically.  From that, I’ve created the helper code below for reuse.

# Get the authorization token
$token = Get-ECRAuthorizationToken -Region us-east-1 -AccessKey your-access-key -SecretKey your-secret-key
# Split the token into username and password segments
$tokenSegments = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($token.AuthorizationToken)).Split(":")
# Get the host name without https, as this can confuse some Windows machines 
$hostName = (New-Object System.Uri $token.ProxyEndpoint).DnsSafeHost
# Perform login
docker login -u $($tokenSegments[0]) -p $($tokenSegments[1]) -e none $hostName

This login should then be valid for 12 hours. Note that if you use your own Region, AccessKey, and SecretKey on the first line. Alternatively, you could use Set-DefaultAWSRegion and Set-AWSCredentials to store them in your Powershell session. If you’re on a build server running in AWS you could also use IAM roles to grant access directly to the build server.

Update 2/8/2017

You can add the section below to your PowerShell profile to add an easy to use cmdlet. To install:

  1. Run “notepad $PROFILE”
  2. Paste the code below into the file and save
  3. Run “. $PROFILE” or restart Powershell
  4. Run “Auth-ECR your-access-key your-secretkey”.
function Auth-ECR {
	[CmdletBinding()]
	param (
		[parameter(Mandatory=$true, Position=0)]
		[string]
		$AccessKey,
		
		[parameter(Mandatory=$true, Position=1)]
		[string]
		$SecretKey,
		
		[parameter()]
		[string]
		$Region = "us-east-1"
	)
	
	# Get the authorization token
	$token = Get-ECRAuthorizationToken -AccessKey $AccessKey -SecretKey $SecretKey -Region $Region `
		-ErrorAction Stop
	
	# Split the token into username and password segments
	$tokenSegments = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($token.AuthorizationToken)).Split(":")
	
	# Get the host name without https, as this can confuse some Windows machines 
	$hostName = (New-Object System.Uri $token.ProxyEndpoint).DnsSafeHost
	
	# Perform login
	docker login -u $($tokenSegments[0]) -p $($tokenSegments[1]) -e none $hostName
}

Note that this script defaults to using the us-east-1 region. You can change the default in your profile, or use “-Region” on the command line.

Rebuild All Couchbase N1QL Indexes After Restore

Overview

When restoring a Couchbase cluster from a backup, the restore utility is kind enough to recreate the N1QL indexes for you.  To improve speed and efficiency, the indexes are only created, they are not built automatically.  Before they can be used, you must execute a build command such as this:

BUILD INDEX ON BucketName (IndexName1, IndexName2, IndexName3)

It is important that this query be issued as a single command for all indexes on a bucket.  This allows the indexes to be built together, resulting in only one read of the data from the cluster while building multiple indexes.

The Problem

Unfortunately, N1QL doesn’t currently offer a wildcard option, so there is no quick way to rebuild all indexes without typing the all of their names.  If you’re trying to script your environments for development or QA this can be particularly problematic, as the list of indexes may not be constant. It could also be a problem when creating scripts for a disaster recovery plan.

The Solution

If you’re running on Linux (you should be for production clusters), the solution is to use this script:

#!/bin/sh

QUERY_HOST=http://localhost:8091

for i in "BucketName1" "BucketName2" "BucketName3"
do
  /opt/couchbase/bin/cbq -e $QUERY_HOST -s="$( \
    echo "BUILD INDEX ON $i (\`$( \
      /opt/couchbase/bin/cbq -e $QUERY_HOST -q=true -s="SELECT name FROM system:indexes where keyspace_id = '$i' AND state = 'deferred'" | \
        sed -n -e '/{/,$p' | \
        jq -r '.results[].name' | \
        sed ':a;/.*/{N;s/\n/\`,\`/;ba}')\`)")"

  # Wait for completion
  until [ `/opt/couchbase/bin/cbq -e $QUERY_HOST -q=true -s="SELECT COUNT(*) as unbuilt FROM system:indexes WHERE keyspace_id = '$i' AND state <> 'online'" | sed -n -e '/{/,$p' | jq -r '.results[].unbuilt'` -eq 0 ];
  do
    sleep 5
  done
done

Replace the QUERY_HOST parameter as needed to point to a query node, and replace the BucketName values with the names of the buckets where indexes must be built.  It will process each bucket one at a time, waiting for the indexes to be built before continuing to the next bucket.

The only depedency is the jq utility, which is a command line JSON parser. On Ubuntu, this can be installed via:

sudo apt-get install -y jq

The script isn’t pretty, but it gets the job done. Hopefully N1QL will get a wildcard BUILD INDEX command in the future.

Note: Revised 9/15/2016 to better strip header information from the query output before the JSON object. Previously it only stripped the first line, now it strips all lines until it encounters the curly brace starting the JSON result.

Testing an SDK for Async/Await SynchronizationContext Deadlocks

Overview

The purpose of this post is to explain how to write unit and/or integration tests that ensure you don’t have synchronization context deadlocks in an SDK. A very detailed explanation of the problem and the solution can be found here. However, before explaining how to write the tests, I will give a brief summary of the problem and the solutions. Then we’ll get into how to test to make sure that the solution is correctly implemented now and won’t regress in the future.

The Problem

One of the common API developer pitfalls in asynchronous development with the .Net TPL is deadlocks. Most commonly, this is caused by SDK consumers using your asynchronous SDK in a synchronous manner. For example:

public ActionResult Index()
{
    // Note to consumers: Where possible DON'T DO THIS.  Just make your MVC action async, it works MUCH better.
    var data = api.SomeActionAsync().Result;

    return View(data);
}

The example above is typically a problem because MVC runs actions inside a SynchronizationContext. The MVC synchronization context prevents more than one thread from operating within the context simultaneously. The process flow works as follows:

  1. Thread A runs the action above, and requests “SomeActionAsync”
  2. Thread A blocks waiting on “Result” for “SomeActionAsync”
  3. Thread B begins processing the work for “SomeActionAsync”
  4. At some point, Thread B attempts to synchronize onto the SynchronizationContext from the MVC action, and is blocked waiting for Thread A to release it.
  5. We have a deadlock!

So why does Step 4 above happen? I know I didn’t write any code that requested that the MVC SynchronizationContext be used! Well, if you are using the async/await programming model, you’re doing so without even knowing it.

public async Task<SomeResult> SomeActionAsync()
{
    // do some work

    var temp = await obj.SomeOtherActionAsync();

    // do more work

    return result;
}

The await above automatically tries to synchronize with the SynchronizationContext. This is actually pretty important for writing async MVC actions. When an async method is awaited in the action, once it’s complete we really want the remainder of the action to run within the SynchronizationContext. But within our SDK, we probably don’t want that to happen because it usually doesn’t have any value to us.

The Solution

There are two solutions to this problem.  The good one, and the bad one.

The Bad Solution: Require that SDK consumers use the SDK asynchronously

I don’t like this solution because it’s difficult to ensure that consumers use it correctly. It’s also a barrier to SDK use. I really believe that consumers should be given the option to consume the SDK how they like, even if it’s a bad way to consume it. The TPL provides a .Result call, so where possible we should make it work.

public async Task<ActionResult> Index()
{
    var data = await api.SomeActionAsync();

    return View(data);
}

An important note for SDK consumers, though. For you, this is the Good Solution. You should always use asynchronous API calls in an asynchronous manner whenever possible. This is only a Bad Solution if the SDK developer is assuming that you always do this.

The Good Solution: Fix the problem on the SDK side

Thankfully, the TPL provides us with a simple workaround in the SDK, using ConfigureAwait(false). Calling this routine before awaiting a task will cause it to ignore the SynchronizationContext.

public async Task<SomeResult> SomeActionAsync()
{
    // do some work

    var temp = await obj.SomeOtherActionAsync().ConfigureAwait(false);

    // do more work

    return result;
}

Most Importantly, The Test

The problem with the Good Solution is that it requires you to place a lot of ConfigureAwait(false) calls throughout the SDK. This can be cumbersome and easy to forget. Though there is a Resharper plugin to help, ConfigureAwaitHelper.

Any good SDK comes with a battery of unit and integration tests. So the trick is to add tests to the SDK that will ensure that we don’t forget any ConfigureAwait(false) calls. So how do we add a test or tests that ensure we called ConfigureAwait(false)?

The trick is understanding how the SynchronizationContext works. Anytime it is used, a call will be made to either it’s Send or Post method. So, all we need to do is make a mock and ensure that these methods never get called:

[Test]
public void Test_SomeActionNoDeadlock()
{
    // Arrange
    var context = new Mock<SynchronizationContext>
    {
        CallBase = true
    };

    // Do other arrange actions here

    SynchronizationContext.SetSynchronizationContext(context.Object);
    try
    {
        // Act
        class.SomeActionAsync().Wait();

        // Assert

        // If the method is incorrectly awaiting on the current SynchronizationContext
        // We will see calls to Post or Send on the mock

        context.Verify(m => m.Post(It.IsAny<SendOrPostCallback>(), It.IsAny<object>()), Times.Never);
        context.Verify(m => m.Send(It.IsAny<SendOrPostCallback>(), It.IsAny<object>()), Times.Never);
    }
    finally
    {
        SynchronizationContext.SetSynchronizationContext(null);
    }
}

This example uses NUnit and Moq, but it should work just as well with other testing frameworks. Now we have a way to guarantee that ConfigureAwait(false) was used throughout the SDK method, so long as we get 100% test coverage through the logical paths in the method.

Of course, you may ask “Why do I need this?  I just looked at the code, I’m always calling ConfigureAwait(false)!” The answer is preventing regressions. You might have remembered today, but next month when you’re making a change it’s very easy to forget. This test is your fallback plan in case you make a mistake in the future.

Couchbase and N1QL Security

As a developer at CenterEdge Software, I’ve had a lot of cause to use Couchbase as our NoSQL databasing platform over the last few years.  I’ve gotten really excited about the potential of the new Couchbase query language in Couchbase 4.0, called N1QL.  So excited that I’ve spent a lot of time contributing to the Linq2Couchbase library, which allows developers to use LINQ to transparently create N1QL queries.

In doing work with N1QL, I quickly realized that it may have some of the same security concerns as SQL.  In particular, N1QL injection could be a new surface area for attack in Couchbase 4.0.  That’s what I call the N1QL equivalent of SQL injection.  I found that while the risks are lower in N1QL than in SQL, there are still some areas that need to be addressed by application developers using Couchbase.

As a result, I did some research and recently wrote a guest post on N1QL security for Couchbase users.  It researches possible N1QL injection security concerns, then goes into how to protect your applications when using N1QL.

http://blog.couchbase.com/2015/september/couchbase-and-n1ql-security-centeredgesoftware