Sonntag, 19. November 2017

noderunner.exe is eating all your memory?

Just as a note to myself: if you have to work with sharepoint you might notice a very high memory consumption from several noderunner.exe-processes.

There alerady exists an explanation of how to reduce this, here are the crucial steps:


  • Open a Sharepoint Management Shell
  • Set-SPEnterpriseSearchService -PerformanceLevel Reduced
  • Get-SPEnterpriseSearchService
    Should show "Performance Level: Reduced" now
  • Open C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config in Notepad
  • Update <nodeRunnerSettings memoryLimitMegabytes=”0″ /> to say 100
  • Restart SharePoint Search Host Controller Service

And you are done.


Dienstag, 24. Oktober 2017

.NET Core Web API: Returning a file from an OutputFormatter

Let's say you want to return a csv file from your web API. To return a file from a controller action is as easy as writing

return File(stream, "text/csv");

It is not that eays if you are using an output formatter to be able to return either json or a file from the same controller method based on the Accept header. Then your controller looks like this:

return Ok(data);

And you need to register your output formatter with MVC:

public void ConfigureServices(IServiceCollection services)
{
    // Add framework services.
    services
        .AddMvc(options =>
            {
                options.RespectBrowserAcceptHeader = true;
                options.OutputFormatters.Add(new CsvOutputFormatter());
            });
};

Within your formatter you need to render your data to Csv. You can easily use a NuGet package for this purpose. Writing the formatter is easy then:

public class ApetitoArticleCsvOutputFormatter : OutputFormatter
{

    public ApetitoArticleCsvOutputFormatter()
    {
        ContentType = "text/csv";
        SupportedMediaTypes.Add(MediaTypeHeaderValue.Parse("text/csv"));
    }
    
    public string ContentType { get; private set; }

    public override async Task WriteResponseBodyAsync(OutputFormatterWriteContext context)
    {

        var articles = context.Object as IEnumerable<Article>;

        var response = context.HttpContext.Response;
        response.Headers.Add("Content-Disposition", "attachment; filename=export.csv");

        using (var writer = context.WriterFactory(response.Body, Encoding.UTF8))
        {
            var csv = new CsvWriter(writer);
            csv.WriteRecords(articles);

            await writer.FlushAsync();

            
        }
    }
    
}


You just get your data from context.Object, get the Repsones object and a writer to the body stream and let CsvHelper write your data to this stream. Usually you will get the data being send back as text, for example when using swagger. But we want to make the browser save a file from the csv we requested.

The bold line does the trick: simply add a Content-Disposition header with value attachment and the name of you file. Now the browser saves the file, or swagger shows you a link to start the download.


Dienstag, 4. April 2017

Team Build with Remote Powershell Cross Domain

I ran into a series of issues when trying to establish a release pipeline in TFS where the build agent is located in the company domain whereas the target server is inside a DMZ-domain. I tried to run the "Run Powershell on Remote Machine" from the company domain build agent computer with a target machine located in the DMZ domain. I do not memorize all the errors I got in detail, but they were all around "WinRM, Could not process request, Kerboros, No authentication Server, Host not found".

Basically the problem comes down to open a remote powershell session. So if this succeeds when logged in to the company domain build agent computer and you connect to srv.mydmz.de being the target server in the dmz-domain:

Enter-PSSession 
    -ComputerName "srv.mydmz.de" 
    -Credential mydmz\username

then your build / release Task "Run Powershell on Remote Machine" should succeed as well. This is useful for testing purposes because you do not need to create a release definition upfront and create a release every time you try to get things up and running.

I found the steps to solve my problem in a blog post from Christopher Hunt, but I want to stress out on thing that I got wrong from many other blog posts providing the same solution.

The solution is rather simple. On the build agent computer and on the target DMZ computer run: 

WinRM Quickconfig 

Then, log in to the build agent computer and run this from an elevated command prompt:

Set-Item wsman:\localhost\Client\TrustedHosts -value "srv.mydmz.de" 

This adds the target server located in the DMZ as trusted host on the company domain joined build agent computer. Then the above command to open a remote powershell succeeded for me where it formerly failed. So a release definition like this should work then if the build agent computer is configured as stated above:

Release Definition executing a powershell across domains


So, call me dumb, but here is the thing I always got wrong until now: you have to add the DMZ-Server as trusted host on the company domain joined server, not the other way round.

To me it appeared more logical that the computer being called (the DMZ-server), i. e. where the remote powershell executes stuff, should trust the computer calling it (the company domain joined server). So I repeatedly tried the Set-Item-Command on the DMZ-Server setting the domain joined build agent computer as the trusted host.

Now that my incompetence in this case is revealed, maybe it might save others some time :-)

Montag, 23. Januar 2017

Handling child collectins in Entity Framework Code First

There are a lot of blog posts out there on the issue of removing items from child collections in Entity Framework.

The Problem

The Problem is, that when you remove an item from a child collection of an entity, EF just sets the foreign key null in the child collections table. It does not delete the item. While this means that the item does no longer appear in your collection, it is not a satisfying situation as it leaves orphaned records around.

Possible solutions and problems with the solutions

There are three possible solutions to the problem:
  1. Explicitly remove the child
  2. Identifying Relationships
  3. SaveChanges
The first one means, that whenever you remove something from a child collection you remove it from the according DbSet-Property of your context as well. This is obviously a bad design, becaus for child collection I do not want a DbSet collection of its own.
The second one means that you need to make the primary key of the parent part of the primary key of the child. This is bad because I need to change my model and bloat it with unnecessary properties - even worse these properties merely contain technical database stuff.  
The third one comes close to a good solution, but not the way I like to handle it. It requires to override the SaveChanges method of the context and handle deleting orphans there. This is the best solution so far because it tackles the problem close to its origin: inside the technical EF code stuff. But almost any implementations on the web tend to do it in a way that comes close to solution 2: the have kind of a navigation propertey in the child that points to the parent and that can be checked for null. Others suggest using domain events, which is generally a great concept but it feels weired to introduce it to solve a infrastructural problem.

My context

So here is my context, the scenario was from a coding dojo we did. It is a very database focused and very simplified implementation of a shopping cart:

public class Basket
{
    public Guid Id { get; set; }
    public Guid CustomerId { get; set; }
    public virtual IList<BasketItem> Positions { get; set; }
}

public class BasketItem
{
    public int Id { get; set; }
    public string ArticleName { get; set; }
    public decimal Amount { get; set; }
    public decimal Price { get; set; }
}

public class BasketContext : DbContext
{
    public BasketContext() : base("BasketContext") 
    {
    }
    
    public DbSet<Basket> Baskets { get; set; }
}

That is it. You see that Basket has a Guid-Id while BasketItem has an int-Id. This reflects the fact that I consider Basket to be an Aggregate Root on the level of my domain model, while Basket Item is just a contained entity, that does not have an id that is relevant outside of its containing basket. The need for the id is just for the sake of a relational database.
So, when searching for a solution to my problem, I made the following premise, stated loud and clear:
"I will try as hard as I can (very very hard) to never change my model just for the sake of a relational database!"
Having said that, all solutions I found on the web are not for me. Because as my requirements were, my model has no need for a Basket property, or even worse a BasketId property, on the BasketItem. This renders the above options 2 and 3 useless. Option 1 is useless as it requires me to add a DbSet for BasketItems, which is not necessary as they dont get queried directly.

My (prototype) solution

The following solution is not production ready. It is the solution I found in a late-at-the-day hacking session after the dojo that brought the problem to the surface. It ignores edge cases. It is not tested well. Keep that in mind, and dont say I did not warn you.
The solution follows pattern number three from the above solutions, overriding SaveChanges(). But it does not require to fiddle around with your model. My idea was: if EF knows that it has to set a value in table to null, it must be able to find out for me as well.

public override int SaveChanges()
{
    var objectContext = ((IObjectContextAdapter) this).ObjectContext;
    
    objectContext.DetectChanges();

    var deletedThings =
        objectContext.ObjectStateManager.GetObjectStateEntries(EntityState.Deleted).ToList();

   
    foreach (var  deletedThing in deletedThings)
    {
        if (deletedThing.IsRelationship)
        {
            var deletedKey = deletedThing.OriginalValues[1] as EntityKey;
            var entityToDelete = objectContext.GetObjectByKey(deletedKey);
            objectContext.DeleteObject(entityToDelete);
        }
    }

    return base.SaveChanges();
}

This can not be done using the DbContext-Api, but needs to be done with the ObjectContext-Api, so we have to obtain it using the IObjectContextAdapter interface. We need to call DetectChanges, otherwise the deletedThings are empty. In case of an orphaned child entry, the deletedThings contain entries for relationships (deletedThing.IsRelationship yields true). In that case we can find the ends of the relationship in the OriginalValues. A two element array where index 0 points to the parent (the basekt in the example) and index 1 points to the child (the BasketItem). By "points to" I mean that the OriginalValues contains an EntityKey-object identifying the object in question. So using GetObjectByKey(deletedKey) we can load the orpahned child. We must delete it using DeleteObject(entityToDelete) because there is no explicit EntitySet holding it.

I hope someone might find it useful.

Dienstag, 3. Januar 2017

Performance Profiling WCF Service running in WCF Test Client

I know the topic itself sounds a bit old school, but despite what many of my colleagues say, I think WCF is still alive and you might encounter it along your way. Furthermore I found it surprisingly hard to get the configration right to use Visual Studio Performance Profiling on a WCF Service that is running in the WCF Test Client - and is therefore not (yet) deployed to IIS. I mean the kind of WCF Application that you can start by simply hitting F5 and get going using the test client.

So as a note to myselft and hopefully as a help for someone else, here ist how I did it.

Choose Analyze | Performance Profiler (Alt + F2 using the standard shortcuts). Then go through the Performance Wizard.

Page 1 of 4: Specify the profiling method

Choose Instrumentation.

Page 2 of 4: Choose the modules to profile using the instrumentation method

Choose One or more available projects and select the WCF Service Project.

Page 3 of 4: These are things you may want to specify about the non-launchable project(s) you are going to profile

This is the tricky one. The thing is, to use the test client you must not start the test client. You must start the WCF Test Service Host. So set the Executeable path to

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\WcfSvcHost.exe

Modyfiy the path as needed for your Visual Studio version. As Command-line arguments pass the following:

/service:<YourServiceProjectName>.dll /config:<YourServiceProjectName>.dll.config

Fill in the the name of your service project appropriately. The name of the config file points to the app.config file in your project, which is being renamed during compilation. As Working directory set

C:\<Path to my service library project>\bin\Debug\

Page 4 of 4: You have completed specifying settings for your new performance session

You might probably want to uncheck the Launch profiling after the wizard finishes checkbox to check the settings that have been created.

Tweaking the settings

Now your performance Wizard should look like this. That was not the final soultion for me, I needed to further tweak these settings.

Figure 1: The performance profiler after finishing the wizard

So, open the properties of your service project node. On the launch tab check Override project settings and repeat the settings you did before:
Executable to launch: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\WcfSvcHost.exe 
Arguments: /service:<YourServiceProjectName>.dll /config:<YourServiceProjectName>.dll.config
Working Directory: C:\<Path to my service library project>\bin\Debug\

Then delete the second node. You might as well delete the first node and rename the second. Maybe the intial version works too, but I ended up with this after running a profiling session:

Figure 2: The final performance profiler after running a session

For the records, here is the content of my .psess file, of course names and Guids will differ in your file, but sometimes it helps to have just the bits:

<?xml version="1.0" encoding="UTF-8"?>
<VSPerformanceSession Version="1.00">
  <Options>
    <Solution>WCFServices.sln</Solution>
    <CollectionMethod>Instrumentation</CollectionMethod>
    <AllocationMethod>None</AllocationMethod>
    <AddReport>true</AddReport>
    <ResourceBasedAnalysisSelected>true</ResourceBasedAnalysisSelected>
    <UniqueReport>Timestamp</UniqueReport>
    <SamplingMethod>Cycles</SamplingMethod>
    <CycleCount>10000000</CycleCount>
    <PageFaultCount>10</PageFaultCount>
    <SysCallCount>10</SysCallCount>
    <SamplingCounter Name="" ReloadValue="00000000000f4240" DisplayName="" />
    <RelocateBinaries>false</RelocateBinaries>
    <HardwareCounters EnableHWCounters="false" />
    <EtwSettings />
    <PdhSettings>
      <PdhCountersEnabled>false</PdhCountersEnabled>
      <PdhCountersRate>500</PdhCountersRate>
      <PdhCounters>
        <PdhCounter>\Arbeitsspeicher\Seiten/s</PdhCounter>
        <PdhCounter>\Physikalischer Datentr&amp;amp;amp;amp;amp;#228;ger(_Total)\Durchschnittl. Warteschlangenl&amp;amp;amp;amp;amp;#228;nge des Datentr&amp;amp;amp;amp;amp;#228;gers</PdhCounter>
        <PdhCounter>\Prozessor(_Total)\Prozessorzeit (%)</PdhCounter>
      </PdhCounters>
    </PdhSettings>
  </Options>
  <ExcludeSmallFuncs>true</ExcludeSmallFuncs>
  <InteractionProfilingEnabled>false</InteractionProfilingEnabled>
  <JScriptProfilingEnabled>false</JScriptProfilingEnabled>
  <PreinstrumentEvent>
    <InstrEventExclude>false</InstrEventExclude>
  </PreinstrumentEvent>
  <PostinstrumentEvent>
    <InstrEventExclude>false</InstrEventExclude>
  </PostinstrumentEvent>
  <Binaries>
    <ProjBinary>
      <Path>WCFServices\obj\Debug\WCFServices.dll</Path>
      <ArgumentTimestamp>01/01/0001 00:00:00</ArgumentTimestamp>
      <Instrument>true</Instrument>
      <Sample>true</Sample>
      <ExternalWebsite>false</ExternalWebsite>
      <InteractionProfilingEnabled>false</InteractionProfilingEnabled>
      <IsLocalJavascript>false</IsLocalJavascript>
      <IsWindowsStoreApp>false</IsWindowsStoreApp>
      <IsWWA>false</IsWWA>
      <LaunchProject>false</LaunchProject>
      <OverrideProjectSettings>true</OverrideProjectSettings>
      <LaunchMethod>Executable</LaunchMethod>
      <ExecutablePath>C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\WcfSvcHost.exe</ExecutablePath>
      <StartupDirectory>WCFServices\bin\Debug\</StartupDirectory>
      <Arguments>/service:WCFServices.dll /config:WCFServices.dll.config</Arguments>
      <NetAppHost>IIS</NetAppHost>
      <NetBrowser>InternetExplorer</NetBrowser>
      <ExcludeSmallFuncs>true</ExcludeSmallFuncs>
      <JScriptProfilingEnabled>false</JScriptProfilingEnabled>
      <PreinstrumentEvent>
        <InstrEventExclude>false</InstrEventExclude>
      </PreinstrumentEvent>
      <PostinstrumentEvent>
        <InstrEventExclude>false</InstrEventExclude>
      </PostinstrumentEvent>
      <ProjRef>{582C5667-EF99-4934-BD8C-938E75803D39}|WCFServices\WCFServices.csproj</ProjRef>
      <ProjPath>WCFServices\WCFServices.csproj</ProjPath>
      <ProjName>WCFServices</ProjName>
    </ProjBinary>
  </Binaries>
  <Reports>
    <Report>
      <Path>WcfSvcHost160926.vsp</Path>
    </Report>
    <Report>
      <Path>WcfSvcHost160927.vsp</Path>
    </Report>
  </Reports>
  <Launches>
    <ProjBinary>
      <Path>:PB:{582C5667-EF99-4934-BD8C-938E75803D39}|WCFServices\WCFServices.csproj</Path>
    </ProjBinary>
  </Launches>
</VSPerformanceSession>


So, this is it. Now you should be able to find your performance bottleneck in your WCF Code!