F# Reflections

Behind the F# editor tootling. Part 1: introduction to compilers.

Tue, 27 Aug 2019 00:00:00 +0200

Behind the F# editor tootling. Part 1: introduction to compilers.

F# editor tooling, while it’s not on the level of mainstream languages like C# or Java, has been in a unique spot among Functional Programming languages. F# tooling is available cross-platform, it’s feature-rich, stable, innovative and, what’s also essential, it’s commercially supported and actively developed. Moreover, all this is available while communities of some other FP languages are discussing whether displaying simple tooltip is not too expensive.

In the series, I’d like to dive behind what we see in the editor and show some of the magic that lets us have this excellent developer tooling experience. In this post I’d like to do a gentle introduction to compiler design, introduce some concepts and vocabulary that will be useful later, and give a general description of the potential approaches to the editor tooling in context of compilers.

Please keep in mind that this post is a high-level introduction to the concepts with some simplifications and without too many details. If you’re one of a few people that work on compilers or editor tooling this post is probably not for you ;-)

Compilers 101

If you ever took university course about compilers or read any introductory book to the topic you probably know that compilers are often treated as a black box - compiler takes a list of files, some options, do magic, and then it outputs some executable or library.

Typically inside the compiler, several things are happening in order:

Lexing - it’s the process that transforms text (source code) into a list of tokens that can be understood by the compiler
Parsing - it takes the list of tokens produced by lexing and transforms it into Abstract Syntax Tree (AST). AST is a tree that represents the structure of your code. Each node of the tree denotes a construct occurring in the source code. The syntax is “abstract” in the sense that it does not represent every detail appearing in the real syntax, but rather just the structural or content-related details. For example, construct like an if-condition-then expression may be denoted as a single node with three branches. Parsing can produce errors if your code contains structural errors (for example, missing parentheses, or wrong indentation)
Typechecking - In a statically typed language, it’s a process of verifying and enforcing the constraints of the types. Usually it takes as an input AST and any additional context (for example external references) and ensures that everything is fine with your code regarding type system - for example, it checks what types are variables you use, if they contain the members you use if your function returned declared type. A result of type checking is a different type of the syntax tree - in F# compiler we call it Typed Abstract Syntax Tree (TAST). TAST is built on a higher level of abstraction - it contains fewer details about the structure of the code, but more information about entities (variables, objects, functions, etc.) including full type information. Type checking can produce a lot of different types of errors and is one of the primary sources of them.
Optimisations - is a process where compiler perform a different type of optimisations on your code such as removing dead code or inlining. Every optimisation is taking TAST as an input and returns new TAST as an output, and the process is repeated as long as the output TAST is different from input one.
Code generation - it’s the final step of the process, in which compiler takes final TAST and generates output code from it (either in the form of the byte code of some form - IL in case of F# - or just native code)

Compiler 101 vs editor tooling

Unfortunately, the naive, black-box approach to the building compilers is not enough. With such design, you can only start getting information about code only after code is built, inspecting the final result. It is problematic for multiple reasons:

You don’t have any information connected to the structure of code - you only have final results. It may be suitable enough to get some tooltip information or maybe a list of usages of the given Symbol, but it’s not enough in case of any editor features depending on code structure (refactorings, autocomplete lists, formatters etc.)
You only have up-to-date information about your code after you’ve saved the file and built project - for example, you don’t get updated errors in your code while you’re typing and making changes in the file
Saving the file and building a project usually takes a while - if our compiler process is entirely stateless it needs to perform all phases. Also, everything has its costs. Especially optimisations and code generation is not interesting for us from editor tooling point of view - we want to provide tooling in a context of the source code we edit, not in the context of optimised, final artefact we get from the compiler.

Due to those problems editor tooling for languages using a black-box design of the compiler usually have huge problems - either they’re limited in what’s possible in such editor tooling, or attempt to work around those problems by re-inventing what compiler does by using technologies such as tree-sitter or just by writing “presentation compiler”.

However, such workarounds are never as accurate and fully featured as systems based on the compiler directly. While building performant, accurate and productive editor tooling, we need to have direct access to the internal state of a compiler (such as AST, TAST, and Symbol information) and that’s why we need a different approach to building compilers.

Modern compiler design

In recent years many mainstream programming languages have moved towards compiler service approach to the editor tooling. In such approach, editor tooling is powered by internal capabilities of the compiler which ensures that editor functionality. Examples of such approach are [C# Roslyn API] or [F# Compiler Service] (on which we will focus more in the next parts of this series).

The modern compiler needs to be:

API - compiler needs to be accessible as a normal library or some other kind of API. We need to be able to run it inside another process, and we need to be able to communicate with different layers of a compiler (usually lexer, parser and type-checker)
Server - it needs to be ready to run as a long-running, stateful application that gives us the ability to do partial operations (for example single file parsing) in the context of a whole compilation unit. The fact it’s the stateful, long-running and asynchronous process has vast implications on the design of compiler and the performance optimisations.
Database - as mentioned above compiler is a long-running stateful process. Additionally, we need to have a way of getting what is usually internal information of the compiler - Tokens, AST, TAST, list of known by the compiler symbols and more.

With such compiler capabilities, we can host it directly in editor process (for example F# or C# tools in VS) or in another application providing higher-level API that is also communication layer which can be used from editors (for example language server using Language Server Protocol)

Summary

I hope this blog post is good enough, the high-level introduction of compiler design, concepts and vocabulary that you’ll need to know in the future posts from this series.

Ionide — Introducing Info Panel

Mon, 06 May 2019 00:00:00 +0200

Ionide — Introducing Info Panel

Today, I’d like to talk about a new feature called the Info Panel that has been released in Ionide 3.36. For me personally, it’s one of the most exciting features we’ve introduced in Ionide so far. Hopefully, it will help F# developers as nicely as placing function signatures in CodeLenses or fast cross-project navigation powered by background symbol caches.

The problem of API exploration

One of the biggest problems some software developers have is remembering all the functions, classes, and namespaces that are available in a vast set of dependencies they use, or even that are present in their code bases. Some IDEs try to help with that problem via tooltips or IntelliSense.

However, these solutions suffer from two problems. First, the visual help they have is temporary. Second, tooltips can sometimes feel intrusive. It’s sometimes quite common to have something pop up and hide code below it, and if you move your cursor the information changes are disappears.. Also, tooltips or additional IntelliSense popups are typically small — the IDE has to play a tricky game of tightrope where it needs to display useful information, but do so subtly enough not to feel intrusive.

One alternative to an Info Panel exists in Visual Studio today, called the Object Browser. It’s a separate window that presents a tree view of references to a project where you can dive into constructs as-represented in their compiled form. Although this is helpful for exploring references, it’s conceptually detached from your codebase. Imagine a situation when you’re reading your code and notice a function call that you don’t understand — it’s not as easy to use Object Browser as just hover over the symbol — you need to click through library content, to find appropriate class or function. And if you want to move to another function next line, you need to do the same.

Rich API exploration

The Info Panel is inspired by XCode’s Quick Info Inspector. Thanks a lot to Dave Thomas for familiarizing me with this feature. BTW, you can support Dave — one of the best F# OSS hackers — on his new Patron page!

Info Panel is an attempt to bridge those two ways of presenting information. It combines the interactivity of tooltips with the less intrusive UI of a separate display panel. The panel auto-updates as you navigate through your code, but you can also navigate information via the panel, independently of the code you’re looking at.

By opening the Info Panel in VSCode, you can display rich information about the current symbol under your caret, such as its signature, nicely formatted XML documentation, its members, properties or functions, used attributes, implemented interfaces. Because it’s a separate panel, the information is persisted and doesn’t overlap with your code window. There is enough space to provide quite a lot of nicely formatted information. It also supports link navigation. For example, if somewhere in the documentation DateTime is present, you can click on it, and move to the documentation about the DateTimetype. The Info Panel uses a combination of data from the compiler service and XML documentation, so it works for both external dependencies and your own code.

Info Panel may update its content on the cursor move or on the hover (it’s configurable) but also supports locking — the state in which the content won’t be updated by normal action but only by click link or forcing update with a command. This means that you can keep information that is interesting for you while moving through a code base.

One of the additional options is complete replacing tooltips — when it’s activated you won’t see normal tooltips when you put the mouse over the symbol, only Info Panel will be updated (feature suggested by Brian Mitchell).

Check all possible options for Info Panel in VSCode — they all start with FSharp.infoPanel prefix

What’s next?

Info Panel is in pretty good shape today however there are still a couple more features that I want to add in a future:

Handling namespaces — currently we don’t display any information for namespaces.
Navigation history — I think it would be nice if go back/forward commands were implemented
Navigation breadcrumb — breadcrumb UX in VSCode is pretty nice for navigating through the code base, I can imagine having something like that for the documentation exploration. (suggested by Nino Floris)
More information for the symbols — we could potentially add even more information such as inheritance hierarchies, declaring entities etc.
Investigate how Info Panel can be used for replacing/enhancing IntelliSense

If you’d be interested in helping there are two places where you can start:

If you’d want to improve generated documentation take a look here
If you’d want to improve UX around the feature take a look here

Summing up

I really hope you will like this feature — from my point of view, it fits into Ionide design of finding good ways to provide as much useful information as possible to the users — such as Code Lenses or rich tooltips we have right now. Try it out, and let us know what you think!

You can support Ionide development on *Open Collective.*

Ionide — A New Hope

Sun, 24 Mar 2019 00:00:00 +0100

Ionide — A New Hope

You probably know that one of my biggest problems with the F# ecosystem and OSS work I’ve been doing has been lack of any sustainability strategy. I’ve attempted to change that by creating Open Collective page for the Ionide, and at the same time, I’ve started talking with existing partners about possible ways of making work on this crucial parts of the ecosystem more sustainable.

Today, I’m really happy to announce that for the next few months I’ll be working full time on the F# open source, cross-platform tooling focusing mainly on Ionide and FsAutoComplete. This is possible thanks to the partnership between my company — Lambda Factory — and Microsoft.

Scope of the work

The contract with Microsoft will enable me to work on certain strategic goals of the Ionide and FsAutoComplete that are important for both the F# community and Microsoft. Our initial goals for the next few months are:

Finishing Language Server Protocol (read more about LSP) implementation in FSAC and plugging it into Ionide
Fix performance reliability issues after triage and full performance rebaselining after LSP support is in
Work on porting all components to .Net Core/Standard and using .Net Core version of FSAC by default in Ionide to improve the getting started experience
A potential stretch goal is working on improvements to the scripting experience in Ionide

These goals are trying to tackle some of the most important issues of the F# cross-platform tooling ecosystem, and will benefit not only Ionide but will have an impact on other tools that F# developers use every day.

Relationship with Open Collective

I’ve been asked by some people who heard the news a few days ago if this affects the Ionide OpenCollective effort. In short, no, but I’ll clarify some specifics.

From my point of view, this doesn’t change anything with respect to OpenCollective. One of the reasons why we’ve decided to choose Open Collective (over Patreon or PayPal donations) is the fact that it’s about supporting the project, not any particular creator. Open Collective is a fund for the Ionide project itself, with a transparent invoice system that ensures any withdrawals are public. This allows any Ionide contributor, not just me, to spend dedicated time on the project in the future. It is a long-term sustainability plan for Ionide.

Contract with Microsoft is task-based — hopefully, it will result in fixing some of the biggest issues with the Ionide and FSAC that we have right now but is not creating long term sustainable environment — something I want to create with Ionide as an organization and with Open Collective.

I hope that F# community will continue supporting Ionide on Open Collective as generously as right now — and I hope that soon, I’ll be able to do some announcements about how we will spend those funds on the project related goals such as promotion, documentation and more. But that will come later, as I’ll be quite busy in the coming months improving Ionide by working with Microsoft.

But Embrace, Extend, and Extinguish…

Ionide is still and always will be MIT licensed project owned by F# community, and FsAutoComplete is and will always be Apache 2.0 licensed project owned by F# Software Foundation.

Microsoft is not acquiring Ionide, nor is there any additional stipulations or attempts to re-prioritize the direction of the Ionide project. In fact, I was asked to help define the scope of the work and I’ll be helping decide what the most important issues are to fix. To be clear, Microsoft is not gaining control on Ionide, and they don’t gain any decision making power different than regular F# users. Ionide is still a community project and the maintainers continue to decide what gets added or removed.

Next few months will be fun…

I need to say that I’m really really happy to have this opportunity — for last few years I’ve been saying that it would be really good for an ecosystem if someone was working full time on F# cross-platform tooling.
My goal for the last few years has been pointing out that there is huge potential in F# cross-platform tooling, in VSCode as an IDE platform, in F# Compiler Services that’s the common infrastructure used by all F# tooling, and that’s been fairly unique project among Functional Programming languages. I really believe that we can make F# tooling best-in-class (of FP languages), and I hope that the partnership with Microsoft is a step in this direction. However, we still need to remember that F# tooling is owned by the F# community and only together we can make it better so… contributions are welcomed!

Additionally, I need to thank Phillip Carter (F# PM @ Microsoft), without his help, and his advocating for F# inside Microsoft things like that would never be possible. Thanks!

PureScript on Azure Functions

Sat, 12 Jan 2019 00:00:00 +0100

PureScript on Azure Functions

Serverless architecture is one of the hottest topics in cloud computing, allowing developers to build and run applications and services without thinking about the servers actually running the code, providing a scalable model for building distributed, event-driven systems, as well as significantly reducing operational costs.

Microsoft’s take on the serverless problem is service called Azure Functions. Out of the box, it supports .Net languages (C# and F#), Python, Java and JavaScript. The last one is very important, as nowadays there exist a wide variety of languages that can compile down to JS. And today, I’ll talk about one of such languages — PureScript.

PureScript is a strongly-typed, purely functional programming language heavily inspired by Haskell. It supports a set of advanced language features such as higher kinded types, type classes, row polymorphism and many other features I don’t understand.

Requirements

To get started with Azure Functions and PureScript you need to install several dependencies. First of all, you need Node.JS and NPM as it is the runtime for Azure Functions, and is used by the PureScript toolchain. Secondly, you need PureScript itself — you can download and install it using npm with npm install -g purescript. Next thing is some additional PureScript tooling, you can get it with npm install -g bower pulp. In the end, you also need to install Azure Functions tooling that will enable you to build and test your serverless application locally. This tooling can be found on GitHub — https://github.com/Azure/azure-functions-core-tools.

Getting Started with Azure Functions

We will start by creating a simple JS function. In the local folder run func init MyFunctionProj. When prompted, use the arrow keys to select a node worker runtime from the language choices. This will create a new folder and a new NodeJS Function Application inside. Inside the new folder initialize new NPM project in it with npm init. After you’re done with answering all the prompts, we will create our first endpoint — func new — name MyHttpTrigger — template “HttpTrigger”.

Without going into details, Azure Functions is an event-driven system that supports multiple different types of triggers and bindings. The template we’ve chosen will create simples possible function that is triggered by an HTTP request and returns an HTTP response. If you want to learn more about possible triggers and bindings visit Azure Function documentation — https://docs.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings

After all those operations are done you should have following folder structure:

MyFunctionProj
 | — MyHttpTrigger
 | | — index.js
 | | — function.json
 | — node_modules
 | — host.json
 | — package.json

You can now test your first Azure Function! Run in terminal func host start and wait a moment. After everything is initialized you should see (among other things something like: MyHttpTrigger: [GET,POST] http://localhost:7071/api/MyHttpTrigger. You can now visit this URL and see some output in the browser.

Adding PureScript to the mix

Adding PureScript to your project is fairly simple, and just requires some small changes to the project files. Firstly, still in the same directory, let’s create new PureScript project — pulp init — force. (You need to use — force flag as there already exists .gitignore file in the folder). Your directory should now contain several additional elements:

bower.json — contains library dependency information
* bower_components/ — a directory for installed dependencies
* src/Main.purs — Entry point module for your project
* test/Main.purs — An empty test suite.

At this point, you should be able to build the project — pulp build, by default the compiled JS code is put into output folder. But it’s not yet plugged into our HTTP trigger. Before doing that, let’s make our PureScript code bit more interesting.

This section is following PureScript getting started guide

The first step will be installing external PureScript library — bower install purescript-lists — save. Now, open the folder in your favourite editor (VSCode) and go the src/Main.purs file. Let’s replace its content with solution to the Project Euler #1

We can now build the code with pulp build — skip-entry-point command. Now it’s high time to plug our PureScript code into Azure Functions. Open MyHttpTrigger/index.js file and put there following code.

As you can see, it’s fairly straightforward. The most important thing is the require call that will load compiled-to-JS PureScript code. After that, we can call any function defined in PureScript as normal JS function.

At this point, you can start your function again (func host start) go to the given URL and check if the answer to the Project Euler problem is correct.

Deployment

There exist multiple different ways of deploying Azure Function applications — from Continues Deployment on every push to the repository, through complex build scripts to simple uploading zip file. For starters, the last method is the simplest one. You need to zip all the necessary files — MyHttpTrigger folder, output folder, host.json and package.json files. After you have zip file you can use Azure Portal, Azure CLI or simple calls to the REST API to upload the Function.

To learn more about deploying Azure Functions Applications visit official documentation — https://docs.microsoft.com/en-us/azure/azure-functions/deployment-zip-push

Summary

Finishing up, in this blog post I’ve shown how easy it is to use modern, advanced, purely functional programming language — PureScript — on the serverless platform — Azure Functions. The example repository containing all sample code can be found on GitHub — https://github.com/Krzysztof-Cieslak/AzureFunctionsPureScript

Future of F# cross-platform editor tooling

Mon, 31 Dec 2018 00:00:00 +0100

Future of F# cross-platform editor tooling

Currently we have this beautiful moment in the year when everyone writes, tweets, and instagrams about their achievements in past year, and plans for the next one. I’ve also done this before (often being week…or month late), however this year I decided to write about something else. So let’s talk about F# cross-platform, community driven editor tooling.

Language Server Protocol

Before diving into F# specific things, I need to mention project that has changed how editor tooling is built over the past two years - Language Server Protocol (LSP). LSP is specification of the communication between a client (editor) and a server that provides language tooling capabilities such as autocomplete, tooltips, etc. It reduces the* m-times-n* problem to the m-plus-n problem —similar to how Virtual Machines solve the same problem for deploying code to many platforms (great example here is Z-machine).

For example, instead of creating a Python VSCode plugin, a Python vim plugin, a Python Atom plugin, and Python plugins for any other potential clients, it allows developers to focus on single implementation of language server, this will be automatically supported by all clients that implement LSP.

Of course idea of language server is not new — it’s been fairly common way to create language tooling. In fact, the F# language already has a language service called the F# Compiler Service. It’s an F# API into the internals of the F# compiler, which does all of the heavy lifting in determining things that are useful for editor tooling. However, it is ultimately just an F# (i.e., .NET) API and cannot be consumed in every environment without an additional interface. An additional layer that can interface between the F# Compiler Service and any editor (i.e., not just Visual Studio) is needed to allow for F# tooling to be used everywhere.

FsAutoComplete

As I’ve mentioned — the idea of a language server is not new. And F# has been having one for years. Initially created in 2011 (as far as I know, that was way before I’ve joined F# community), FsAutoComplete (FSAC) has been used by all editors that aren’t Visual Studio, such as Atom, Emacs, Vim, and VSCode. The project provides a high level API over the F# Compiler Services, and communication layer (standard I/O, and HTTP web server) that enables using it from non -.Net platforms. It’s been long standing project, with many past iterations — in fact, prior to the F# Compiler Service API being made available, it used reflection over the F# compiler to access internal APIs! It has had many great maintainers and contributors over the years (I especially need to mention here Dave Thomas, Robin Neatherway, Tomas Petricek, and Enrico Sada) and it has been the magic that powers Ionide since day-0.

Current status

When I started Ionide 4 years ago, FSAC was in pretty good shape, providing many crucial APIs and one communication layer (stdio). Over the years, I started to contribute more and more to the project, and with help of all awesome people involved, new contributors and community support we’ve managed to transform FSAC into one of the best language servers I know (especially considering it is not a commercial language service, such as those authored by Microsoft or JetBrains), and I feel that it has been one of the most important tools created by the .NET OSS community.

Also, what is important personally for me is the level of innovation that we’ve managed to achieve in FSAC and Ionide. Additional diagnostics such as unused opens or unused bindings, integration with a third party linter, features powering Ionide’s CodeLenses, background caching that enables very fast use of any feature that requires symbols (such as Find References, CodeLens showing number of references, etc.), and even custom F# Analyzers. These are not the features that you typically see in the editor tooling created by independent vendors, nor especially in editor tooling for FP languages (yes, some of those features have been present in powerful IDEs for particular languages — CodeLenses for C# in VS, or famous background caches in JetBrains IDEs — but never for FP languages, and never provided by community driven tools)

Bright Future?

So the title of the post promised talking about the future… but the future is now. Today, I’d like to show you the thing I’ve been working on over the Xmas break for last 2 weeks — the LSP communication layer for the FSAC.

Huge thanks goes to Julien Roncaglia for his initial work on LSP + FSAC proof of concept few months ago, and his F# LSP server abstraction implementation that has been used as a base for my current work.

Being active maintainer for editor tooling for 2 different languages, a developer that created plugins for multiple editors, and someone that is generally interested in editor tooling, it has become obvious for me that LSP has won (at least in the niche of cross platform tooling provided by community/independent vendors). There are more and more client implementations (including really interesting online IDEs like GitPod), more and more server implementations (which means there are more and more investments into clients) and the protocol itself is becoming more powerful and feature-ful. I strongly believe that it is the future of the F# cross platform tooling. From the F# point of view there is couple of important advantages:

It moves lot of code from Ionide code base to the FSAC code base — it means that other editors than VSCode can use some of the improvements that has been “hacked” in Ionide itself
It makes the contribution process easier — Ionide is a VSCode plugin written in F# compiled to JS with Fable using FSAC (totally separate code base). This makes the process of contribution a bit awkward — if you want to contribute you need to know which code base to work in, if it’s Ionide you need to have the Fable toolchain set up. The idea behind using F# for Ionide was lowering the bar for contributors (“people interested in making F# tooling better are F# developers so the code should be F#”). As lot of Ionide code is moved to the FSAC, it makes contributions even easier by moving a lot of code to a normal F#/.NET Core project
It’s good news for anyone wanting to use Atom or Vim or Emacs (or maybe ST3?) — some of those plugins have not been maintained very actively (or have been deprecated like Ionide-Atom), but moving to LSP will mean that getting all new fancy features should be way easier or “free”.
I’ll soon start to work on Ionide 4.0 release that will be based on LSP. Lot of Ionide code will be removed, since Ionide provides more functionalities than just language features — things like solution explorer, creating new projects, right click -> debug, and many features focused on developer experience are implementations specific to VSCode (and that will probably never change).As FSAC will take over lot of responsibilities, I hope to focus more on the UX in Ionide itself, which is also a very interesting problem on its own.

As a side note — we are looking for new maintainers for FSAC. We would love your help!

Grim Future?

Of course, every real life story have also bad parts. And there are couple of the dark clouds on the horizon. First of all, some of the problems with Ionide/FSAC won’t go away with moving to LSP— they are connected to the complexity of build and require more knowledge and time than I have (if someone from MSFT is reading it — I would love to get, for late Xmas gift, open source, cross platform language server for the .Net project system). But that’s just a technical problem. The bigger one is that nothing has changed for last year and I can repeat my words from last years’ post. While I believe that F# ecosystem is in its best shape ever, it’s still building a castle on the sand — not enough contributors, depending on same handful of people that overproduce, no commercial support for OSS projects, no companies interested in investing into F# OSS ecosystem. Ionide and FSAC are both in unsustainable state — crucial projects for F# community, with thousands of users, that are used by a huge part of the community every day — they get no commercial support and there are not enough contributors to make up for this.

So… 2019?

So, how will 2019 will look like? Only time will tell. But in general I believe that there are more bright sides to the story than dark clouds. F# ecosystem is in fairly OK place — tooling is really OK, .NET Core adoption is rising which is a good sign, there are more and more interesting projects, community is active, and projects like the SAFE stack are becoming mature and are getting decent adoption.

However, as a community we still haven’t solved the problem of OSS maintainability. One one hand, it’s nothing unusual — most communities struggle with those problems. On the other hand, it’s obvious that F# commercial adoption is very often driven by the few people building awesome OSS things, and maybe we should help them somehow?

Anyway, I hope you will build great things with F# in 2019!

Introducing F# Analyzers

Fri, 14 Sep 2018 00:00:00 +0200

Introducing F# Analyzers

One of the most exciting features of Roslyn (modern C# compiler) is ability to plug into it custom extensions called Roslyn Analyzers. They are live, real-time, project based plugins that enables to diagnose source code and surface custom errors, warnings and code fixes into Visual Studio. What’s really important is fact that there are project based, and distributed through NuGet which means they are easy to install, and ensure that all developers working on the project have exactly same analyzers installed. This is really useful if you want to ensure common best practices, style etc while working on the project.

Today I’d like to introduce preview of the project called F# Analyzers which adds similar capabilities to Ionide (F# support in VS Code). Project is still in the early days, so design and technical details can change in time, but I’m really excited to share it with users even in its current state.

Architecture overview

Before diving deeper into Analyzers themselves, it’s important to understand general architecture of Ionide, and where Analyzers fits into this architecture. This has really important impact on the Analyzers capabilities and limitations.

On the one hand we have Ionide itself. It’s VS Code plugin which means it’s using NodeJS runtime, running as a child process of the editor, communicating with VS Code using set of APIs provided by VS Code. On the other hand we have F# compiler, or exactly speaking FSharp.Compiler.Service which is library version of F# compiler, exposing some additional APIs useful for creating tooling for F#. Of course, FSharp.Compiler.Service is .Net library which means it can’t be used directly by NodeJS process.

Last important part of the architecture is FsAutoComplete. It’s a project that provides communication layer and high level API between FCS and external world. It’s used by vim, emacs and Ionide as all those plugins are hosted in non .Net environments.

The F# Analyzers are using extension point provided by the FsAutoComplete which means that in current shape they can be supported only by the editors using it (and they requires some work on the plugin side of things - currently only Ionide supports them). And it means that diagnostics provided by them are visible only in the editor, not during compilation.

Analyzers basics

Analyzer is normal F# function. As an input it takes Context object which contains current information about the file - Parse Tree, Type Abstract Syntax Tree, Symbols information, and returns list of the Diagnostics - records containing diagnostic message, type, description, and potentially list of fixes that can be used to fix the problem.

F# Analyzer contains 3 parts - first one is an extension point in FSAC which is responsible for loading and running analyzers, second one is a editor handler (part of Ionide, in this case) that displays diagnostics and fixes. Last part is FSharp.Analayzers.SDK project that’s used to create analyzers.

Building your first analyzer

From infrastructure point of view, creating analyzer is simple - you need to create normal F# project, netstandard2.0 , add reference to FSharp.Analyzers.SDK (avaliable on NuGet - https://www.nuget.org/packages/FSharp.Analyzers.SDK) and create analyzer functions - normal F# functions taking Context returning list of diagnostics, and marked with Analyzer attribute.

Due to unresolved technical limitations your Analyzer must contain “Analyzer” in the name of output dll.

However, from technical point of view creating analyzer is fairly difficult task - you need to be familiar with processing typed or untyped abstract syntax tree, or accessing FSharp Symbols information. Currently SDK doesn’t provide any helper functions, but we hope to add some higher level abstractions in the future. Currently, I recommend going through FCS documentation, to get more familiar with the concepts

Generally speaking processing Abstract Syntax Trees is usually done by recursive pattern matching on the DU representing syntax nodes. The following snippet shows how to travers Type Abstract Syntax Tree and call some function on every node that represents call of the member. While it looks bit scary, and complex if you take a look on it bit longer you will notice that it’s just big recursive pattern matching. Also, similar pattern matching will be base for most analyzers, which means you can reuse the code pretty easily.

Having this bit of infrastructure code, let’s now move to the analyzer itself. As I’ve mentioned before, analyzers are normal functions that have particular signature and are marked with attribute.

In the above snippet we create analyzer that will detect any call of x.Value where x is an F# Option. As you probably know calling .Value on an option is not total function and can throw exception if an option instance is None, so checking for it may be good idea.

The handler function in above example is executed for every member call. In it we check if the name of the function we call, and if it’s Option.Value we add the range of the symbol into the state. After we’ve processed whole tree we map all ranges into the warnings that will be displayed in the editor. There also exist possibility of providing set of code fixes, but we don’t use it in this example.

Whole code of the sample analyzer is available on GitHub - https://github.com/Krzysztof-Cieslak/FSharp.Analyzers.Sample, and analyzer is published on NuGet - https://www.nuget.org/packages/FSharp.Analyzers.Sample/

Using analyzer

Analyzers are enabled by FSharp.enableAnalyzers setting inside VSCode. By default it’s disabled, so you need to edit settings to enable analyzers support. Using analyzers is as simple as adding Analyzer NuGet package with Paket - for out of the box support in Iondie you need to use Analyzer group for it.

After the package is added to paket.dependencies and restored, Ionide will automatically detect and load analyzers. It may require editor restart as analyzers are loaded on plugin startup.

Ionide also contains setting FSharp.analyzersPath that will enable you to configure paths from which Ionide loads analyzers, by default it’s using packages/Analyzers and analyzers folders.

For example let’s test our new analyzer on the following snippet.

This will result in underlying x.Value with a warning saying that Option.Value shouldn’t be used

Sample project using analyzer can be found on GitHub - https://github.com/Krzysztof-Cieslak/FSharp.Analyzers.Sample.Usage

So what’s next?

Currently there are several important limitation of the F# Analyzers. First of all they’re supported only by Ionide. Secondly to write analyzer you need to be familiar with F# (T)AST pretty well. And in the end writing fixes requires providing new version of code in textual form (unlike in case of Roslyn Analyzers where fixes can be provided as AST transformation), what may be difficult and error prone. Also, whole mechanism is currently in an experimental phase, relies on Reflection etc. so it can be bit unstable.

However, I treat this as an experiment and proof of concept, that may be used as a base for discussing similar functionalities being added to F# compiler or other IDEs. Also I’m super excited to see what potentially great things can F# Community create having such extension point in the editor (Evil Hint: You can use this feature to run any code inside of the language server and return some information to editor… which sounds awesome and crazy at the same time)

Summary

In this post I’ve introduced preview of new tooling related project that enables users to add custom analyzers to your editor, bringing functionality similar to Roslyn Analyzers. While this is only experiment, and proof-of-concept I’m looking forward to seeing all innovative things it can enable. Analyzers support has been released in new Ionide version - 3.27.0 - so it’s out there and ready for experimenting!

Building an MVP with F# and Saturn

Thu, 16 Aug 2018 00:00:00 +0200

Building an MVP with F# and Saturn

Creating an MVP (minimum viable product) is one of the best way of bootstrapping your startup. As a new company getting a quick feedback from the application users, bringing an application to users as fast as possible, being able to adapt as quickly as possible to the market changes, and providing frequent application updates is crucial for the initial success of the product. But it’s also important to understand that an MVP software development is not synonymous with unfinished or a primitive product that was created in a hurry.

What is an MVP?

Minimum Viable Product (MVP) is the smallest, most concise version of your product you can initially release for feedback. It enables a full turn of the feedback loop with the least amount of development time and effort. This allows the targeted users to try a product and evaluate it to make the complete version better. It is a frequently updated environment with the new features that could be seen and tested by clients. New registration page, an admin management panel, as well as email notifications, and any other new features you can imagine.

MVP development allows early adopters to understand the vision or promise of the final product and provide valuable feedback to guide developers moving forward. The main advantages of the MVP software development are:

MVP has enough value that people can already use or buy it
It demonstrates enough features to hook and retain early users
It provides feedback loop to guide future development
It shows enough future potential to start marketing around the project

What’s important - MVP development is not only the strategy for startups. When you already have well-established company and customers, you also need to have some way to experiment with new potential products or features. In a lot of cases, the minimum viable product really is just a way to do that.

Few words about F#

F# is a functional-first programming language running on the .Net platform. Paired with .Net Core - modern, cross platform implementation of the .Net Framework - it is fantastic tool for writing modern applications. .Net Core provides industrial level of the performance, security, huge ecosystem of the libraries, and always growing open source community. F#, itself an open source language with fantastic community , thanks to expressiveness, functional-first approach, great developers tooling and advanced language features provides unmatched developer experience and fast development speed, so important for building MVPs. It can also be your secret weapon that makes you stand out from many different companies, and that let you hire more talented people that just want to work with more niche technology.

Introducing Saturn

F# due to its functional nature is perfect fit for the web applications. In the end HTTP web server may be treated as a function that takes Request and returns Response.

Saturn is a modern web framework for the F# focusing on high level abstractions and great developer experience, allowing developers focus on the things that matters for your business. Influenced by the popular frameworks such as Rails or Phoenix it comes with all the batteries included. Its ecosystem includes development tools that allows developers to quickly create new applications, add new features to existing applications, and test new features; it integrates with existing ASP.NET Core ecosystem that provides huge set of existing modules that you can use in your application; it provides pre-defined configurations that limits the number of decisions that developers need to make. And all this while using highly expressive and modern programming language.

Typical difficulties

While MVP development is really powerful technique, just like every other methodology it has its own drawbacks. First of all you need to understand it’s not about delivering product as fast as possible so customers can try it out. MVP development is about refined and validated learning. This simple fact must be understood by everyone - stakeholders, developers, customers. You have to understand that final product may drastically change due to user’s feedback. And this is something that must be embraced by the development team, and should impact the way in which they work. Developers needs to have regular feedback session, learn from them, prioritise features.

As a result, this should have an impact on the technologies chosen for developing MVP:

refactoring plays the key roles, when you see that some part of the product is not scalable or flexible enough, or that it just need changes based on the user’s feedback you need to be able to quickly change an existing code base
correctness is important, especially the one you can get for free. Using static type checking, code linting, and code quality tools is crucial to provide product that’s working well enough, without spending much development time on it
flexible but simple design - design of your product must be flexible enough to provide ability to change the product based on the feedback, but at the same time it must be relatively simple - for MVP development you want to get to the market as fast as possible, and you don’t want to spend time building sophisticated architecture that may not work after all changes that will happen in the future

Fortunately, F# and .Net Core are great solutions for all those problems - functional first approach naturally pushes you toward pit of success, and simple, flexible architecture, static typing with powerful type inference enables you to create correct software without much overhead, best in class editor tooling helps you to quickly refactor existing code.

Commercial support

The last important factor in choosing right technology is having an ability to hire someone to develop the product, or just get help. F# ecosystem is including wide range of the options for getting commercial support - starting from the established companies providing support for SAFE Stack through multiple independent consultants to Lambda Factory. Lambda Factory specialises in creating web applications, F# training and consulting, MVP development, and developer relationships. We provide everything you need to transform your startup into prospecting business - help with creating MVP, iterating over the feedback loop, hiring talented developers, and DevRel marketing.

Challenges of post-OSS world

Sat, 11 Aug 2018 00:00:00 +0200

Challenges of post-OSS world

Open Source Software has won. After years of convincing people to use open source software, fighting with false dichotomy between OSS and industrial, commercial software, and defending against negative biases, position of OSS is no longer disputed - it’s been used by vast majority of companies around the world, it has become default choice when choosing technologies, it powers the internet, our PCs, mobile phones and most devices we use every day, it’s been accepted by huge, conservative companies that were against whole concept few years ago.

But have we really been ready for that? For most of its history OSS community was busy fighting for survival. And we created strategy, methodologies, and public relations attitudes that were helping us fight for this survival. But the times have changed and now we face totally different set of problems that we need to solve.

Outdated foundations

93% - percentage of npm packages with just one maintainer

General understanding of the mechanism ruling the OSS ecosystem is relatively outdated. As mentioned above - our knowledge about foundation of community based software was shaped when OSS was treated as something bad and undesirable by mainstream programming industry. Organisations like Free Software Foundation, licenses like GPL have their roots in 80s and 90s. The amazing essay *“The Cathedral and the Bazaar” *that has shaped how we understand the community based software was published in 1999. Those were totally different times - when the software market was dominated by companies like IBM, Oracle and Microsoft that were clearly against any ideas of free software. And back then we need different ways of speaking about OSS - the Bazaar analogy was designed to win hearts of the developers, to show that they don’t need to be afraid of the open source software. And it has done that really, really well. However it shouldn’t be used in 2018 as a way of describing how OSS works - we now face different set of issues.

While many of the ideas from *The Cathedral and the Bazaar” *have aged really well, and still apply to any kind of software development, OSS nowadays is not about creating huge communities, applying “given enough eyeballs, all bugs are shallow” law, or in principle following Bazaar model. On the contrary, given all the success that OSS has achieved the Bazaar model has been unsuccessful - only 3% of popular OSS projects are using such development model. Of course those 3% contains some hugely popular and impactful projects - such as Rails or Linux - but one could argue that they’re using this model as a result of their success and not other way round. In the end, no project starts as a community, there is always someone in charge, and communities around the projects are created only after they are successful.

Unlimited growth

28 millions of developers - number of registered users at GitHub.com

No-one in the 90s could predict the scale of the success that OSS will achieve. The number of developers doing OSS, number of projects, and number of users has raised by several orders of magnitude over last 20 years. The Bazaar model was created when huge percentage of the people using OSS were people developing OSS. But current state of the world is bit different. Let’s just compare some numbers.

The number of downloads per 2 weeks:

1998 - 180K downloads of Netscape (most popular OSS project at the time)
2017 - 21M downloads of Loadash (random JS library)

The number of registered users:

2001 - 208K on SourceForge
2018 - 28M on GitHub

The number of active projects:

2008 - 150K on SourceForge
2017 - 29M on GitHub

As we can see, OSS nowadays is massive, global and popular. But there are main 2 issues here. First one is yet increasing fragmentation of the ecosystem - unlike what was presented as big advantage of the Bazaar model, modern OSS is not about creating those singular points of focus for the communities. We don’t get “enough eyeballs”, instead we all rely on the projects that are very often driven forward by single maintainer and one or two other contributors. Second one is that growth of the user-base doesn’t resulted in proportional growth of contributors of the projects. And what’s even more important the growth of the contributors hasn’t resulted in the proportional growth of the maintainers - which results in more and more pressure put on the focal parts of our communities which itself is really huge issue.

Maintainers struggle

“The more successful you are, the more you get punished with GitHub notifications.” - What it feels like to be an open-source maintainer

Maintainers are the focal points of the community of any OSS project. Not only they’re often main developers behind the project but also they have many other responsibilities - they need to keep users happy, they need to handle contributors to create good workflow that will enable new code contributions, they need to help first time contributors, they need to review PRs, handle releases, and keep general direction of the project.

“Current status - anxious and depressed whenever I open my GitHub notifications. Programming is so much fun”

However we don’t really get new maintainers too often. The number of users growth really fast, the number of developers grows - and while those 2 things are great, they create huge pressure on the maintainers. And the Bazaar analogy is part of the problem here - we expect someone to always step up, we expect contributors to become more and more involved. But the reality is bit different - 1/2 contributors contributes only once, and they account for around 2% of the commits. And every contribution is additional work for the maintainer, additional PR to review, and additional code to maintain later. Of course, every contribution is valuable, and welcomed but in general OSS community has huge “second time contribution” problem - we put lot of emphasis and work to optimise first contribution experience - but it unfortunately is not resulting in many long standing members of the community.

Sustainability

$143,000,000 - Estimated value of open source software in $1B Instagram acquisition

OSS movement was shaped in the times were most mainstream software was proprietary, it was not vastly used, and no business depended on it. But nowadays, OSS is everywhere - it powers internet, operating systems, software and products we use everyday. It’s hard to imagine building new software without using any OSS technologies and OSS is used by the biggest, most conservative (In terms of technological choices) companies of the world - 100% of Fortune 500 companies are using npm.

And yet, even some most popular OSS projects ever struggle to get any founding, and build any notion of sustainability. But we continue to build businesses using those tools and libraries. One would hope that in the future we will start to think about open source software in terms of digital infrastructure - that, just like our normal infrastructure, requires constant funding to function properly.

How we talk about OSS

“Just like physical infrastructure, digital infrastructure needs regular upkeep and maintenance.” - Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure

Probably to solve some of those problems we need to change a way we talk about OSS - instead of convincing our managers that we need to contribute because it is “good” behavior, or because it will improve “morale” of the team we should talk in business terms - risk and opportunity. Contributing to the projects decrease our risk as a business - it decrease risk that project maintainer will rage quit and we will be stuck with unmaintained project, it decrease risk of needing help of other people if we have some problems with the project. Contributing to the project is also opportunity - we can shape project, add new features that will enable us to make our business solutions better.

And the other interesting value proposition for contributing to OSS is just plain marketing value - especially if your product targets technical communities. We know live in the times of Developers Relationships movement, were any company building software projects for developers spends lot of resources for promotion. Contributing to OSS is great way of such promotion. And even if your business is not building tools for developers - it will make hiring talented developers way easier.

Solutions? I have none

I don’t know how to solve the problems of the current OSS world. But it is pretty clear that OSS needs evolve - we need new definitions, new solutions and new leaders for the movement that already changed the world. OSS nowadays struggle with totally different set of the problems than it did 20 years ago, so using the analogies that were used 20 years ago, may not be the way forward. And I do believe that we need to find those solutions pretty fast… or it will turn out that we’ve built our great castle on the sand.

This blog post is adaptation of my “Challenges of post-OSS world” talk I’ve presented on couple of events. If you’d like to have me talk about OSS on your conference and your company feel free to reach me at krzysztof_cieslak@windowslive.com

Both talk and this blog post has been deeply influenced by the work of Nadia Eghbal.

Using OAuth with Saturn

Fri, 03 Aug 2018 00:00:00 +0200

Using OAuth with Saturn

Saturn is new F# web framework that provides flexible, high level model of creating web applications using principles of functional programming and MVC architectural pattern. Main design goals of Saturn includes high level abstractions that lets developers focus on creating business code , and general developer experience. One of the features provided by Saturn is support for popular way of authentication - OAuth

This blog post was created for Saturn version 0.7.5

OAuth intro

OAuth 2 is an authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Twitter, GitHub, or DigitalOcean. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account. OAuth 2 provides authorization flows for web and desktop applications, and mobile devices.

While OAuth supports multiple different types of authorization, useful for different use cases, here we will talk about authorization code flow. The authorization code grant type is the most commonly used because it is optimized for server-side applications (such as web applications using Saturn), where source code is not publicly exposed, and Client Secret confidentiality can be maintained. This is a redirection-based flow, which means that the application must be capable of interacting with the user-agent (i.e. the user’s web browser) and receiving API authorization codes that are routed through the user-agent.

Flow of Authorization Code OAuth grant

First steps

The first step in creating application using OAuth is registering application in the OAuth provider. In this example I’ll use GitHub.

To start off with, you will need to register an application by going to the GitHub Developer Settings. Click on the button to Register a new application, and complete the information for your application. Specify http://localhost:5000/signin-github as the value for the Authorization callback URL field:

Register a new OAuth application

Take note of the values for Client ID and Client Secret, that will be shown on the next screen after pressing Register application button, as you will need those shortly when registering the OAuth.

Path goes forward

Next step is creating new Saturn application. This can be easily done using dotnet CLI tool.

mkdir oauthBlogPost
cd oauthBlogPost
dotnet new saturn

If you don’t have Saturn template installed you can get it by running dotnet new -i Saturn.Template

Next step is installing Saturn.Extensions.Authorization - NuGet package that contains additional helpers for handling authorization, including GitHub provider.

Open paket.dependencies and src\oauthBlogPost\paket.references file and add Saturn.Extensions.Authorization in there. After it’s done run .paket\paket.exe install and dotnet restore.

If you’re on Linux / MacOS you will need mono to run paket.exe

Diving deep

Now, at last, it’s time to look at some code. Open Program.fs file that contains basic configuration of your Saturn application. Firstly, edit URL to use port 5000. Secondly add new entry in the application block - use_github_oauth. This custom operation takes couple of input parameters - Client ID, Client Secret, Callback URL, and the list of tuples used to mapping from GitHub’s JSON response to the claims that Saturn understands.

Second step is creating pipelines that will handle or require authentication. Create User.fs file, and add it to .fsproj file above the Router.fs.

The matchUpUsers function mock ups check of the data coming from GitHub with your real users database, loggedIn pipeline forces authentication on the request, and isAdmin pipeline checks if the request is done by someone with Admin role.

Now, it’s time to plug it into our routing. Open Router.fs file. We will create additional router that will display two views - one for logged in user, second one for the admin. Now you need to plug this new router into our top level router - browserRouter . The last thing you need to do here is also handling the URL that will be called by GitHub after successful authentication. We will redirect it into logged user view.

Last step is creating some HTML views that we will return to user. Under Templates folder, create UserView.fs file and add it to .fsproj as last file in Template/ group.

Light at the end

It’s time to run our application. But there is one small step you still need to do. Go to src\oauthBlogPost folder and run from terminal dotnet saturn migration

This step is necessary because dotnet new saturn creates pretty complex application scaffold including connection to local database

Now, from root of your repository, you can run .\build.sh Run which will start the application. You can go to http://localhost:5000/ to see standard startup page of the scaffolded application

If you will try to go to http://localhost:5000/members-only URL, you will get redirected to GitHub authorization page.

And after authorization, you’ll get redirected back to our application.

You can also try going to http://localhost:5000/members-only/admin . If you are me (or you’ve updated the code ;-) ) you should see the admin page.

End of the way

In this post I’ve shown how to introduce OAuth authorization to your Saturn application. As you could see Saturn is using high level, declarative syntax that enables you to plug features like that easily into your application. Currently Saturn.Extensions.Authorization provides predefined authorization providers for Google and GitHub, but more will come to it soon. Feel free to PR your favorite OAuth provider!

Introducing Saturn on Functions

Sat, 21 Jul 2018 00:00:00 +0200

Introduction

Azure Functions is a service that provides serverless execution model while running code in the cloud. Serverless model is getting more and more popular nowadays, being great solution for building distributed, scalable, event-driven applications. However, those are not the only use cases - compute-on-demand story, automatic, low friction scalability and great pricing model means that Azure Functions can be used to host normal (REST-ish) APIs.

On the other hand, Saturn is new F# web framework that provides flexible, high level model of creating web applications using principles of functional programming and MVC architectural pattern. Main design goals of Saturn includes high level abstractions that lets developers focus on creating business code , and general developer experience.

Saturn on Functions

Today I want to introduce new extension to Saturn that adds ability to easily host Saturn controllers and routers inside Azure Functions HTTP triggers. Saturn is a library built on top of Giraffe and ASP.NET Core which means it can easily integrate with existing .Net ecosystem. HTTP triggers in Azure Functions as one of the input parameters are getting standard HttpRequest object that can be passed into Saturn’s controller or router (and any other HttpHandler).

Computation Expression

To reduce amount of boilerplate required to call any HttpHandler and provide, in Saturn’s spirit, opinionated way of hosting your controllers in Azure Functions we’ve created Saturn.AzureFunctions project that adds new computation expression - azureFunction

Example:

let testCntl = controller {
    index (fun ctx -> Controller.text ctx "Hello world")
    show (fun ctx id -> id |> sprintf "Hello world, %s" |> Controller.text ctx)
}
let func log = azureFunction {
    host_prefix "/api"
    use_router testCntl
    logger log
    error_handler customErrorHandler
    not_found_handler customNotFoundHandler
}

azureFunction CE provides set of custom operations that can be used to configure Azure Functions adapter:

host_prefix - prefix of the routes used by Azure Functions. By default Functions are using /api prefix.
use_router - plugs HttpHandler that will be used. Setting it is required.
logger - enables passing TraceWriter into your HttpHandler . It’s passed into your functions by ctx.Items.["TraceWriter"] property. It’s also used for logging error in default error handler.
error_handler - enables plugging custom error handler for unhandled exceptions.
not_found_handler - enables plugging custom handler for not matched requests.

Using it together

azureFunction CE is transformed into a function that accepts HttpRequest as an input and returns Task<HttpResponse> - something that is understood by Functions runtime, and as such can be easily used in normal Azure Functions endpoint.

For example using azureFunction defined above would look like this:

[<FunctionName("HelloWorld")>]
let helloWorld ([<HttpTrigger(Extensions.Http.AuthorizationLevel.Anonymous, Route = "{route?}")>]req: HttpRequest, log: TraceWriter) =
    func log req

Summary

In this post I’ve presented new way of hosting your Saturn applications - using Azure Functions. As you can see Saturn will provide easy to use, opinionated way to embed Saturn controllers or routers in Azure Functions providing great, alternative way of hosting your web applications.

Saturn.AzureFunctions is currently still developed - if you’d like to check out current implementation, test it, or provide some feedback feel free to check this PR - https://github.com/SaturnFramework/Saturn/pull/121