Tag Archives: SharePoint

Using Power BI to Report on Lookup Fields in SharePoint

This post is the sixth and final post in a series exploring Power BI and complex data types in SharePoint. This post examines the various options in Power BI for working with lookup fields. The previous posts are:

A lookup field in SharePoint contains values looked up from another list in the same SharePoint site. Strictly speaking, the field contains only the ID from the item in the source list, and the value(s) is/are looked up whenever the field is displayed. The lookup field can also be used to display multiple field values from the target list items.

Defining a lookup field in SharePoint

The List

Consider the following list that contains a lookup field named “Neighbourhood”:

The lookup field neighbourhood in the SharePoint view

We can see from the screenshot above that the text value for neighbourhood is displayed in the view, although only the row identifier is stored in the column. We will be able to get both values and more if desired in a Power BI report, but first we need to build the report using Power BI Desktop.

Loading the Data

We first launch Power BI Desktop, select “Get Data” and then choose SharePoint Online list (if connecting to SharePoint Online) or SharePoint List (if using SharePoint Server). We are then prompted for the URL of the SharePoint Site. The dialog is titled SharePoint lists, but the value is the URL of the site, NOT the list itself. Once this is entered we are prompted for credentials if we haven’t connected to this site before. After entering credentials, we can select the list that we want to report on. In our case, it’s named “Listings”. We select it, and then click on the Edit button.

Loading data rom the Listings list

Once the data loads in, one of the first things that you’ll notice is that there are a lot of columns to choose from, and it’s a good idea to remove the columns that you don’t need. We can do this by right clicking on the desired column titles and selecting “Remove”. In this case, we can remove the ContentTypeId column and everything to the right of it, with two important exceptions. We want to keep the “FieldValuesAsText” in addition to the special “Neighbourhood” column at the far right of the columns, as we’ll be needing them for our options below.

Using FieldValuesAsText

Examining our columns, we can see that amongst the simpler text fields, we don’t have a “Neighbourhood” column, but instead, a “NeighbourhoodId” column, with numeric values. We do have a Neighbourhood column further off to the right, but it doesn’t display simple text (we’ll come back to this shortly). If we simply want the text value of our lookup target, we can use the “FieldValuesAsText” column quickly.

Scrolling right in the Query editor view, we find the “FieldValuesAsText” column. The record values represent a one to one relationship with the text values of the list row, so we can click on the column expander at the right of the column title. From there we can extract the text value of any column, including our lookup field, “Neighbourhood”.

Extracting text using FieldValuesAsText

With “Neighbourhood” checked, and nothing else, including the “Use original column name..” option, we can click OK, and the “FieldNameAsText” column is replaced by a new column, “Neighbourhood” that contains the text values for Neighbourhood.

If this value is all that is needed, then this is a totally valid approach, and we can move on to report building. However, this is only one way to achieve this goal. If more information is needed, then other methods may be more suitable.

Retrieving all Lookup Field Values from the Extended Column

Given that the lookup target item is a SharePoint list item, all that item’s properties are available to us. We can access them from the extended column set up for the field. In our case, the original “Neighbourhood” column is the extended column. We can expand this column by selecting its column expander.

Extracting field values using the extended column

We then deselect all of the columns except the ones that we want to use in the report. The fields available are the fields available in the target list. In our case, we select the “Title” field, as it is the one being looked up. We can however retrieve any of the fields that we need from the target list.

Keep in mind that “Title” in our example is a simple text field, so no further action is necessary. The retrieved fields can be complex (person, MMS, etc), but keep in mind that if a complex field type is retrieved, it will need to be transformed just like any from the list in question.

The field name in the target list may not adequately describe its function for the report. In our case, “Title” actually means “Neighbourhood” in this report. It’s a good idea to rename it.

Finally, if multiple field values are to be retrieved, the data model could grow significantly. This is because the values for every field are repeated in every row of data. Given that the original lookup column adds a measure of relational behaviour to SharePoint, using this relationship is the most efficient way to work with this data. Power Query allows us to do just that.

Working with Related Tables

To work with related tables, we need not only the original data table (in our case, “Listings”) but also the table for the lookup list itself. To do this, from the Query Editor, we create a new data source like the one created above for “Listings”, but instead we select the lookup list (“Neighbourhoods”).

Once imported, we can remove any extraneous columns, and then set the data type for the ID field to be “Whole Number”.

Setting the ID column to whole number

We also need to set the data type of the “NeighbourhoodId” column in the Listings table to “Whole Number”. Once these options have been set, we are ready to work with the data model and the report. We select “Close and Apply” from the ribbon to load the data into the model.

Once loaded, we launch the relationship builder in the design pane in order to establish the relationship between the two tables.

Power BI relationship editor with default relationship defined

We can see that Power BI has already detected a relationship. However, it is not correct. The model designer assumes that because both tables contain an “id” column, then they must be related. However, the true relationship is between the “Id” column in our “Neighbourhoods” table, and the “NeighbourhoodId” column in our “Listings” table.

We must first delete the detected relationship by selecting the connector between the two tables and pressing “Delete”. We can then create the proper relationship by dropping one of our related columns onto the other. Once this is created, we also need to ensure that the “Cross filter direction” is set to “Both”. We do this by double clicking on the relationship connector and selecting the appropriate option.

Setting cross filter direction

Power BI relationship editor with correct relationship defined

Once the relationship has been established, we can return to the design pane and construct a rudimentary report. We drag a few fields from Listings into a table, create a calculated measure for the number of listings, and we add the “Title” field (renamed to “Neighbourhood”) to the canvas separately. Once we set the visualization for “Neighbourhood” to a slicer, we can easily slice our listings data by neighbourhood.

Slicing report with the values from a lookup field

We can therefore see that there are several options for accessing data for a lookup field, ranging from simple to complex. The trade-off for simplicity is flexibility. Which approach used will depend on your requirements but storing the lookup table separately is the most efficient as the data is only stored once and referenced.

Using Power BI to Report on Hyperlink or Picture Fields in SharePoint

This post is the fifth in a series exploring Power BI and complex data types in SharePoint. This one details the use of SharePoint Hyperlink or Picture fields. The previous posts are:

A hyperlink or picture field in SharePoint consists of a name-value pair. The value is always a URL, and the name is the descriptor for that URL. When the field is created, the creator can specify the character of the field, whether it is a hyperlink, or a picture.

Hyperlink or Picture field in SharePoint

If picture is chosen, then all values will be treated as images, and SharePoint renders them as such wherever displayed, in forms, views, etc. The name part of the name-value pair is used as the alt tag for the image when it is written. If hyperlink is chosen, the name portion will be displayed wrapped in a link to the value. These behaviours are particularly suitable to the way that Power BI works with both link and images, as we’ll see shortly. In our example below, we’ll be working with a list that contains 2 instances of this field type, one configured as a picture, and the other as a hyperlink.

The List

Consider the following list that contains two of these fields. The first named “Picture” is not surprisingly configured as a picture type, and the other, “More Info” is configured as a hyperlink:

Both Picture and hyperlink fields in a SharePoint view

This view renders a thumbnail of the image for the “Picture” field, and a clickable hyperlink using the link name for the “More Info” field. Behind the scenes however, the data is simply stored as that name-value pair. We will be able to get both field types to render with appropriate behaviours in a Power BI report, but first we need to build the report using Power BI Desktop.

Loading the Data

We first launch Power BI Desktop, select “Get Data” and then choose SharePoint Online list (if connecting to SharePoint Online) or SharePoint List (if using SharePoint Server). We are then prompted for the URL of the SharePoint Site. The dialog is titled SharePoint lists, but the value is the URL of the site, NOT the list itself. Once this is entered we are prompted for credentials if we haven’t connected to this site before. After entering credentials, we can select the list that we want to report on. In our case, it’s named “Listings”. We select it, and then click on the Edit button.

Loading the Listings data

Once the data loads in, one of the first things that you’ll notice is that there are a lot of columns to choose from, and it’s a good idea to remove the columns that you don’t need. We can do this by right clicking on the desired column titles and selecting “Remove”. In this case, we can remove the ContentTypeId column and everything to the right of it, with two important exceptions. We want to keep the “FieldValuesAsText” column, as we’ll be needing that to extract the text values.

Using FieldValuesAsText

In our example, both the “Picture” and “More Info” fields are displayed with a linkable value of “record” for every row. We will explore using these columns below, but to use FieldValuesAsText, it is best to remove them to avoid naming conflicts. As with most complex field type, the “FieldValuesAsText” column can be used to extract the URL for the Hyperlink or Picture field.

We scroll right and select the expander icon for the “FieldValuesAsText” column, then deselect all available fields except the “Picture” and “More Info” columns. In addition, we want to ensure that the “Use original column name as prefix” option is deselected to avoid a lot of messy renaming later.

Retrieving data with FieldValuesAsText

We then select OK, and two new columns are added in place of FieldValuesAsText named “Picture” and “More Info”. These columns contain the value portion of the name-value pair that make up the Hyperlink or Picture field, but the name portion is dropped.

Raw data for the Hyperlink or Picture Field

If all that is needed for the report is the URL portion, then this approach is sufficient, and you can continue working with the data model and report as below. However, to retrieve both the name and the value from the field, an alternate method is required.

Retrieving all Field Values

Instead of removing the “Picture” and “More Info” columns as described in the previous section, retrieving all of the values requires us to use them. In this case, we can safely remove the “FieldValuesAsText” column first, as it won’t be needed. The “Record” values shown for the field value on each row are Power BI’s way of expressing a one-to-one relationship. In this case, each relationship is with a record that has 2 fields, “Description” and “Url”. To use them in a report, they must first be flattened. We do this by selecting the column expander in the upper right of the column title, selecting both fields, and clicking OK.

Expanding the Hyperlink or Picture field

All Hyperlink or Picture fields will have the same properties, and in our case, this needs to be done for both the “Picture” and “More Info” fields. Because of this, it’s likely a good idea to check the “Use original column names as prefix” box to help keep everything straight. The columns can be renamed at any time if desired. Once This is done for both columns, we will see both the description, and the actual URL value for both of our “Hyperlink or Picture fields, as seen below:

Extracting all Hyperlink or Picture properties

At this point, we are ready to load the data into the data model by selecting the “Close and Apply” button from the ribbon. Once loaded, we are placed into the report design canvas. From here, we need to do a small amount of model editing.

Using Picture or Link Data in the Report

We can add a new table to the design surface, and then add “Picture.Url” to the table. We can quickly see that the default behavior is not optimal – it only displays the URL, not the rendered picture.

Raw picture data in a Power BI report

This is because the data model only knows the contents of the field to be text. We need to tell the model that this is a picture, and we can do that by selecting the Model tab in the ribbon, selecting the field in the field selection pane,

Setting the field properties in Power BI

Once flagged in this manner, the images will automatically render as images whenever they are used in tables, and several other visuals.

Picture field rendering in a Power BI table

The hyperlink field must be categorized in a similar fashion as the picture field, with one difference – instead of Image URL as the data category, we pick Web URL. Once we have done this, we can add it to our table above along with a couple of other fields, including the link description.

Adding Web URL data to the Power BI table

The hyperlinks are active and clickable, but they’re not the nicest to look at. They also take up a significant amount of space on the visual. Happily, there is a table feature that we can take advantage of to help us with this. To turn it on, open the table properties pane (the paint roller), open the Values section, and turn on the option for “URL icon” . All of the long links in the table will be reduced down to a compact link icon.

Web URL data formatted as an icon

Ideally, I would like to be able to recombine the link description and the link in the visual, the same way that it is rendered in SharePoint. However, this does work well, and it lends us a nice level of interactivity in our reports.

As we can see above, the SharePoint “Hyperlink or Picture” field is not only available to Power BI, but much of its utility can also be brought forward into Power BI reports.

Using Power BI to Report on Rich Text Data Fields in SharePoint

This post is the fourth in a series exploring Power BI and complex data types in SharePoint. It discusses working with using Power BI to report on rich text fields from SharePoint. The previous posts are:

A rich text field in SharePoint is a special instance of the multi-line column type which contains formatting attributes. The column becomes “rich” when either the “Rich text” or “Enhanced rich text” options are selected in the field’s definition.

SharePoint rich text fields

The List

Consider the following list that contains a rich text field named “Description”:

Rich text field displayed in SharePoint view

This view displays the value of the rich text field and retains its formatting. Internally all of the formatting commands are stored as HTML, rendering is a simple task for SharePoint. However, None of the default Power BI visuals support HTML rendering. We are left then with two options. We must either retrieve the raw text from the rich text field, and lose the formatting, or find a visual that supports HTML rendering. Happily, both options are possible. As with any SharePoint data, we must first start with Power BI Desktop.

Loading the Data

We first launch Power BI Desktop, select “Get Data” and then choose SharePoint Online list (if connecting to SharePoint Online) or SharePoint List (if using SharePoint Server). We are then prompted for the URL of the SharePoint Site. The dialog is titled SharePoint lists, but the value is the URL of the site, NOT the list itself. Once this is entered we are prompted for credentials if we haven’t connected to this site before. After entering credentials, we can select the list that we want to report on. In our case, it’s named “Listings”. We select it, and then click on the Edit button.

Loading the Listings data

Once the data loads in, one of the first things that you’ll notice is that there are a lot of columns to choose from, and it’s a good idea to remove the columns that you don’t need. We can do this by right clicking on the desired column titles and selecting “Remove”. In this case, we can remove the ContentTypeId column and everything to the right of it, with two important exceptions. We want to keep the “FieldValuesAsText” column, as we’ll be needing that to extract the text values.

Extracting Plain Text from Rich Text

One thing that you will notice right away is that he more simple column types like “Title” show their value directly in the Query editor. Rich text fields also show their values directly in the editor, but they include both the text and the html formatting commands. The rich text field in our example is named “Description”, and each entry begins with “<div class=…..”.

Rich text field contents in Power BI

Given that none of the standard Power BI visuals support HTML rendering, this is clearly not what we need for our report.

We could perform a series of text substitutions to strip out all the HTML formatting from the column, but that process would be highly tedious, not to mention messy. Happily, The Power BI SharePoint connector can do this for us automatically through the FieldValuesAsText column.

First, we can remove the Description column altogether. Next, with our example in the Query Editor, we scroll right and select the expander icon for the “FieldValuesAsText” column. We then then deselect all available fields except the “Description” column. In addition, we want to deselect the “Use original column name as prefix” option.

Expanding FieldValuesAsText

We then select OK, and we once again have a Description column, but this time it is free of all HTML formatting tags.

Description field with all formatting removed

At this point, we can click on “Close & Apply” in the ribbon, add a table to the design surface, and add a number of dimensions, including our “Description” field. It is displayed in the visual free of the HTML formatting.

Unformatted rich text in a Power BI table

Showing Rich Text in Full Fidelity

There may be cases where we want to use the rich text formatting, and not remove it. As mentioned above, that’s not possible using the out-of-box visuals. We are therefore left to find a custom visual to do this, and at the moment, there is only one such visual to the best of my knowledge. This is the HTML Viewer visual by Pragmatic Works, and its purpose is to do exactly as the name suggests.

To begin with, we need to start at the beginning of the previous section, prior to the removal of the Description column. In this case, we want to use the HTML codes, so the initial “Description” column is perfectly adequate as-is. All that we need to do is to select “Close and Apply” in the ribbon to load the data into the data model.

We must now get the HTML Viewer from the custom visuals marketplace. We click on the ellipsis in the visuals pane, select “marketplace” and then search for the visual using the search box. Once located, we select it, and click “Add”.

Acquiring the HTML Viewer

A new icon for the visual then appears in the gallery.

The HTML viewer displays and renders the values as a list. Only one dimension is allowed, so it is not possible to use it as a replacement for a table. The best way to use this visual is to make it the target of a selection. For example, to see the description of our listings, we can add a slicer on the page using “Title” and the HTML Viewer using “Description”.

Full fidelity HTML in a Power BI report

While it is limited, it is possible to render rich text fields from SharePoint in full fidelity. However, if only the text is necessary (as is likely the case for reporting), Power BI gives us a rich set of tools to make this process relatively painless.

Using Power BI to Report on Person Fields in SharePoint

This post is the second in a series exploring Power BI and complex data types in SharePoint. The first post explores working with multi-value columns. In this one, we’ll explore some of the nuances of working with person fields

Person fields in SharePoint are just a special case of the lookup field, and the Power BI SharePoint list connector is aware of them. As such, it provides helpers to make it relatively easy to get the person’s name. However, more information is also available. We’ll examine three approaches to extracting this information. It is worth noting that all SharePoint lists contain person fields for “Created By” and “Modified By”, and they are always available.

The List

Consider the following list that contains a multi-value choice field named Amenities:

The view displays the person’s name, although the column is a complex data type. There is more information than just the person’s name available behind it but this is unavailable to the SharePoint view. Power BI can however access this information in reports. Report requirements will ultimately dictate the best approach to extracting this information, but the good news is that there are several to choose from. In all cases the data first needs to be brought into Power BI Desktop.

Loading the Data

We first launch Power BI Desktop, select “Get Data” and then choose SharePoint Online list (if connecting to SharePoint Online) or SharePoint List (if using SharePoint Server). We are then prompted for the URL of the SharePoint Site. The dialog is titled SharePoint lists, but the value is actually the URL of the site, NOT the list itself. Once this is entered we are prompted for credentials if we haven’t connected to this site before. After entering credentials, we can select the list that we want to report on. In our case, it’s named “Listings”. We select it, and then click on the Edit button.

Once the data loads in, one of the first things that you’ll notice is that there are a lot of columns to choose from, and it’s a good idea to remove the data that you don’t need. In this case, we can remove the ContentTypeId column and everything to the right of it, with two important exceptions We want to keep the “FieldValuesAsText” and “Agent” columns (we’ll come back to that shortly). Remember, for our purposes here, we want to report on the person column, “Agent”. The simplest way to represent this data is with the person’s full name, as it is displayed in the SharePoint view. As noted above, it is also possible to use this data in a more sliceable, or structured way. Let’s start with the simplest.

Extracting the Full Name

One thing that you will notice right away is that he more simple column types like “Title” show their value directly in the Query editor. In our case, there are two fields related to Agent, the “AgentId” and “Agent” columns. The Agent ID column displays a number, and the “Agent” column displays a record data type. We will explore these columns, but if all we need is the user’s full name, we can use the highly useful “FieldValuesAsText” Column.

We scroll right and select the expander icon for the “FieldValuesAsText” column, then deselect all available fields except the Agent column.

We then select OK, and rename the column to “Agent Name”. The Full name of the Person is retrieved and used for the column. At this point, it’s ready to use in a report.

Linking to the User Information List

In many cases, the user name of the person may not be enough. As mentioned above, the Person Field is really just a lookup column that is automatically looking up data from a specific list. That list is the User Information list which is a hidden list that exists in the root site of every SharePoint site collection. This list gets populated automatically when the site collection is accessed. When Power BI loads a person column, it automatically creates a ColumnNameId column as well containing the ID value of the person field from this list. In our example, this is the “AgentId” column.

To leverage the data in this list it must first be loaded into the model. Following the same steps taken for loading the Listing data above, we select the “User Information List” which does get exposed to the Power BI Query editor. Once loaded, we remove all of the unnecessary columns from the query, being sure that we leave the ID column.

When ready, we select the “Close and Apply” button from the Query Editor Ribbon. At this point, we have two tables in the model, Listings and User Information List. We then select the relationship editor tab. The “AgentId” column in the Listings table is related to the “id” column un the User Information list table, and we establish this relationship by dragging one onto the other. Once established, we double click on the relationship line to set the value of “Cross filter direction” to “Both”.

We can now return to the design pane, add a table visual, and add columns from both tables. In such a way, we can show the agent’s name, email, phone, etc.

Expanding the Person Column

Although linking to the User Information List is powerful, and easier, and arguably better way to do the same thing is to use the automatically generated person column. This column is named the same as the original person column and contains a series of “Record” type values. The records in question are the corresponding records from the User Information List.

To access the data in this column, we click on the column expander and then select all of the columns that we will work with. Values from the related User Information List will be added to the table automatically.

This approach is clearly simpler than manually loading the entire User Information List, and only loads the records that are related. It will however likely result in a large amount of repeated data that the two table approach avoids. It is possible to achieve a two table solution with the person field using the technique outlined in my earlier article on working with multi-value fields, but the resultant table will still only contain related records. If it is necessary to show people regardless of whether or not there is a related record, then the manual approach is the only way.

Which approach is ultimately used will depend on the requirements of the report, but it is possible to reach deep into the person object in a SharePoint person field.