assignment 3 building a custom visualization

Nick Malinowski's Portfolio

My data science portfolio, assignment 3 - building a custom visualization.

In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work. The options differ in challenge level, but there are no grades associated with the challenge level you chose. However, your peers will be asked to ensure you at least met a minimum quality for a given technique in order to pass. Implement the technique fully (or exceed it!) and you should be able to earn full grades for the assignment.

In this paper the authors describe the challenges users face when trying to make judgements about probabilistic data generated through samples. As an example, they look at a bar chart of four years of data (replicated below in Figure 1). Each year has a y-axis value, which is derived from a sample of a larger dataset. For instance, the first value might be the number votes in a given district or riding for 1992, with the average being around 33,000. On top of this is plotted the 95% confidence interval for the mean (see the boxplot lectures for more information, and the yerr parameter of barcharts).

A challenge that users face is that, for a given y-axis value (e.g. 42,000), it is difficult to know which x-axis values are most likely to be representative, because the confidence levels overlap and their distributions are different (the lengths of the confidence interval bars are unequal). One of the solutions the authors propose for this problem (Figure 2c) is to allow users to indicate the y-axis value of interest (e.g. 42,000) and then draw a horizontal line and color bars based on this value. So bars might be colored red if they are definitely above this value (given the confidence interval), blue if they are definitely below this value, or white if they contain this value.

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Easiest option: Implement the bar coloring as described above - a color scale with only three colors, (e.g. blue, white, and red). Assume the user provides the y axis value of interest as a parameter or variable.

Harder option: Implement the bar coloring as described in the paper, where the color of the bar is actually based on the amount of data covered (e.g. a gradient ranging from dark blue for the distribution being certainly below this y-axis, to white if the value is certainly contained, to dark red if the value is certainly not contained as the distribution is above the axis).

Even Harder option: Add interactivity to the above, which allows the user to click on the y axis to set the value of interest. The bar colors should change with respect to what value the user has selected.

Hardest option: Allow the user to interactively set a range of y values they are interested in, and recolor based on this (e.g. a y-axis band, see the paper for more details).

Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different. __ __ __ __ __ __ __ __ __ __ ___

Randomizing data

Use the following data for this assignment:

assignment 3 building a custom visualization

I had to transpose the data to use it in my graph.

Calculating mean and standard deviation

I calculated the mean and standard deviation for each graph per the requirements.

Creating confidence intervals

The next step was creating the confidence intervals.

Creating the graph

I normalized the curve and used a colormap for the data.

Now to plot the data and add labels.

Assignment 3 - Building a Custom Visualization ¶

Ferreira, N., Fisher, D., & Konig, A. C. (2014, April). Sample-oriented task-driven visualizations: allowing users to make better, more confident decisions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 571-580). ACM. ( video )

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Hardest option: Allow the user to interactively set a range of y values they are interested in, and recolor based on this (e.g. a y-axis band, see the paper for more details).

Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different.

4 rows × 3650 columns

Assignment 3 - Building a Custom Visualization ¶

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Hardest option: Allow the user to interactively set a range of y values they are interested in, and recolor based on this (e.g. a y-axis band, see the paper for more details).

Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different.

Assignment 3 - Building a Custom Visualization ¶

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Hardest option: Allow the user to interactively set a range of y values they are interested in, and recolor based on this (e.g. a y-axis band, see the paper for more details).

Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different.

4 rows × 3650 columns

Assignment 3

This assignment is due on Tuesday, May 25 2021 at 11:59pm PST.

Starter code containing Colab notebooks can be downloaded here .

Q1: Image Captioning with Vanilla RNNs (30 points)

Q2: image captioning with transformers (20 points), q3: network visualization: saliency maps, class visualization, and fooling images (15 points), q4: generative adversarial networks (15 points), q5: self-supervised learning for image classification (20 points), extra credit: image captioning with lstms (5 points), submitting your work.

Please familiarize yourself with the recommended workflow before starting the assignment. You should also watch the Colab walkthrough tutorial below.

Note . Ensure you are periodically saving your notebook ( File -> Save ) so that you don’t lose your progress if you step away from the assignment and the Colab VM disconnects.

While we don’t officially support local development, we’ve added a requirements.txt file that you can use to setup a virtual env.

Once you have completed all Colab notebooks except collect_submission.ipynb , proceed to the submission instructions .

In this assignment, you will implement language networks and apply them to image captioning on the COCO dataset. Then you will explore methods for visualizing the features of a pretrained model on ImageNet and train a Generative Adversarial Network to generate images that look like a training dataset. Finally, you will be introduced to self-supervised learning to automatically learn the visual representations of an unlabeled dataset.

The goals of this assignment are as follows:

Understand and implement RNN and Transformer networks. Combine them with CNN networks for image captioning.
Explore various applications of image gradients, including saliency maps, fooling images, class visualizations.
Understand how to train and implement a Generative Adversarial Network (GAN) to produce images that resemble samples from a dataset.
Understand how to leverage self-supervised learning techniques to help with image classification tasks.

You will use PyTorch for the majority of this homework.

The notebook RNN_Captioning.ipynb will walk you through the implementation of vanilla recurrent neural networks and apply them to image captioning on COCO.

The notebook Transformer_Captioning.ipynb will walk you through the implementation of a Transformer model and apply it to image captioning on COCO.

The notebook Network_Visualization.ipynb will introduce the pretrained SqueezeNet model, compute gradients with respect to images, and use them to produce saliency maps and fooling images.

In the notebook Generative_Adversarial_Networks.ipynb you will learn how to generate images that match a training dataset and use these models to improve classifier performance when training on a large amount of unlabeled data and a small amount of labeled data. When first opening the notebook, go to Runtime > Change runtime type and set Hardware accelerator to GPU .

In the notebook Self_Supervised_Learning.ipynb , you will learn how to leverage self-supervised pretraining to obtain better performance on image classification tasks. When first opening the notebook, go to Runtime > Change runtime type and set Hardware accelerator to GPU .

The notebook LSTM_Captioning.ipynb will walk you through the implementation of Long-Short Term Memory (LSTM) RNNs and apply them to image captioning on COCO.

Important . Please make sure that the submitted notebooks have been run and the cell outputs are visible.

Once you have completed all notebooks and filled out the necessary code, you need to follow the below instructions to submit your work:

1. Open collect_submission.ipynb in Colab and execute the notebook cells.

This notebook/script will:

Generate a zip file of your code ( .py and .ipynb ) called a3_code_submission.zip .
Convert all notebooks into a single PDF file called a3_inline_submission.pdf .

If your submission for this step was successful, you should see the following display message:

### Done! Please submit a3_code_submission.zip and a3_inline_submission.pdf to Gradescope. ###

2. Submit the PDF and the zip file to Gradescope .

Remember to download a3_code_submission.zip and a3_inline_submission.pdf locally before submitting to Gradescope.

Syllabus (post-hoc)
Assignments

Visualization10 - CS838 Course Web

Course web for CS838 Spring 2010, Visualization

Assignment 3: Visualization Research

January 23, 2010

in Assignments

assignment due Thursday, January 28th (please post your comments before 9am so we can read them before class)

The goal of this assignment is to give you an idea of what is going on in “Visualization Research” as a Computer Science Discipline. This is only one perspective on visualization, and this will give you a particular slice of it, but its better than nothing.

The premiere academic venue for Visualization (as a computer science sub-area) is IEEE “VisWeek.” Its a set of 3 conferences that are co-located. The proceedings are published as a special issue of IEEE Transactions on Visualization and Computer Graphics. The event is evolving. Its usually in October.

The past few years, there’s been 3 events “Vis,” “InfoVis,” and “Vast” (Visual Analytics Science and Technology). The most recent one (2009) was in Atlantic City this past October.

The goal of this assignment is to give you an idea of what kinds of things go on at this venue (as a way of sampling what “Visualization Research” is.

Your task is to look through the “proceedings” of the “conference” (really the 3 co-located events) and see what catches your eye. Of course, this being the modern era, you won’t actually look at the printed proceedings (they don’t even give it out at the conference – they give out a USB stick). One down side is that printed proceedings are great to flip through for this kind of purpose, and online proceedings are less skimmable. You don’t need to read the papers, but I want you to get a sense of what kinds of topics are there (and might be interesting to you). If you had the printed proceedings, you could flip through and see what pictures stood out.

What you should do (the resources for doing this are below):

Look over all the titles, see what catches your eye.
For some subset of those, look a little more closely. Read the abstract, look at the pictures, maybe the author has a website or something…
Pick a few of your favorites. Between 3-5. At least one must come from InfoVis, at least one must come from Vis. Give your list as a comment to this message. Please either remember your list or bring it to class.

Without the printed proceedings, your resources for doing it:

The VGTC website . VGTC is the committee of IEEE that organizes the conferences. They have a great website. For example, at this page you can see a list of all the papers, links to the abstracts, and links to the slides from most of the talks from Vis09. (there’s a similar page for infovis).
The graphics papers on the web resource has links for Vis09 and InfoVis09 . These are unofficial, but they usually have links to either the author’s web pages or the project web pages, where you can find more info (and even the PDF of the paper).
The official digital library page . Most useful to get the actual papers. We have a campus-wide subscription (so either access it with a campus IP address, or use the library’s proxy server).
If you’re off campus, you need to access the IEEE DL via a proxy. I think this works .

All you need to do is add a list of 3-5 papers as a comment to this posting, and come prepared to talk about what you’ve found. Again, I don’t expect you to actually read any complete papers (you are welcome to, but there will be plenty of time for that later in the semester) – but I do want you to get a sense of the range of topics that people are writing about.

Addendum: the digital library, while inconvenient, is the only real way to get the papers reliably and officially. Many authors put copies of the papers on their personal or group websites, but not everyone (and its unclear with the IEEE copyright agreement if this is a legal thing to do. it is OK with ACM).

Addendum 2: I understand that the papers people pick will be biased towards those that are convenient to find. There is no notion that this will be an unbiased sampling.

{ 20 comments }

Actually: you can follow links through the proxy server even while you’re on-campus. And you can try sending a URL through the proxy server by appending “ezproxy.library.wisc.edu” to the hostname.

So the digital library page’s URL, proxied, becomes:

http://ieeexplore.ieee.org.ezproxy.library.wisc.edu/xpl/tocresult.jsp?isYear=2009&isnumber=5290686&Submit32=View+Contents

General Notes ————-

a. I found InfoVis to be more interesting than Vis.

b. Since not even the abstracts were easily accessible for most of the papers in Vis, I ignored everything past Session 4.

c. The word “novel” is used by most authors to describe their own presentation. This must be a fairly overused term in the vis community, or perhaps in the computing community in general.

d. I understand that every discipline has its own jargon, fully understood only by its own tribe, but this community seems to be particularly jargon prone. Consider —

“Capitalizing on recent advances in matrix approximation and decomposition, ( Exemplar-based Visualization) presents a probabilistic multidimensional projection model in the low-rank text subspace with a sound objective function.”

Good lord! Now, my english is pretty good by any standard, but I have no idea what the authors are trying to say there.

InfoVis 2009 ————

1. Protovis: A Graphical Toolkit for Visualization by Bostock and Heer.

Intrigued by this paper because am familiar with Heer’s work, particularly his Vizster application (I believe that was his Ph.D. thesis). Downloaded the Protovis toolkit, but in spite of the logic of the authors on why such a toolkit was needed (declarative vs. imperative programming) in spite of many similar alternatives already out in the wild, I found the tool to be less than compelling.

2. Spatiotemporal Analysis of Sensor Logs using Growth Ring Maps by Bak, Mansmann, Janetzko and Keim.

Tackles a problem familiar to me in my mapping context, that is, how to symbolize overlapping markers (think residents in a multistory condo). I have been contemplating creating a prototype that disaggregates overlapping markers in an animation, much like Google Earth does with overlapping markers, but in a web-based context. Might be worthwhile as a project for this class.

3. Harnessing the Web Information Ecosystem with Wiki-based Visualization Dashboards by McKeon.

I once tried to create a dashboard for a project, but did not quite succeed. Since dashboards, but their nature, provide a controlled view of some complicated activity with many variables and views, it is hard to imagine their creation to be wikified. Hence, I found this paper to be of interest. It is based on ManyEyes, so it is easy to experiment with. I like stuff I can dig into, get my hands dirty. This paper took me there.

Vis 2009 ——–

4. Time and Streak Surfaces for Flow Visualization in Large Time-Var ying Data Sets by Krishnan, Garth and Joy.

Showing motion in a static visualization is a common challenge. The authors present a method for computing such visualizations that they term as “time and streak surfaces.”

gah! Beware, comments can be edited/updated. Once you hit [Submit], its immutable.

where by “comments can be edited” I mean “comments can *not* be edited.”

1. Exploring the Millennium Run – Scalable Rendering of Large-Scale Cosmological Datasets (Vis 2009)

2. Hue-Preserving Color Blending (Vis 2009)

3. Visualizing Social Photos on a Hasse Diagram for Eliciting Relations and Indexing New Photos (InfoVis 2009)

4. Flow Mapping and Multivariate Visualization of Large Spatial Interaction Data (InfoVis 2009)

5. Combining automated analysis and visualization techniques for effective exploration of high-dimensional data (VAST 2009)

1. Focus+Context Route Zooming and Information Overlay in 3D Urban Environments, Vis 2009.

2. Visual Human+Machine Learning, Vis 2009.

3. Hue-Preserving Color Blending, Vis 2009

4. Protovis: A Graphical Toolkit for Visualization, InfoVis 2009.

5. A Nested Model for Visualization Design and Validation, InfoVis 2009.

1. VisMashup: Streamlining the Creation of Custom Visualization Applications Emanuele Santos, Lauro Lins, James P. Ahrens, Juliana Freire, Cláudio T. Silva There wasn’t much information available (i.e. none) but this interested me because so much network visualization software (my major field of interest) is custom, takes considerable time to build, and doesn’t necessarily involve best practices in visualization. 2. Interaction Techniques for Selecting and Manipulating Subgraphs in Network Visualizations, McGuffin, M.J.; Jurisica, I. Much of my work involves making complex networks clear and useful to laypeople. Subnetworks are central to this process, and finding and visualizing subnetworks clearly is an important aspect of this work. 3. “Search, Show Context, Expand on Demand”: Supporting Large Graph Exploration with Degree-of-Interest, van Ham, F.; Perer, A. This is a very useful paper for the inverse of (2) above. For a variety of reasons, I work with whole community networks (community in this context means literal cities). I want to find ways for people to look at the entire community and see clusters or subnets that are useful to them. This involves multiple problems: what *is* useful to laypeople in exploring complex networks; how do you make this useful material stand out and visible; and how does this vary across users classes? 4. ResultMaps: Visualization for Search Interfaces, Clarkson, E.; Desai, K.; Foley, J. This appealed to me because my work involves the question of how people find useful networks in communities, and what is the easiest way “in” to this problem: whole network visualization, subnetworks, simple search, or some combination.

1. Loop surgery for volumetric meshes: Reeb graphs reduced to contour trees 2. Exploring the Millennium Run – Scalable Rendering of Large-Scale Cosmological Datasets 3. code swarm: A Design Study in Organic Software Visualization 4. Towards Utilizing GPUs in Information Visualization: A Model and Implementation of Image-Space Operations 5. Visual Analysis of Inter-Process Communication for Large-Scale Parallel Computing

1. Visual Exploration of Nasal Airflow 2. Scalable and Interactive Segmentation and Visualization of Neural Processes in EM Datasets 3. A Nested Model for Visualization Design and Validation 4. Hue-Preserving Color Blending

A few general notes:

* In general, InfoVis looks like more fun than Vis. * The pages for both sites are surprisingly hard to scan, considering it’s for they’re for vis sites.

And stuff I thought was really neat — or at least, notable:

1: Depth-Dependent Halos: Illustrative Rendering of Dense Line Data

2: ABySS-Explorer : Visualizing Genome Sequence Assemblies

3: Constructing Overview + Detail Dendrogram-Matrix Views

And, harkening back to my library days:

4: Mapping Text with Phrase Nets

I went through the abstracts of the articles I pick. My personal interest is mainly on how to make data telling more. Therefore, I select one with sampling methodology, one with surface descriptor, one with bioinfo background, and one from multidimensional data visualization.

Vis09 1. Sampling and Visualizing Creases with Scale-Space Particles Gordon L. Kindlmann (University of Chicago), Raál San José Estépar (Harvard Medical School), Stephen M. Smith, Carl-Fredrik Westin (Harvard Medical School)

In improving mesh generation and shape analysis, a good sampling from particle system is important. This paper proposes the role of particle systems in sampling structure from unsegmented data and describes a particle system that computes samplings of features effectively represent many anatomical structures in scanned medical data.

2. Intrinsic Geometric Scale Space by Shape Diffusion Guangyu Zou, Jing Hua, Zhaoqiang Lai (Wayne State University), Xianfeng Gu (State University of New York at Stony Brook), Ming Dong

Given any signal by observation, features inherently exist at various scales. Intrinsic Geometric Scale Space (IGSS) is a scale space of the Riemannian surface geometry, and it provides a framework for feature extraction and matching with inherent scale information.

3. Constructing Overview + Detail Dendrogram-Matrix Views Jin Chen, Alan M. MacEachren, Donna J. Peuquet (GeoVISTA Center, The Pennsylvania State University)

The research proposes a strategy that links an overview dendrogram and a detail-view dendrogram, each integrated with a re-orderable matrix. This tree plots strategy could be useful in many research fields.

4. Scattering Points in Parallel Coordinates Xiaoru Yuan, Peihong Guo, He Xiao (Peking University), Hong Zhou, Huamin Qu (The Hong Kong University of Science and Technology)

I am particularly interested in that they present a novel parallel coordinates design integrated with points, by taking advantage of both parallel coordinates and scatterplots.

I guess its only fair if i do this too. Some notes I made while skimming the proceedings (i had a physical copy) are at http://www.cs.wisc.edu/graphics/Wiki/Gleicher/Vis09

Me doing this is a little different, since I have different information (I was at the conference, saw lots of presentations and the fast-forward), and several competing interests (find things for class, find things for my own work, things that just seem interesting)…

I too found the infovis papers more interested in terms of their concepts, but also like some of the more mathematical issues addressed in the vis community.

But some of my “favorites” – OK, I picked too many. * Nested Model for Validation… – good food for thought, we’ll read this * Bubblesets – just a cute idea, and it looks nice – but i have some doubts about how they did it * Qualitiative Texton Sequences – good follow-through on a foundational thing * Route Zooming – I like lots of things about this (problem, approach, details, …) * Interactive Vis of Molecular Surface Dynamics – I saw the talk, and I still don’t think its possible 🙂 but i’m really interesting in the problem. * Structuring Feature Space: A Non-Parametric Method for Volumetric Transfer Function Generation – seems like a good idea for a basic problem. pulls out some appropriately sophisticated mathematics.

I read the following five paper which are interesting and have many applications.

Paper 1: Vis2009 Title : Loop Surgery for Volumetric Meshes: Reeb Graphs Reduced to Contour Trees Authors : Julien Tierny, Attila Gyulassy, Eddie Simon, Valerio Pascucci

Comments: In this paper, authors propose a new technique to construct Reebs graph using “Surgery Technique” and claim that it is 6500 times faster than previous known techniques. Which is true remarkable, but the following sentence “In this case, our technique produces results in matter of seconds even for the largest models” is rather overstatement.

Paper 2 : Vis2009 Title : Applying Manifold Learning to Plotting Approximate Contour Trees Authors : Shigeo Takahashi, Member, IEEE, Issei Fujishiro, Member, IEEE, and Masato Okada Comments: A Contour tree ( or Sketon tree ) encapsulates evolution of scalar field which is used to understand topological properties of a field. In this paper, authors have used a well known nonlinear data reduction technique “Manifold learning” to construct contour trees in higher dimension. They also use hierarchical representation for handling large scale dataset.

Paper 3 : Vis2009 Title : Multi-Scale Surface Descriptors

Authors : Gregory Cipriano, George N. Phillips Jr., Michael Gleicher Comments : In this paper, authors make an attempt to characterize surfaces based on curvature based descriptors. They show use of local shape descriptors in many applications such as shape matching, stylized rendering, visualization and analysis of scientific data.

Paper 4 : InfoVis2009 Title : Search, Show Context, Expand on Demand: Supporting Large Graph Exploration with Degree-of-Interest Authors : Frank van Ham and Adam Perer Comments : Exploring information from large sparse graph is challenging. In this paper, authors believes that by using “degree of interest”, the complexity of graph exploration could be reduced. They have developed a prototype on legal documents and showed that it was useful in extracting relevant documents.

paper 5 : VAST2009 Title : Combining automated analysis and visualization techniques for effective exploration of high-dimensional data Authors : Andrada Tatu, Georgia Albuquerque, Martin Eisemann, Jörn Schneidewind, Holger Theisel, Marcus Magnork, Daniel Keim Comment : The projection of higher-dimensional dataset to lower dimensional space is although quite attractive, but its complexity is too high to rely on manual selection of lower-dimensional views. In this paper, the authors automate ranking of scatterplots and parallel coordinate views which has high probability of providing useful information.

* Depth-Dependent Halos: Illustrative Rendering of Dense Line Data – Vis09 * Perception-Based Transparency Optimization for Direct Volume Rendering – Vis09 * Bubble Sets: Revealing Set Relations with Isocontours over Existing Visualizations – infoVis09 * A Nested Model for Visualization Design and Validation * Smooth Graphs for Visual Exploration of Higher-Order State Transitions – InfoVi09

1. Perception-Based Transparency Optimization for Direct Volume Rendering I like to stare at pictures with visible structures.

2. Document Cards: A Top Trumps Visualization for Documents My friend and I thought that a summary diagram of a paper will be more efficient than an abstract or even the whole paper in the aspect of understanding and time.

3. A Comparison of User-Generated and Automatic Graph Layouts It is interesting the layouts look so different to each other.

Here are a few papers/presentations that look interesting to me:

InfoVis Systems A Multi-Threading Architecture to Support Interactive Visual Exploration Harald Piringer, Christian Tominski (University of Rostock), Philipp Muigg, Wolfgang Berger

comment: I really appreciate it when an effort is made to extend the technical limitations associated with complex visual projects.

Vis Perception-Guided Visualization Best Paper Award: Depth-Dependent Halos: Illustrative Rendering of Dense Line Data Maarten H. Everts, Henk Bekker, Jos B.T.M. Roerdink, Tobias Isenberg (University of Groningen)

comment: The density of each grouping of line data is really easy digest in a short amount of time.

InfoVis Text Visualization Best Paper Award: Mapping Text with Phrase Nets Frank van Ham, Martin Wattenberg, Fernanda B. Viégas (Visual Communication Lab, IBM Watson Research Center)

comment: Just like with a tag-cloud, the viewer can quickly get a sense of the scope and theme of the topic.

InfoVis Collaborative Visualization SpicyNodes: Radial Layout Authoring for the General Public Michael Douma, Grzegorz Ligierko, Ovidiu Ancuta, Pavel Gritsai, Sean Liu (Human-Computer Interaction and Design, University of Washington)

comment: I thought the node based layout was well executed, although the transitions could be more fluid.

1. On the Visualization of Social and other Scale-Free Networks (InfoVis 2008. Yuntao Jia et al.) : social network analysis is all about finding meaningful patterns, which depends largely on effective condensing(clustering, filtering etc) of the edges.

2. Narratives: A Visualization to Track Narrative Events as they Develop (VAST 2008. Danyel Fisher et al.) : “Visualizing news stories in their historical and social context… by understanding how the major topics associated with them have changed over time”. Just the right topic anybody in my department should look at.

3. VisComplete: Automating Suggestions for Visualization Pipelines (Vis 2008. David Koop) : Pure heuristics, effectively visualized.

4. Configuring Hierarchical Layouts to Address Research Questions (InfoVis 2009, Aidan Slingsby et al.) : Explains how to use hierarchial displays to show multiple aspects of large multivariate datasets. Provides insights to build visual narratives in complex data exploration.

InfoVis ——–

1. Viz-A-Vis: Toward Visualizing Video through Computer Vision 2. Visual Analysis of Inter-Process Communication for Large-Scale Parallel Computing 3. Distributed Cognition as a Theoretical Framework for Information Visualization 4. Vispedia: Interactive Visual Exploration of Wikipedia Data via Search-Based Integration

Vis —- 1. BrainGazer – Visual Queries for Neurobiology Research

– “Hue-Preserving Color Blending” I don’t have any experience at all with this problem, yet the main idea of the paper is very simple, and clea rly described. The results were convincing too.

– “Interaction Techniques for Selecting and Manipulating Subgraphs in Network Visualizations” I like this one because graph data is ubiquitous, and though I couldn’t find the paper, the demo looked like it allows some useful interactions.

– “Perception-Based Transparency Optimization for Direct Volume Rendering” This one is interesting because it is an interesting problem I have never considered, and it may be useful fo r visualizing brain scan data. Fig. 8 shows their approach on a head CT scan.

– “Applying Manifold Learning to Plotting Approximate Contour Trees” This one is interesting just because it is close to my research area.

-Molecular Surface Abstraction (Vis 2007) Greg Cipriano, Michael Gleicher (University of Wisconsin, Madison)

– Mapping Text with Phrase Nets Frank van Ham, Martin Wattenberg, Fernanda B. Viégas (infoVis 2009)

-Multi-Scale Surface Descriptors Gregory Cipriano, George N. Phillips Jr., Michael Gleicher (Vis 2009)

-A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets Jibonananda Sanyal, Song Zhang, Gargi Bhattacharya, Phil Amburn, Robert J. Moorhead (Vis2009)

Previous post: A very cool visualization website

Next post: This week in 838 (Jan 25-29)

Assignments (23)
BasicInfo (10)
Cool Stuff (29)
Critiques (21)
Final Project (12)
Ideas for 2012 (6)
Lecture Stuff (7)
Project Ideas (3)
Resources (3)
Student Posts (83)
Uncategorized (13)

Using visualization components to build a custom visualization

This tutorial is intended for experienced JavaScript developers and assumes some familiarity with functional programming techniques.

In this example, we begin with a query that is related to hypothetical quarterly sales information for some brands. First we will filter the query for specific brands and then pivot the results by sales quarter. See the following table for an example.

Then, we will use visualization components to build a custom visualization that shows how each brand's products have been trending over the past quarter. The result will be a new kind of visualization that is composed of a series of sparklines nested within a table, which looks like this example:

Along with showing you how to create a custom visualization, this example demonstrates some best practices for working with the Looker API within a React application.

To build a customized visualization with Looker components, make sure your setup meets the requirements , and then perform the following steps:

Build a query in an Explore and copy the qid value
Pass the data to a custom visualization component
Build out the CustomVis component
Transform the normalized data
Insert the transformed data into CustomVis
Generate the custom visualization

Using visualization components to build a custom visualization is appropriate when the custom visualization is intended for an embedded application or extension. If you want to make the custom visualization available to Looker users across a Looker instance, follow the instructions on the visualization documentation page. If you want to develop a custom visualization and upload it to the Looker Marketplace, follow the instructions on the Developing a custom visualization for the Looker Marketplace documentation page.

Requirements

Before you start, a few elements are needed:

You must have access to a Looker instance.
Whether you're building in the extension framework or your own stand-alone React application, it is important to authenticate with Looker's API and have access to the Looker SDK object. Read about Looker API authentication or our extension framework for more information.
Make sure you have installed the Looker Visualization Components NPM package and the @looker/components-data NPM package . Information on installing and using the visualization components package can be found in the README document, available in GitHub and NPM .

Step 1: Build a query in an Explore and copy the query ID

In this example, we use hypothetical quarterly sales information for brands we are tracking over time.

We will pivot this data, as pivoting is Looker's built-in way to group the query results. In an Explore , we can run a query and create a chart of the data using one of Looker's native visualization types . The chart provides a lot of information, but it is hard to parse at a glance how each brand's products are trending:

As with the example of rendering a simple visualization , the next step is to copy the qid value from the URL bar of the Explore. For the purposes of this example, the qid value will be Uijcav7pCA4MZY2MompsPZ , but that value is specific to our test instance; your value will differ.

Step 2: Pass the data to a custom visualization component

To begin, pass the qid value taken from the Explore's URL into the Query component , and the authenticated SDK object to DataProvider .

Next, rather than rendering a native Looker visualization through the Visualization component , we will build our own custom component called CustomVis .

The Query component can accept any React element as a child, and will simply pass down the config , data , fields , and totals values as properties to render your own visualization components. We will render CustomVis as a child of Query , so it can receive all relevant data as properties.

Step 3: Build out the CustomVis component

Next, let's build the CustomVis component. The properties that are inherited from the Query component are config , fields , data , pivots , and totals :

config describes all the ways the data should be rendered in a chart, such as the thickness of the line in a sparkline or the size and shape of the points of a scatterplot.
fields stores additional metadata about the measure and dimension values returned from the query, such as how the values should be formatted or what to label each axis.
data is the key/value response that was returned from the query.
pivots describes the dimension by which the query is pivoted.
totals references Looker's row totals for use in table-based visualizations.

We can pass these unmodified properties onto a table visualization by inserting a Table component.

This gives us a sense of the data as it is returned directly from the SDK. In the rendered response, there is a row for every brand with results grouped, or pivoted, by quarter.

Step 4: Transform the normalized data

To convert this pivoted data to be rendered with nested sparklines, we isolate all the measure values and pass them to the subcharts. In the following chart, the relevant data for a single row is highlighted to illustrate the data we will be collapsing and rendering with a child visualization:

Chart of data results with the order counts in the second row highlighted.

We will create a custom transformation for this. The following is an example that is specific to this scenario; you will need to parse your own data accordingly.

The function is created using the following steps:

Reduce over the data set to isolate the brand name from the quarterly order data for each row.
Update each row to include the dimension and a rendered React component that can represent the values for each row in the table.

Step 5: Insert the transformed data into CustomVis

Now transform the data using our new function, and assign the output to a new variable called nestedData :

Step 6: Generate the custom visualization

After you have inserted the transformed data and configured the chart, the visualization will look like this example of a table with individual sparkline charts for each row:

The entirety of the code necessary to render the above visualization is as follows:

Using visualization components and the dashboard property to render a simple visualization
Using visualization components and the query property to render a simple visualization
Using visualization components to render custom visualizations
Visualization and Query property tables

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2023-11-17 UTC.

Instantly share code, notes, and snippets.

LalehT / Assignment 3 - Building a Custom Visualization.txt

Download ZIP
Star ( 0 ) 0 You must be signed in to star a gist
Fork ( 0 ) 0 You must be signed in to fork a gist
Embed Embed this gist in your website.
Share Copy sharable link for this gist.
Clone via HTTPS Clone using the web URL.
Learn more about clone URLs
Save LalehT/3378b2a2782a35006fe071636c7bfbf5 to your computer and use it in GitHub Desktop.

0.00 $ 0 items

AppliedPlotting Week 3-Building a Custom Visualization Solved

25.00 $

If Helpful Share:

Description

1 assignment 3 – building a custom visualization.

Ferreira, N., Fisher, D., & Konig, A. C. (2014, April). Sample-oriented task-driven visualiza tions: allowing users to make better, more confident decisions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 571-580). ACM. ( video )

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Easiest option: Implement the bar coloring as described above – a color scale with only three colors, (e.g. blue, white, and red). Assume the user provides the y axis value of interest as a parameter or variable.

Hardest option: Allow the user to interactively set a range of y values they are interested in, and recolor based on this (e.g. a y-axis band, see the paper for more details).

Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different.

In [1]: # Use the following data for this assignment:

import pandas as pd import numpy as np import matplotlib.pyplot as plt

np.random.seed(12345)

df = pd.DataFrame([np.random.normal(32000,200000,3650), np.random.normal(43000,100000,3650), np.random.normal(43500,140000,3650), np.random.normal(48000,70000,3650)], index=[1992,1993,1994,1995])

Out[1]: 0 1 2 3

-8941.531897 127788.667612 -71887.743011 -79146.060869
-51896.094813 198350.518755 -123518.252821 -129916.759685
932066 192947.128056 389950.263156 -93006.152024
-69708.439062 -13289.977022 -30178.390991 181256

4 5 6 7

114501 310681.166595 50581.575349 88349.230566
147314 49845.883728 149135.648505 62807.672113
575896 5529.230706 -32989.370488 223942.967178
621657 12930.835194 63700.461932 64148.489835

8 9 … 3640

513522 281286.947277 … 171938.760289
577348 -109686.264981 … -44566.520071
-66721.580898 269111 … 165085.806360
-29316.268556 677367 … -13901.388118

3641 3642 3643 3644

759924 203663.976475 -377877.158072 -197214.093861
122475 117648.199945 160475.622607 -13759.888342
174090 107329.726875 199250.734156 -36792.202754
686673 53965.990717 4128.990173 72202.595138

3645 3646 3647 3648 36

008589 -56826.729535 -67319.766489 113377.299342 -4494.8785
-37333.493572 103019.841174 179746.127403 493990 34442.8988
-71861.846997 113219 -29328.078384 65858.761714 -91542.0010
199964 139472.114293 59386.186379 73362.229590 28705.0829

[4 rows x 3650 columns]

In [19]: #df[‘mean’]=df.mean(axis=1).tolist()

#df.drop(df.iloc[:,0:(len(df.columns)-1)],axis=1,inplace=True)

#df.reset_index(inplace=True)

In [3]: import scipy.stats as ss

import matplotlib.pyplot as plt import matplotlib.colors as col import matplotlib.cm as cm

# get the means and standard deviations means = df.mean(axis=1) std = df.std(axis=1)

n = df.shape[1]

# compute the 95% confidence intervals yerr = std/np.sqrt(n) * ss.t.ppf(1-0.05/2, n-1)

#Setup the plot plt.figure() ax= plt.subplot(111) plt.show()

bars = plt.bar(range(df.shape[0]), means, yerr = yerr,color = ‘grey’) index = range(len(df.index)) plt.xticks(index, df.index) plt.title(‘Data between 1992 and 1995’)

[plt.gca().spines[loc].set_visible( False ) for loc in [‘top’, ‘right’]] for spine in ax.spines.values(): spine.set_visible( False )

In [4]: #fig = plt.gcf() threshold = 42000

# Add the horizontal line and add its value as a y-tick plt.axhline(y = threshold, zorder=1, color = ‘brown’) ytick = plt.gca().get_yticks() ytick = np.append(ytick,threshold) plt.gca().set_yticks(ytick)

#Setup the colormap

colormap = col.LinearSegmentedColormap.from_list(“colormap”,[“b”, “white”, cpick = cm.ScalarMappable(cmap=colormap) cpick.set_array([])

#Computing each column against threshold value percentages = [] for bar, yerr_ in zip(bars, yerr): low = bar.get_height() – yerr_ high = bar.get_height() + yerr_ percentage = (high-threshold)/(high-low) if percentage>1: percentage = 1 if percentage<0: percentage=0 percentages.append(percentage)

percentages

#Updating the plot

bars = plt.bar(range(df.shape[0]), means, yerr = yerr, color = cpick.to_rgb

#Add the colorbar plt.colorbar(cpick, orientation=’horizontal’, boundaries=np.linspace(0,1,12

Related products

AppliedPlotting Week 2 Solved

Appliedplotting week 4 solved.

Please enter your information to subscribe to the Microsoft Fabric Blog.

Microsoft fabric updates blog.

Microsoft Fabric May 2024 Update

Monthly Update

Welcome to the May 2024 update.

Here are a few, select highlights of the many we have for Fabric. You can now ask Copilot questions about data in your model, Model Explorer and authoring calculation groups in Power BI desktop is now generally available, and Real-Time Intelligence provides a complete end-to-end solution for ingesting, processing, analyzing, visualizing, monitoring, and acting on events.

There is much more to explore, please continue to read on.

Microsoft Build Announcements

At Microsoft Build 2024, we are thrilled to announce a huge array of innovations coming to the Microsoft Fabric platform that will make Microsoft Fabric’s capabilities even more robust and even customizable to meet the unique needs of each organization. To learn more about these changes, read the “ Unlock real-time insights with AI-powered analytics in Microsoft Fabric ” announcement blog by Arun Ulag.

Fabric Roadmap Update

Last October at the Microsoft Power Platform Community Conference we announced the release of the Microsoft Fabric Roadmap . Today we have updated that roadmap to include the next semester of Fabric innovations. As promised, we have merged Power BI into this roadmap to give you a single, unified road map for all of Microsoft Fabric. You can find the Fabric Roadmap at https://aka.ms/FabricRoadmap .

We will be innovating our Roadmap over the coming year and would love to hear your recommendation ways that we can make this experience better for you. Please submit suggestions at https://aka.ms/FabricIdeas .

Earn a discount on your Microsoft Fabric certification exam!

We’d like to thank the thousands of you who completed the Fabric AI Skills Challenge and earned a free voucher for Exam DP-600 which leads to the Fabric Analytics Engineer Associate certification.

If you earned a free voucher, you can find redemption instructions in your email. We recommend that you schedule your exam now, before your discount voucher expires on June 24 th . All exams must be scheduled and completed by this date.

If you need a little more help with exam prep, visit the Fabric Career Hub which has expert-led training, exam crams, practice tests and more.

Missed the Fabric AI Skills Challenge? We have you covered. For a limited time , you could earn a 50% exam discount by taking the Fabric 30 Days to Learn It Challenge .

Modern Tooltip now on by Default

Matrix layouts, line updates, on-object interaction updates, publish to folders in public preview, you can now ask copilot questions about data in your model (preview), announcing general availability of dax query view, copilot to write and explain dax queries in dax query view public preview updates, new manage relationships dialog, refreshing calculated columns and calculated tables referencing directquery sources with single sign-on, announcing general availability of model explorer and authoring calculation groups in power bi desktop, microsoft entra id sso support for oracle database, certified connector updates, view reports in onedrive and sharepoint with live connected semantic models, storytelling in powerpoint – image mode in the power bi add-in for powerpoint, storytelling in powerpoint – data updated notification, git integration support for direct lake semantic models.

Editor’s pick of the quarter
New visuals in AppSource
Financial Reporting Matrix by Profitbase
Horizon Chart by Powerviz

Milestone Trend Analysis Chart by Nova Silva

Sunburst Chart by Powerviz
Stacked Bar Chart with Line by JTA

Fabric Automation

Streamlining fabric admin apis, microsoft fabric workload development kit, external data sharing, apis for onelake data access roles, shortcuts to on-premises and network-restricted data, copilot for data warehouse.

Unlocking Insights through Time: Time travel in Data warehouse

Copy Into enhancements

Faster workspace resource assignment powered by just in time database attachment, runtime 1.3 (apache spark 3.5, delta lake 3.1, r 4.3.3, python 3.11) – public preview, native execution engine for fabric runtime 1.2 (apache spark 3.4) – public preview , spark run series analysis, comment @tagging in notebook, notebook ribbon upgrade, notebook metadata update notification, environment is ga now, rest api support for workspace data engineering/science settings, fabric user data functions (private preview), introducing api for graphql in microsoft fabric (preview), copilot will be enabled by default, the ai and copilot setting will be automatically delegated to capacity admins, abuse monitoring no longer stores your data, real-time hub, source from real-time hub in enhanced eventstream, use real-time hub to get data in kql database in eventhouse, get data from real-time hub within reflexes, eventstream edit and live modes, default and derived streams, route streams based on content in enhanced eventstream, eventhouse is now generally available, eventhouse onelake availability is now generally available, create a database shortcut to another kql database, support for ai anomaly detector, copilot for real-time intelligence, eventhouse tenant level private endpoint support, visualize data with real-time dashboards, new experience for data exploration, create triggers from real-time hub, set alert on real-time dashboards, taking action through fabric items, general availability of the power query sdk for vs code, refresh the refresh history dialog, introducing data workflows in data factory, introducing trusted workspace access in fabric data pipelines.

Introducing Blob Storage Event Triggers for Data Pipelines
Parent/child pipeline pattern monitoring improvements

Fabric Spark job definition activity now available

Hd insight activity now available, modern get data experience in data pipeline.

Power BI tooltips are embarking on an evolution to enhance their functionality. To lay the groundwork, we are introducing the modern tooltip as the new default , a feature that many users may already recognize from its previous preview status. This change is more than just an upgrade; it’s the first step in a series of remarkable improvements. These future developments promise to revolutionize tooltip management and customization, offering possibilities that were previously only imaginable. As we prepare for the general availability of the modern tooltip, this is an excellent opportunity for users to become familiar with its features and capabilities.

Discover the full potential of the new tooltip feature by visiting our dedicated blog . Dive into the details and explore the comprehensive vision we’ve crafted for tooltips, designed to enhance your Power BI experience.

We’ve listened to our community’s feedback on improving our tabular visuals (Table and Matrix), and we’re excited to initiate their transformation. Drawing inspiration from the familiar PivotTable in Excel , we aim to build new features and capabilities upon a stronger foundation. In our May update, we’re introducing ‘ Layouts for Matrix .’ Now, you can select from compact , outline , or tabular layouts to alter the arrangement of components in a manner akin to Excel.

As an extension of the new layout options, report creators can now craft custom layout patterns by repeating row headers. This powerful control, inspired by Excel’s PivotTable layout, enables the creation of a matrix that closely resembles the look and feel of a table. This enhancement not only provides greater flexibility but also brings a touch of Excel’s intuitive design to Power BI’s matrix visuals. Only available for Outline and Tabular layouts.

To further align with Excel’s functionality, report creators now have the option to insert blank rows within the matrix. This feature allows for the separation of higher-level row header categories, significantly enhancing the readability of the report. It’s a thoughtful addition that brings a new level of clarity and organization to Power BI’s matrix visuals and opens a path for future enhancements for totals/subtotals and rows/column headers.

We understand your eagerness to delve deeper into the matrix layouts and grasp how these enhancements fulfill the highly requested features by our community. Find out more and join the conversation in our dedicated blog , where we unravel the details and share the community-driven vision behind these improvements.

Following last month’s introduction of the initial line enhancements, May brings a groundbreaking set of line capabilities that are set to transform your Power BI experience:

Hide/Show lines : Gain control over the visibility of your lines for a cleaner, more focused report.
Customized line pattern : Tailor the pattern of your lines to match the style and context of your data.
Auto-scaled line pattern : Ensure your line patterns scale perfectly with your data, maintaining consistency and clarity.
Line dash cap : Customize the end caps of your customized dashed lines for a polished, professional look.
Line upgrades across other line types : Experience improvements in reference lines, forecast lines, leader lines, small multiple gridlines, and the new card’s divider line.

These enhancements are not to be missed. We recommend visiting our dedicated blog for an in-depth exploration of all the new capabilities added to lines, keeping you informed and up to date.

This May release, we’re excited to introduce on-object formatting support for Small multiples , Waterfall , and Matrix visuals. This new feature allows users to interact directly with these visuals for a more intuitive and efficient formatting experience. By double-clicking on any of these visuals, users can now right-click on the specific visual component they wish to format, bringing up a convenient mini-toolbar. This streamlined approach not only saves time but also enhances the user’s ability to customize and refine their reports with ease.

We’re also thrilled to announce a significant enhancement to the mobile reporting experience with the introduction of the pane manager for the mobile layout view. This innovative feature empowers users to effortlessly open and close panels via a dedicated menu, streamlining the design process of mobile reports.

We recently announced a public preview for folders in workspaces, allowing you to create a hierarchical structure for organizing and managing your items. In the latest Desktop release, you can now publish your reports to specific folders in your workspace.

When you publish a report, you can choose the specific workspace and folder for your report. The interface is simplistic and easy to understand, making organizing your Power BI content from Desktop better than ever.

To publish reports to specific folders in the service, make sure the “Publish dialogs support folder selection” setting is enabled in the Preview features tab in the Options menu.

Learn more about folders in workspaces.

We’re excited to preview a new capability for Power BI Copilot allowing you to ask questions about the data in your model! You could already ask questions about the data present in the visuals on your report pages – and now you can go deeper by getting answers directly from the underlying model. Just ask questions about your data, and if the answer isn’t already on your report, Copilot will then query your model for the data instead and return the answer to your question in the form of a visual!

We’re starting this capability off in both Edit and View modes in Power BI Service. Because this is a preview feature, you’ll need to enable it via the preview toggle in the Copilot pane. You can learn more about all the details of the feature in our announcement post here! (will link to announcement post)

We are excited to announce the general availability of DAX query view. DAX query view is the fourth view in Power BI Desktop to run DAX queries on your semantic model.

DAX query view comes with several ways to help you be as productive as possible with DAX queries.

Quick queries. Have the DAX query written for you from the context menu of tables, columns, or measures in the Data pane of DAX query view. Get the top 100 rows of a table, statistics of a column, or DAX formula of a measure to edit and validate in just a couple clicks!
DirectQuery model authors can also use DAX query view. View the data in your tables whenever you want!
Create and edit measures. Edit one or multiple measures at once. Make changes and see the change in action in a DA query. Then update the model when you are ready. All in DAX query view!
See the DAX query of visuals. Investigate the visuals DAX query in DAX query view. Go to the Performance Analyzer pane and choose “Run in DAX query view”.
Write DAX queries. You can create DAX queries with Intellisense, formatting, commenting/uncommenting, and syntax highlighting. And additional professional code editing experiences such as “Change all occurrences” and block folding to expand and collapse sections. Even expanded find and replace options with regex.

Learn more about DAX query view with these resources:

Deep dive blog: https://powerbi.microsoft.com/blog/deep-dive-into-dax-query-view-and-writing-dax-queries/
Learn more: https://learn.microsoft.com/power-bi/transform-model/dax-query-view
Video: https://youtu.be/oPGGYLKhTOA?si=YKUp1j8GoHHsqdZo

DAX query view includes an inline Fabric Copilot to write and explain DAX queries, which remains in public preview. This month we have made the following updates.

Run the DAX query before you keep it . Previously the Run button was disabled until the generated DAX query was accepted or Copilot was closed. Now you can Run the DAX query then decide to Keep or Discard the DAX query.

2. Conversationally build the DAX query. Previously the DAX query generated was not considered if you typed additional prompts and you had to keep the DAX query, select it again, then use Copilot again to adjust. Now you can simply adjust by typing in additional user prompts.

3. Syntax checks on the generated DAX query. Previously there was no syntax check before the generated DAX query was returned. Now the syntax is checked, and the prompt automatically retried once. If the retry is also invalid, the generated DAX query is returned with a note that there is an issue, giving you the option to rephrase your request or fix the generated DAX query.

4. Inspire buttons to get you started with Copilot. Previously nothing happened until a prompt was entered. Now click any of these buttons to quickly see what you can do with Copilot!

Learn more about DAX queries with Copilot with these resources:

Deep dive blog: https://powerbi.microsoft.com/en-us/blog/deep-dive-into-dax-query-view-with-copilot/
Learn more: https://learn.microsoft.com/en-us/dax/dax-copilot
Video: https://www.youtube.com/watch?v=0kE3TE34oLM

We are excited to introduce you to the redesigned ‘Manage relationships’ dialog in Power BI Desktop! To open this dialog simply select the ‘Manage relationships’ button in the modeling ribbon.

Once opened, you’ll find a comprehensive view of all your relationships, along with their key properties, all in one convenient location. From here you can create new relationships or edit an existing one.

Additionally, you have the option to filter and focus on specific relationships in your model based on cardinality and cross filter direction.

Learn more about creating and managing relationships in Power BI Desktop in our documentation .

Ever since we released composite models on Power BI semantic models and Analysis Services , you have been asking us to support the refresh of calculated columns and tables in the Service. This month, we have enabled the refresh of calculated columns and tables in Service for any DirectQuery source that uses single sign-on authentication. This includes the sources you use when working with composite models on Power BI semantic models and Analysis Services.

Previously, the refresh of a semantic model that uses a DirectQuery source with single-sign-on authentication failed with one of the following error messages: “Refresh is not supported for datasets with a calculated table or calculated column that depends on a table which references Analysis Services using DirectQuery.” or “Refresh over a dataset with a calculated table or a calculated column which references a Direct Query data source is not supported.”

Starting today, you can successfully refresh the calculated table and calculated columns in a semantic model in the Service using specific credentials as long as:

You used a shareable cloud connection and assigned it and/or.
Enabled granular access control for all data connection types.

Here’s how to do this:

Create and publish your semantic model that uses a single sign-on DirectQuery source. This can be a composite model but doesn’t have to be.
In the semantic model settings, under Gateway and cloud connections , map each single sign-on DirectQuery connection to a specific connection. If you don’t have a specific connection yet, select ‘Create a connection’ to create it:

If you are creating a new connection, fill out the connection details and click Create , making sure to select ‘Use SSO via Azure AD for DirectQuery queries:

Finally, select the connection for each single sign-on DirectQuery source and select Apply :

2. Either refresh the semantic model manually or plan a scheduled refresh to confirm the refresh now works successfully. Congratulations, you have successfully set up refresh for semantic models with a single sign-on DirectQuery connection that uses calculated columns or calculated tables!

We are excited to announce the general availability of Model Explorer in the Model view of Power BI, including the authoring of calculation groups. Semantic modeling is even easier with an at-a-glance tree view with item counts, search, and in context paths to edit the semantic model items with Model Explorer. Top level semantic model properties are also available as well as the option to quickly create relationships in the properties pane. Additionally, the styling for the Data pane is updated to Fluent UI also used in Office and Teams.

A popular community request from the Ideas forum, authoring calculation groups is also included in Model Explorer. Calculation groups significantly reduce the number of redundant measures by allowing you to define DAX formulas as calculation items that can be applied to existing measures. For example, define a year over year, prior month, conversion, or whatever your report needs in DAX formula once as a calculation item and reuse it with existing measures. This can reduce the number of measures you need to create and make the maintenance of the business logic simpler.

Available in both Power BI Desktop and when editing a semantic model in the workspace, take your semantic model authoring to the next level today!

Learn more about Model Explorer and authoring calculation groups with these resources:

Use Model explorer in Power BI (preview) – Power BI | Microsoft Learn
Create calculation groups in Power BI (preview) – Power BI | Microsoft Learn

Data connectivity

We’re happy to announce that the Oracle database connector has been enhanced this month with the addition of Single Sign-On support in the Power BI service with Microsoft Entra ID authentication.

Microsoft Entra ID SSO enables single sign-on to access data sources that rely on Microsoft Entra ID based authentication. When you configure Microsoft Entra SSO for an applicable data source, queries run under the Microsoft Entra identity of the user that interacts with the Power BI report.

We’re pleased to announce the new and updated connectors in this release:

[New] OneStream : The OneStream Power BI Connector enables you to seamlessly connect Power BI to your OneStream applications by simply logging in with your OneStream credentials. The connector uses your OneStream security, allowing you to access only the data you have based on your permissions within the OneStream application. Use the connector to pull cube and relational data along with metadata members, including all their properties. Visit OneStream Power BI Connector to learn more. Find this connector in the other category.
[New] Zendesk Data : A new connector developed by the Zendesk team that aims to go beyond the functionality of the existing Zendesk legacy connector created by Microsoft. Learn more about what this new connector brings.
[New] CCH Tagetik
[Update] Azure Databricks

Are you interested in creating your own connector and publishing it for your customers? Learn more about the Power Query SDK and the Connector Certification program .

Last May, we announced the integration between Power BI and OneDrive and SharePoint. Previously, this capability was limited to only reports with data in import mode. We’re excited to announce that you can now seamlessly view Power BI reports with live connected data directly in OneDrive and SharePoint!

When working on Power BI Desktop with a report live connected to a semantic model in the service, you can easily share a link to collaborate with others on your team and allow them to quickly view the report in their browser. We’ve made it easier than ever to access the latest data updates without ever leaving your familiar OneDrive and SharePoint environments. This integration streamlines your workflows and allows you to access reports within the platforms you already use. With collaboration at the heart of this improvement, teams can work together more effectively to make informed decisions by leveraging live connected semantic models without being limited to data only in import mode.

Utilizing OneDrive and SharePoint allows you to take advantage of built-in version control, always have your files available in the cloud, and utilize familiar and simplistic sharing.

While you told us that you appreciate the ability to limit the image view to only those who have permission to view the report, you asked for changes for the “Public snapshot” mode.

To address some of the feedback we got from you, we have made a few more changes in this area.

Add-ins that were saved as “Public snapshot” can be printed and will not require that you go over all the slides and load the add-ins for permission check before the public image is made visible.
You can use the “Show as saved image” on add-ins that were saved as “Public snapshot”. This will replace the entire add-in with an image representation of it, so the load time might be faster when you are presenting your presentation.

Many of us keep presentations open for a long time, which might cause the data in the presentation to become outdated.

To make sure you have in your slides the data you need, we added a new notification that tells you if more up to date data exists in Power BI and offers you the option to refresh and get the latest data from Power BI.

Developers

Direct Lake semantic models are now supported in Fabric Git Integration , enabling streamlined version control, enhanced collaboration among developers, and the establishment of CI/CD pipelines for your semantic models using Direct Lake.

Learn more about version control, testing, and deployment of Power BI content in our Power BI implementation planning documentation: https://learn.microsoft.com/power-bi/guidance/powerbi-implementation-planning-content-lifecycle-management-overview

Visualizations

Editor’s pick of the quarter .

– Animator for Power BI Innofalls Charts SuperTables Sankey Diagram for Power BI by ChartExpo Dynamic KPI Card by Sereviso Shielded HTML Viewer Text search slicer

New visuals in AppSource

Mapa Polski – Województwa, Powiaty, Gminy Workstream Income Statement Table

Gas Detection Chart

Seasonality Chart PlanIn BI – Data Refresh Service

Chart Flare

PictoBar ProgBar

Counter Calendar Donut Chart image

Financial Reporting Matrix by Profitbase

Making financial statements with a proper layout has just become easier with the latest version of the Financial Reporting Matrix.

Users are now able to specify which rows should be classified as cost-rows, which will make it easier to get the conditional formatting of variances correctly:

Selecting a row, and ticking “is cost” will tag the row as cost. This can be used in conditional formatting to make sure that positive variances on expenses are a bad for the result, while a positive variance on an income row is good for the result.

The new version also includes more flexibility in measuring placement and column subtotals.

Measures can be placed either:

Default (below column headers)
Above column headers

Conditionally hide columns
+ much more

Highlighted new features:

Measure placement – In rows
Select Column Subtotals
New Format Pane design
Row Options

Get the visual from AppSource and find more videos here !

Horizon Chart by Powerviz

A Horizon Chart is an advanced visual, for time-series data, revealing trends and anomalies. It displays stacked data layers, allowing users to compare multiple categories while maintaining data clarity. Horizon Charts are particularly useful to monitor and analyze complex data over time, making this a valuable visual for data analysis and decision-making.

Key Features:

Horizon Styles: Choose Natural, Linear, or Step with adjustable scaling.
Layer: Layer data by range or custom criteria. Display positive and negative values together or separately on top.
Reference Line : Highlight patterns with X-axis lines and labels.
Colors: Apply 30+ color palettes and use FX rules for dynamic coloring.
Ranking: Filter Top/Bottom N values, with “Others”.
Gridline: Add gridlines to the X and Y axis.
Custom Tooltip: Add highest, lowest, mean, and median points without additional DAX.
Themes: Save designs and share seamlessly with JSON files.

Other features included are ranking, annotation, grid view, show condition, and accessibility support.

Business Use Cases: Time-Series Data Comparison, Environmental Monitoring, Anomaly Detection

🔗 Try Horizon Chart for FREE from AppSource

📊 Check out all features of the visual: Demo file

📃 Step-by-step instructions: Documentation

💡 YouTube Video: Video Link

📍 Learn more about visuals: https://powerviz.ai/

✅ Follow Powerviz : https://lnkd.in/gN_9Sa6U

Exciting news! Thanks to your valuable feedback, we’ve enhanced our Milestone Trend Analysis Chart even further. We’re thrilled to announce that you can now switch between horizontal and vertical orientations, catering to your preferred visualization style.

The Milestone Trend Analysis (MTA) Chart remains your go-to tool for swiftly identifying deadline trends, empowering you to take timely corrective actions. With this update, we aim to enhance deadline awareness among project participants and stakeholders alike.

In our latest version, we seamlessly navigate between horizontal and vertical views within the familiar Power BI interface. No need to adapt to a new user interface – enjoy the same ease of use with added flexibility. Plus, it benefits from supported features like themes, interactive selection, and tooltips.

What’s more, ours is the only Microsoft Certified Milestone Trend Analysis Chart for Power BI, ensuring reliability and compatibility with the platform.

Ready to experience the enhanced Milestone Trend Analysis Chart? Download it from AppSource today and explore its capabilities with your own data – try for free!

We welcome any questions or feedback at our website: https://visuals.novasilva.com/ . Try it out and elevate your project management insights now!

Sunburst Chart by Powerviz

Powerviz’s Sunburst Chart is an interactive tool for hierarchical data visualization. With this chart, you can easily visualize multiple columns in a hierarchy and uncover valuable insights. The concentric circle design helps in displaying part-to-whole relationships.

Arc Customization: Customize shapes and patterns.
Color Scheme: Accessible palettes with 30+ options.
Centre Circle: Design an inner circle with layers. Add text, measure, icons, and images.
Conditional Formatting: Easily identify outliers based on measure or category rules.
Labels: Smart data labels for readability.
Image Labels: Add an image as an outer label.
Interactivity: Zoom, drill down, cross-filtering, and tooltip features.

Other features included are annotation, grid view, show condition, and accessibility support.

Business Use Cases:

Sales and Marketing: Market share analysis and customer segmentation.
Finance : Department budgets and expenditures distribution.
Operations : Supply chain management.
Education : Course structure, curriculum creation.
Human Resources : Organization structure, employee demographics.

🔗 Try Sunburst Chart for FREE from AppSource

Stacked Bar Chart with Line by JTA

Clustered bar chart with the possibility to stack one of the bars

Stacked Bar Chart with Line by JTA seamlessly merges the simplicity of a traditional bar chart with the versatility of a stacked bar, revolutionizing the way you showcase multiple datasets in a single, cohesive display.

Unlocking a new dimension of insight, our visual features a dynamic line that provides a snapshot of data trends at a glance. Navigate through your data effortlessly with multiple configurations, gaining a swift and comprehensive understanding of your information.

Tailor your visual experience with an array of functionalities and customization options, enabling you to effortlessly compare a primary metric with the performance of an entire set. The flexibility to customize the visual according to your unique preferences empowers you to harness the full potential of your data.

Features of Stacked Bar Chart with Line:

Stack the second bar
Format the Axis and Gridlines
Add a legend
Format the colors and text
Add a line chart
Format the line
Add marks to the line
Format the labels for bars and line

If you liked what you saw, you can try it for yourself and find more information here . Also, if you want to download it, you can find the visual package on the AppSource .

We have added an exciting new feature to our Combo PRO, Combo Bar PRO, and Timeline PRO visuals – Legend field support . The Legend field makes it easy to visually split series values into smaller segments, without the need to use measures or create separate series. Simply add a column with category names that are adjacent to the series values, and the visual will do the following:

Display separate segments as a stack or cluster, showing how each segment contributed to the total Series value.
Create legend items for each segment to quickly show/hide them without filtering.
Apply custom fill colors to each segment.
Show each segment value in the tooltip

Read more about the Legend field on our blog article

Drill Down Combo PRO is made for creators who want to build visually stunning and user-friendly reports. Cross-chart filtering and intuitive drill down interactions make data exploration easy and fun for any user. Furthermore, you can choose between three chart types – columns, lines, or areas; and feature up to 25 different series in the same visual and configure each series independently.

📊 Get Drill Down Combo PRO on AppSource

🌐 Visit Drill Down Combo PRO product page

Documentation | ZoomCharts Website | Follow ZoomCharts on LinkedIn

We are thrilled to announce that Fabric Core REST APIs are now generally available! This marks a significant milestone in the evolution of Microsoft Fabric, a platform that has been meticulously designed to empower developers and businesses alike with a comprehensive suite of tools and services.

The Core REST APIs are the backbone of Microsoft Fabric, providing the essential building blocks for a myriad of functionalities within the platform. They are designed to improve efficiency, reduce manual effort, increase accuracy, and lead to faster processing times. These APIs help with scale operations more easily and efficiently as the volume of work grows, automate repeatable processes with consistency, and enable integration with other systems and applications, providing a streamlined and efficient data pipeline.

The Microsoft Fabric Core APIs encompasses a range of functionalities, including:

Workspace management: APIs to manage workspaces, including permissions.
Item management: APIs for creating, reading, updating, and deleting items, with partial support for data source discovery and granular permissions management planned for the near future.
Job and tenant management: APIs to manage jobs, tenants, and users within the platform.

These APIs adhere to industry standards and best practices, ensuring a unified developer experience that is both coherent and easy to use.

For developers looking to dive into the details of the Microsoft Fabric Core APIs, comprehensive documentation is available. This includes guidelines on API usage, examples, and articles managed in a centralized repository for ease of access and discoverability. The documentation is continuously updated to reflect the latest features and improvements, ensuring that developers have the most current information at their fingertips. See Microsoft Fabric REST API documentation

We’re excited to share an important update we made to the Fabric Admin APIs. This enhancement is designed to simplify your automation experience. Now, you can manage both Power BI and the new Fabric items (previously referred to as artifacts) using the same set of APIs. Before this enhancement, you had to navigate using two different APIs—one for Power BI items and another for new Fabric items. That’s no longer the case.

The APIs we’ve updated include GetItem , ListItems , GetItemAccessDetails , and GetAccessEntities . These enhancements mean you can now query and manage all your items through a single API call, regardless of whether they’re Fabric types or Power BI types. We hope this update makes your work more straightforward and helps you accomplish your tasks more efficiently.

We’re thrilled to announce the public preview of the Microsoft Fabric workload development kit. This feature now extends to additional workloads and offers a robust developer toolkit for designing, developing, and interoperating with Microsoft Fabric using frontend SDKs and backend REST APIs. Introducing the Microsoft Fabric Workload Development Kit .

The Microsoft Fabric platform now provides a mechanism for ISVs and developers to integrate their new and existing applications natively into Fabric’s workload hub. This integration provides the ability to add net new capabilities to Fabric in a consistent experience without leaving their Fabric workspace, thereby accelerating data driven outcomes from Microsoft Fabric.

By downloading and leveraging the development kit , ISVs and software developers can build and scale existing and new applications on Microsoft Fabric and offer them via the Azure Marketplace without the need to ever leave the Fabric environment.

The development kit provides a comprehensive guide and sample code for creating custom item types that can be added to the Fabric workspace. These item types can leverage the Fabric frontend SDKs and backend REST APIs to interact with other Fabric capabilities, such as data ingestion, transformation, orchestration, visualization, and collaboration. You can also embed your own data application into the Fabric item editor using the Fabric native experience components, such as the header, toolbar, navigation pane, and status bar. This way, you can offer consistent and seamless user experience across different Fabric workloads.

This is a call to action for ISVs, software developers, and system integrators. Let’s leverage this opportunity to create more integrated and seamless experiences for our users.

We’re excited about this journey and look forward to seeing the innovative workloads from our developer community.

We are proud to announce the public preview of external data sharing. Sharing data across organizations has become a standard part of day-to-day business for many of our customers. External data sharing, built on top of OneLake shortcuts, enables seamless, in-place sharing of data, allowing you to maintain a single copy of data even when sharing data across tenant boundaries. Whether you’re sharing data with customers, manufacturers, suppliers, consultants, or partners; the applications are endless.

How external data sharing works

Sharing data across tenants is as simple as any other share operation in Fabric. To share data, navigate to the item to be shared, click on the context menu, and then click on External data share . Select the folder or table you want to share and click Save and continue . Enter the email address and an optional message and then click Send .

The data consumer will receive an email containing a share link. They can click on the link to accept the share and access the data within their own tenant.

Click here for more details about external data sharing .

Following the release of OneLake data access roles in public preview, the OneLake team is excited to announce the availability of APIs for managing data access roles. These APIs can be used to programmatically manage granular data access for your lakehouses. Manage all aspects of role management such as creating new roles, editing existing ones, or changing memberships in a programmatic way.

Do you have data stored on-premises or behind a firewall that you want to access and analyze with Microsoft Fabric? With OneLake shortcuts, you can bring on-premises or network-restricted data into OneLake, without any data movement or duplication. Simply install the Fabric on-premises data gateway and create a shortcut to your S3 compatible, Amazon S3, or Google Cloud Storage data source. Then use any of Fabric’s powerful analytics engines and OneLake open APIs to explore, transform, and visualize your data in the cloud.

Try it out today and unlock the full potential of your data with OneLake shortcuts!

Data Warehouse

We are excited to announce Copilot for Data Warehouse in public preview! Copilot for Data Warehouse is an AI assistant that helps developers generate insights through T-SQL exploratory analysis. Copilot is contextualized your warehouse’s schema. With this feature, data engineers and data analysts can use Copilot to:

Generate T-SQL queries for data analysis.
Explain and add in-line code comments for existing T-SQL queries.
Fix broken T-SQL code.
Receive answers regarding general data warehousing tasks and operations.

There are 3 areas where Copilot is surfaced in the Data Warehouse SQL Query Editor:

Code completions when writing a T-SQL query.
Chat panel to interact with the Copilot in natural language.
Quick action buttons to fix and explain T-SQL queries.

Learn more about Copilot for Data Warehouse: aka.ms/data-warehouse-copilot-docs. Copilot for Data Warehouse is currently only available in the Warehouse. Copilot in the SQL analytics endpoint is coming soon.

Unlocking Insights through Time: Time travel in Data warehouse (public preview)

As data volumes continue to grow in today’s rapidly evolving world of Artificial Intelligence, it is crucial to reflect on historical data. It empowers businesses to derive valuable insights that aid in making well-informed decisions for the future. Preserving multiple historical data versions not only incurred significant costs but also presented challenges in upholding data integrity, resulting in a notable impact on query performance. So, we are thrilled to announce the ability to query the historical data through time travel at the T-SQL statement level which helps unlock the evolution of data over time.

The Fabric warehouse retains historical versions of tables for seven calendar days. This retention allows for querying the tables as if they existed at any point within the retention timeframe. Time travel clause can be included in any top level SELECT statement. For complex queries that involve multiple tables, joins, stored procedures, or views, the timestamp is applied just once for the entire query instead of specifying the same timestamp for each table within the same query. This ensures the entire query is executed with reference to the specified timestamp, maintaining the data’s uniformity and integrity throughout the query execution.

From historical trend analysis and forecasting to compliance management, stable reporting and real-time decision support, the benefits of time travel extend across multiple business operations. Embrace the capability of time travel to navigate the data-driven landscape and gain a competitive edge in today’s fast-paced world of Artificial Intelligence.

We are excited to announce not one but two new enhancements to the Copy Into feature for Fabric Warehouse: Copy Into with Entra ID Authentication and Copy Into for Firewall-Enabled Storage!

Entra ID Authentication

When authenticating storage accounts in your environment, the executing user’s Entra ID will now be used by default. This ensures that you can leverage A ccess C ontrol L ists and R ole – B ased a ccess c ontrol to authenticate to your storage accounts when using Copy Into. Currently, only organizational accounts are supported.

How to Use Entra ID Authentication

Ensure your Entra ID organizational account has access to the underlying storage and can execute the Copy Into statement on your Fabric Warehouse.
Run your Copy Into statement without specifying any credentials; the Entra ID organizational account will be used as the default authentication mechanism.

Copy into firewall-enabled storage

The Copy Into for firewall-enabled storage leverages the trusted workspace access functionality ( Trusted workspace access in Microsoft Fabric (preview) – Microsoft Fabric | Microsoft Learn ) to establish a secure and seamless connection between Fabric and your storage accounts. Secure access can be enabled for both blob and ADLS Gen2 storage accounts. Secure access with Copy Into is available for warehouses in workspaces with Fabric Capacities (F64 or higher).

To learn more about Copy into , please refer to COPY INTO (Transact-SQL) – Azure Synapse Analytics and Microsoft Fabric | Microsoft Learn

We are excited to announce the launch of our new feature, Just in Time Database Attachment, which will significantly enhance your first experience, such as when connecting to the Datawarehouse or SQL endpoint or simply opening an item. These actions trigger the workspace resource assignment process, where, among other actions, we attach all necessary metadata of your items, Data warehouses and SQL endpoints, which can be a long process, particularly for workspaces that have a high number of items.

This feature is designed to attach your desired database during the activation process of your workspace, allowing you to execute queries immediately and avoid unnecessary delays. However, all other databases will be attached asynchronously in the background while you are able to execute queries, ensuring a smooth and efficient experience.

Data Engineering

We are advancing Fabric Runtime 1.3 from an Experimental Public Preview to a full Public Preview. Our Apache Spark-based big data execution engine, optimized for both data engineering and science workflows, has been updated and fully integrated into the Fabric platform.

The enhancements in Fabric Runtime 1.3 include the incorporation of Delta Lake 3.1, compatibility with Python 3.11, support for Starter Pools, integration with Environment and library management capabilities. Additionally, Fabric Runtime now enriches the data science experience by supporting the R language and integrating Copilot.

We are pleased to share that the Native Execution Engine for Fabric Runtime 1.2 is currently available in public preview. The Native Execution Engine can greatly enhance the performance for your Spark jobs and queries. The engine has been rewritten in C++ and operates in columnar mode and uses vectorized processing. The Native Execution Engine offers superior query performance – encompassing data processing, ETL, data science, and interactive queries – all directly on your data lake. Overall, Fabric Spark delivers a 4x speed-up on the sum of execution time of all 99 queries in the TPC-DS 1TB benchmark when compared against Apache Spark. This engine is fully compatible with Apache Spark™ APIs (including Spark SQL API).

It is seamless to use with no code changes – activate it and go. Enable it in your environment for your notebooks and your SJDs.

This feature is in the public preview, at this stage of the preview, there is no additional cost associated with using it.

We are excited to announce the Spark Monitoring Run Series Analysis features, which allow you to analyze the run duration trend and performance comparison for Pipeline Spark activity recurring run instances and repetitive Spark run activities from the same Notebook or Spark Job Definition.

Run Series Comparison: Users can compare the duration of a Notebook run with that of previous runs and evaluate the input and output data to understand the reasons behind prolonged run durations.
Outlier Detection and Analysis: The system can detect outliers in the run series and analyze them to pinpoint potential contributing factors.
Detailed Run Instance Analysis: Clicking on a specific run instance provides detailed information on time distribution, which can be used to identify performance enhancement opportunities.
Configuration Insights : Users can view the Spark configuration used for each run, including auto-tuned configurations for Spark SQL queries in auto-tune enabled Notebook runs.

You can access the new feature from the item’s recent runs panel and Spark application monitoring page.

We are excited to announce that Notebook now supports the ability to tag others in comments, just like the familiar functionality of using Office products!

When you select a section of code in a cell, you can add a comment with your insights and tag one or more teammates to collaborate or brainstorm on the specifics. This intuitive enhancement is designed to amplify collaboration in your daily development work.

Moreover, you can easily configure the permissions when tagging someone who doesn’t have the permission, to make sure your code asset is well managed.

We are thrilled to unveil a significant enhancement to the Fabric notebook ribbon, designed to elevate your data science and engineering workflows.

In the new version, you will find the new Session connect control on the Home tab, and now you can start a standard session without needing to run a code cell.

You can also easily spin up a High concurrency session and share the session across multiple notebooks to improve the compute resource utilization. And you can easily attach/leave a high concurrency session with a single click.

The “ View session information ” can navigate you to the session information dialog, where you can find a lot of useful detailed information, as well as configure the session timeout. The diagnostics info is essentially helpful when you need support for notebook issues.

Now you can easily access the powerful “ Data Wrangler ” on Home tab with the new ribbon! You can explore your data with the fancy low-code experience of data wrangler, and the pandas DataFrames and Spark DataFrames are all supported.

We recently made some changes to the Fabric notebook metadata to ensure compliance and consistency:

Notebook file content:

The keyword “trident” has been replaced with “dependencies” in the notebook content. This adjustment ensures consistency and compliance.
Notebook Git format:
The preface of the notebook has been modified from “# Synapse Analytics notebook source” to “# Fabric notebook source”.
Additionally, the keyword “synapse” has been updated to “dependencies” in the Git repo.

The above changes will be marked as ‘uncommitted’ for one time if your workspace is connected to Git. No action is needed in terms of these changes , and there won’t be any breaking scenario within the Fabric platform . If you have any further updates or questions, feel free to share with us.

We are thrilled to announce that the environment is now a generally available item in Microsoft Fabric. During this GA timeframe, we have shipped a few new features of Environment.

Git support

The environment is now Git supported. You can check-in the environment into your Git repo and manipulate the environment locally with its YAML representations and custom library files. After updating the changes from local to Fabric portal, you can publish them by manual action or through REST API.

Deployment pipeline

Deploying environments from one workspace to another is supported. Now, you can deploy the code items and their dependent environments together from development to test and even production.

With the REST APIs, you can have the code-first experience with the same abilities through Fabric portal. We provide a set of powerful APIs to ensure you the efficiency in managing your environment. You can create new environments, update libraries and Spark compute, publish the changes, delete an environment, attach the environment to a notebook, etc., all actions can be done locally in the tools of your choice. The article – Best practice of managing environments with REST API could help you get started with several real-world scenarios.

Resources folder

Resources folder enables managing small resources in the development cycle. The files uploaded in the environment can be accessed from notebooks once they’re attached to the same environment. The manipulation of the files and folders of resources happens in real-time. It could be super powerful, especially when you are collaborating with others.

Sharing your environment with others is also available. We provide several sharing options. By default, the view permission is shared. If you want the recipient to have access to view and use the contents of the environment, sharing without permission customization is the best option. Furthermore, you can grant editing permission to allow recipients to update this environment or grant share permission to allow recipients to reshare this environment with their existing permissions.

We are excited to announce the REST api support for Fabric Data Engineering/Science workspace settings. Data Engineering/Science settings allows users to create/manage their Spark compute, select the default runtime/default environment, enable or disable high concurrency mode or ML autologging.

Now with the REST api support for the Data Engineering/Science settings, you would be able to

Choose the default pool for a Fabric Workspace
Configure the max nodes for Starter pools
Create/Update/Delete the existing Custom Pools, Autoscale and Dynamic allocation properties
Choose Workspace Default Runtime and Environment
Select a default runtime
Select the default environment for the Fabric workspace
Enable or Disable High Concurrency Mode
Enable or Disable ML Auto logging.

Learn more about the Workspace Spark Settings API in our API documentation Workspace Settings – REST API (Spark) | Microsoft Learn

We are excited to give you a sneak peek at the preview of User Data Functions in Microsoft Fabric. User Data Functions gives developers and data engineers the ability to easily write and run applications that integrate with resources in the Fabric Platform. Data engineering often presents challenges with data quality or complex data analytics processing in data pipelines, and using ETL tools may present limited flexibility and ability to customize to your needs. This is where User data functions can be used to run data transformation tasks and perform complex business logic by connecting to your data sources and other workloads in Fabric.

During preview, you will be able to use the following features:

Use the Fabric portal to create new User Data Functions, view and test them.
Write your functions using C#.
Use the Visual Studio Code extension to create and edit your functions.
Connect to the following Fabric-native data sources: Data Warehouse, Lakehouse and Mirrored Databases.

You can now create a fully managed GraphQL API in Fabric to interact with your data in a simple, flexible, and powerful way. We’re excited to announce the public preview of API for GraphQL, a data access layer that allows us to query multiple data sources quickly and efficiently in Fabric by leveraging a widely adopted and familiar API technology that returns more data with less client requests. With the new API for GraphQL in Fabric, data engineers and scientists can create data APIs to connect to different data sources, use the APIs in their workflows, or share the API endpoints with app development teams to speed up and streamline data analytics application development in your business.

You can get started with the API for GraphQL in Fabric by creating an API, attaching a supported data source, then selecting specific data sets you want to expose through the API. Fabric builds the GraphQL schema automatically based on your data, you can test and prototype queries directly in our graphical in-browser GraphQL development environment (API editor), and applications are ready to connect in minutes.

Currently, the following supported data sources can be exposed through the Fabric API for GraphQL:

Microsoft Fabric Data Warehouse
Microsoft Fabric Lakehouse via SQL Analytics Endpoint
Microsoft Fabric Mirrored Databases via SQL Analytics Endpoint

Click here to learn more about how to get started.

Data Science

As you may know, Copilot in Microsoft Fabric requires your tenant administrator to enable the feature from the admin portal. Starting May 20th, 2024, Copilot in Microsoft Fabric will be enabled by default for all tenants. This update is part of our continuous efforts to enhance user experience and productivity within Microsoft Fabric. This new default activation means that AI features like Copilot will be automatically enabled for tenants who have not yet enabled the setting.

We are introducing a new capability to enable Copilot on Capacity level in Fabric. A new option is being introduced in the tenant admin portal, to delegate the enablement of AI and Copilot features to Capacity administrators. This AI and Copilot setting will be automatically delegated to capacity administrators and tenant administrators won’t be able to turn off the delegation.

We also have a cross-geo setting for customers who want to use Copilot and AI features while their capacity is in a different geographic region than the EU data boundary or the US. By default, the cross-geo setting will stay off and will not be delegated to capacity administrators automatically. Tenant administrators can choose whether to delegate this to capacity administrators or not.

Figure 1. Copilot in Microsoft Fabric will be auto enabled and auto delegated to capacity administrators.

Capacity administrators will see the “Copilot and Azure OpenAI Service (preview)” settings under Capacity settings/ Fabric Capacity / <Capacity name> / Delegated tenant settings. By default, the capacity setting will inherit tenant level settings. Capacity administrators can decide whether to override the tenant administrator’s selection. This means that even if Copilot is not enabled on a tenant level, a capacity administrator can choose to enable Copilot for their capacity. With this level of control, we make it easier to control which Fabric workspaces can utilize AI features like Copilot in Microsoft Fabric.

To enhance privacy and trust, we’ve updated our approach to abuse monitoring: previously, we retained data from Copilot in Fabric, including prompt inputs and outputs, for up to 30 days to check for misuse. Following customer feedback, we’ve eliminated this 30-day retention. Now, we no longer store prompt related data, demonstrating our unwavering commitment to your privacy and security. We value your input and take your concerns seriously.

Real-Time Intelligence

This month includes the announcement of Real-Time Intelligence, the next evolution of Real-Time Analytics and Data Activator. With Real-Time Intelligence, Fabric extends to the world of streaming and high granularity data, enabling all users in your organization to collect, analyze and act on this data in a timeline manner making faster and more informed business decisions. Read the full announcement from Build 2024.

Real-Time Intelligence includes a wide range of capabilities across ingestion, processing, analysis, transformation, visualization and taking action. All of this is supported by the Real-Time hub, the central place to discover and manage streaming data and start all related tasks.

Read on for more information on each capability and stay tuned for a series of blogs describing the features in more detail. All features are in Public Preview unless otherwise specified. Feedback on any of the features can be submitted at https://aka.ms/rtiidea

Ingest & Process

Introducing the Real-Time hub
Get Events with new sources of streaming and event data
Source from Real-Time Hub in Enhanced Eventstream
Use Real-Time hub to Get Data in KQL Database in Eventhouse
Get data from Real-Time Hub within Reflexes
Eventstream Edit and Live modes
Default and derived streams
Route data streams based on content

Analyze & Transform

Eventhouse GA
Eventhouse OneLake availability GA
Create a database shortcut to another KQL Database
Support for AI Anomaly Detector
Copilot for Real-Time Intelligence
Tenant-level private endpoints for Eventhouse

Visualize & Act

Visualize data with Real-Time Dashboards
New experience for data exploration
Create triggers from Real-Time Hub
Set alert on Real-time Dashboards
Taking action through Fabric Items

Ingest & Process

Real-Time hub is the single place for all data-in-motion across your entire organization. Several key features are offered in Real-Time hub:

1. Single place for data-in-motion for the entire organization

Real-Time hub enables users to easily discover, ingest, manage, and consume data-in-motion from a wide variety of sources. It lists all the streams and KQL tables that customers can directly act on.

2. Real-Time hub is never empty

All data streams in Fabric automatically show up in the hub. Also, users can subscribe to events in Fabric gaining insights into the health and performance of their data ecosystem.

3. Numerous connectors to simplify data ingestion from anywhere to Real-Time hub

Real-Time hub makes it easy for you to ingest data into Fabric from a wide variety of sources like AWS Kinesis, Kafka clusters, Microsoft streaming sources, sample data and Fabric events using the Get Events experience.

There are 3 tabs in the hub:

Data streams : This tab contains all streams that are actively running in Fabric that user has access to. This includes all streams from Eventstreams and all tables from KQL Databases.
Microsoft sources : This tab contains Microsoft sources (that user has access to) and can be connected to Fabric.
Fabric events : Fabric now has event-driven capabilities to support real-time notifications and data processing. Users can monitor and react to events including Fabric Workspace Item events and Azure Blob Storage events. These events can be used to trigger other actions or workflows, such as invoking a data pipeline or sending a notification via email. Users can also send these events to other destinations via Event Streams.

Learn More

You can now connect to data from both inside and outside of Fabric in a mere few steps. Whether data is coming from new or existing sources, streams, or available events, the Get Events experience allows users to connect to a wide range of sources directly from Real-Time hub, Eventstreams, Eventhouse and Data Activator.

This enhanced capability allows you to easily connect external data streams into Fabric with out-of-box experience, giving you more options and helping you to get real-time insights from various sources. This includes Camel Kafka connectors powered by Kafka connect to access popular data platforms, as well as the Debezium connectors for fetching the Change Data Capture (CDC) streams.

Using Get Events, bring streaming data from Microsoft sources directly into Fabric with a first-class experience. Connectivity to notification sources and discrete events is also included, this enables access to notification events from Azure and other clouds solutions including AWS and GCP. The full set of sources which are currently supported are:

Microsoft sources : Azure Event Hubs, Azure IoT hub
External sources : Google Cloud Pub/Sub, Amazon Kinesis Data Streams, Confluent Cloud Kafka
Change data capture databases : Azure SQL DB (CDC), PostgreSQL DB (CDC), Azure Cosmos DB (CDC), MySQL DB (CDC)
Fabric events : Fabric Workspace Item events, Azure Blob Storage events

Learn More

With enhanced Eventstream, you can now stream data not only from Microsoft sources but also from other platforms like Google Cloud, Amazon Kinesis, Database change data capture streams, etc. using our new messaging connectors. The new Eventstream also lets you acquire and route real-time data not only from stream sources but also from discrete event sources, such as: Azure Blob Storage events, Fabric Workspace Item events.

To use these new sources in Eventstream, simply create an eventstream with choosing “Enhanced Capabilities (preview)”.

You will see the new Eventstream homepage that gives you some choices to begin with. By clicking on the “Add external source”, you will find these sources in the Get events wizard that helps you to set up the source in a few steps. After you add the source to your eventstream, you can publish it to stream the data into your eventstream.

Using Eventstream with discrete sources to turn events into streams for more analysis. You can send the streams to different Fabric data destinations, like Lakehouse and KQL Database. After the events are converted, a default stream will appear in Real-Time Hub. To turn them, click Edit on ribbon, select “Stream events” on the source node, and publish your eventstream.

To transform the stream data or route it to different Fabric destinations based on its content, you can click Edit in ribbon and enter the Edit mode. There you can add event processing operators and destinations.

With Real-Time hub embedded in KQL Database experience, each user in the tenant can view and add streams which they have access to and directly ingest it to a KQL Database table in Eventhouse.

This integration provides each user in the tenant with the ability to access and view data streams they are permitted to. They can now directly ingest these streams into a KQL Database table in Eventhouse. This simplifies the data discovery and ingestion process by allowing users to directly interact with the streams. Users can filter data based on the Owner, Parent and Location and provides additional information such as Endorsement and Sensitivity.

You can access this by clicking on the Get Data button from the Database ribbon in Eventhouse.

This will open the Get Data wizard with Real-Time hub embedded.

You can use events from Real-Time hub directly in reflex items as well. From within the main reflex UI, click ‘Get data’ in the toolbar:

This will open a wizard that allows you to connect to new event sources or browse Real-Time Hub to use existing streams or system events.

Search new stream sources to connect to or select existing streams and tables to be ingested directly by Reflex.

You then have access to the full reflex modeling experience to build properties and triggers over any events from Real-Time hub.

Eventstream offers two distinct modes, Edit and Live, to provide flexibility and control over the development process of your eventstream. If you create a new Eventstream with Enhanced Capabilities enabled, you can modify it in an Edit mode. Here, you can design stream processing operations for your data streams using a no-code editor. Once you complete the editing, you can publish your Eventstream and visualize how it starts streaming and processing data in Live mode .

In Edit mode, you can:

Make changes to an Eventstream without implementing them until you publish the Eventstream. This gives you full control over the development process.
Avoid test data being streamed to your Eventstream. This mode is designed to provide a secure environment for testing without affecting your actual data streams.

For Live mode, you can :

Visualize how your Eventstream streams, transforms, and routes your data streams to various destinations after publishing the changes.
Pause the flow of data on selected sources and destinations, providing you with more control over your data streams being streamed into your Eventstream.

When you create a new Eventstream with Enhanced Capabilities enabled, you can now create and manage multiple data streams within Eventstream, which can then be displayed in the Real-Time hub for others to consume and perform further analysis.

There are two types of streams:

Default stream : Automatically generated when a streaming source is added to Eventstream. Default stream captures raw event data directly from the source, ready for transformation or analysis.
Derived stream : A specialized stream that users can create as a destination within Eventstream. Derived stream can be created after a series of operations such as filtering and aggregating, and then it’s ready for further consumption or analysis by other users in the organization through the Real-Time Hub.

The following example shows that when creating a new Eventstream a default stream alex-es1-stream is automatically generated. Subsequently, a derived stream dstream1 is added after an Aggregate operation within the Eventstream. Both default and derived streams can be found in the Real-Time hub.

Customers can now perform stream operations directly within Eventstream’s Edit mode, instead of embedding in a destination. This enhancement allows you to design stream processing logics and route data streams in the top-level canvas. Custom processing and routing can be applied to individual destinations using built-in operations, allowing for routing to distinct destinations within the Eventstream based on different stream content.

These operations include:

Aggregate : Perform calculations such as SUM, AVG, MIN, and MAX on a column of values and return a single result.
Expand : Expand array values and create new rows for each element within the array.
Filter : Select or filter specific rows from the data stream based on a condition.
Group by : Aggregate event data within a certain time window, with the option to group one or more columns.
Manage Fields : Customize your data streams by adding, removing, or changing data type of a column.
Union : Merge two or more data streams with shared fields (same name and data type) into a unified data stream.

Analyze & Transform

Eventhouse, a cutting-edge database workspace meticulously crafted to manage and store event-based data, is now officially available for general use. Optimized for high granularity, velocity, and low latency streaming data, it incorporates indexing and partitioning for structured, semi-structured, and free text data. With Eventhouse, users can perform high-performance analysis of big data and real-time data querying, processing billions of events within seconds. The platform allows users to organize data into compartments (databases) within one logical item, facilitating efficient data management.

Additionally, Eventhouse enables the sharing of compute and cache resources across databases, maximizing resource utilization. It also supports high-performance queries across databases and allows users to apply common policies seamlessly. Eventhouse offers content-based routing to multiple databases, full view lineage, and high granularity permission control, ensuring data security and compliance. Moreover, it provides a simple migration path from Azure Synapse Data Explorer and Azure Data Explorer, making adoption seamless for existing users.

Engineered to handle data in motion, Eventhouse seamlessly integrates indexing and partitioning into its storing process, accommodating various data formats. This sophisticated design empowers high-performance analysis with minimal latency, facilitating lightning-fast ingestion and querying within seconds. Eventhouse is purpose-built to deliver exceptional performance and efficiency for managing event-based data across diverse applications and industries. Its intuitive features and seamless integration with existing Azure services make it an ideal choice for organizations looking to leverage real-time analytics for actionable insights. Whether it’s analyzing telemetry and log data, time series and IoT data, or financial records, Eventhouse provides the tools and capabilities needed to unlock the full potential of event-based data.

We’re excited to announce that OneLake availability of Eventhouse in Delta Lake format is Generally Available.

Delta Lake is the unified data lake table format chosen to achieve seamless data access across all compute engines in Microsoft Fabric.

The data streamed into Eventhouse is stored in an optimized columnar storage format with full text indexing and supports complex analytical queries at low latency on structured, semi-structured, and free text data.

Enabling data availability of Eventhouse in OneLake means that customers can enjoy the best of both worlds: they can query the data with high performance and low latency in their Eventhouse and query the same data in Delta Lake format via any other Fabric engines such as Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more.

To learn more, please visit https://learn.microsoft.com/en-gb/fabric/real-time-analytics/one-logical-copy

A database shortcut in Eventhouse is an embedded reference to a source database. The source database can be one of the following:

(Now Available) A KQL Database in Real-Time Intelligence
An Azure Data Explorer database

The behavior exhibited by the database shortcut is similar to that of a follower database

The owner of the source database, the data provider, shares the database with the creator of the shortcut in Real-Time Intelligence, the data consumer. The owner and the creator can be the same person. The database shortcut is attached in read-only mode, making it possible to view and run queries on the data that was ingested into the source KQL Database without ingesting it.

This helps with data sharing scenarios where you can share data in-place either within teams, or even with external customers.

AI Anomaly Detector is an Azure service for high quality detection of multivariate and univariate anomalies in time series. While the standalone version is being retired October 2026, Microsoft open sourced the anomaly detection core algorithms and they are now supported in Microsoft Fabric. Users can leverage these capabilities in Data Science and Real-Time Intelligence workload. AI Anomaly Detector models can be trained in Spark Python notebooks in Data Science workload, while real time scoring can be done by KQL with inline Python in Real-Time Intelligence.

We are excited to announce the Public Preview of Copilot for Real-Time Intelligence. This initial version includes a new capability that translates your natural language questions about your data to KQL queries that you can run and get insights.

Your starting point is a KQL Queryset, that is connected to a KQL Database, or to a standalone Kusto database:

Simply type the natural language question about what you want to accomplish, and Copilot will automatically translate it to a KQL query you can execute. This is extremely powerful for users who may be less familiar with writing KQL queries but still want to get the most from their time-series data stored in Eventhouse.

Stay tuned for more capabilities from Copilot for Real-Time Intelligence!

Customers can increase their network security by limiting access to Eventhouse at a tenant-level, from one or more virtual networks (VNets) via private links. This will prevent unauthorized access from public networks and only permit data plane operations from specific VNets.

Visualize & Act

Real-Time Dashboards have a user-friendly interface, allowing users to quickly explore and analyze their data without the need for extensive technical knowledge. They offer a high refresh frequency, support a range of customization options, and are designed to handle big data.

The following visual types are supported, and can be customized with the dashboard’s user-friendly interface:

You can also define conditional formatting rules to format the visual data points by their values using colors, tags, and icons. Conditional formatting can be applied to a specific set of cells in a predetermined column or to entire rows, and lets you easily identify interesting data points.

Beyond the support visual, Real-Time Dashboards provide several capabilities to allow you to interact with your data by performing slice and dice operations for deeper analysis and gaining different viewpoints.

Parameters are used as building blocks for dashboard filters and can be added to queries to filter the data presented by visuals. Parameters can be used to slice and dice dashboard visuals either directly by selecting parameter values in the filter bar or by using cross-filters.
Cross filters allow you to select a value in one visual and filter all other visuals on that dashboard based on the selected data point.
Drillthrough capability allows you to select a value in a visual and use it to filter the visuals in a target page in the same dashboard. When the target page opens, the value is pushed to the relevant filters.

Real-Time Dashboards can be shared broadly and allow multiple stakeholders to view dynamic, real time, fresh data while easily interacting with it to gain desired insights.

Directly from a real-time dashboard, users can refine their exploration using a user-friendly, form-like interface. This intuitive and dynamic experience is tailored for insights explorers craving insights based on real-time data. Add filters, create aggregations, and switch visualization types without writing queries to easily uncover insights.

With this new feature, insights explorers are no longer bound by the limitations of pre-defined dashboards. As independent explorers, they have the freedom for ad-hoc exploration, leveraging existing tiles to kickstart their journey. Moreover, they can selectively remove query segments, and expand their view of the data landscape.

Dive deep, extract meaningful insights, and chart actionable paths forward, all with ease and efficiency, and without having to write complex KQL queries.

Data Activator allows you to monitor streams of data for various conditions and set up actions to be taken in response. These triggers are available directly within the Real-Time hub and in other workloads in Fabric. When the condition is detected, an action will automatically be kicked off such as sending alerts via email or Teams or starting jobs in Fabric items.

When you browse the Real-Time Hub, you’ll see options to set triggers in the detail pages for streams.

Selecting this will open a side panel where you can configure the events you want to monitor, the conditions you want to look for in the events, and the action you want to take while in the Real-Time hub experience.

Completing this pane creates a new reflex item with a trigger that monitors the selected events and condition for you. Reflexes need to be created in a workspace supported by a Fabric or Power BI Premium capacity – this can be a trial capacity so you can get started with it today!

Data Activator has been able to monitor Power BI report data since it was launched, and we now support monitoring of Real-Time Dashboard visuals in the same way.

From real-time dashboard tiles you can click the ellipsis (…) button and select “Set alert”

This opens the embedded trigger pane, where you can specify what conditions, you are looking for. You can choose whether to send email or Teams messages as the alert when these conditions are met.

When creating a new reflex trigger, from Real-time Hub or within the reflex item itself, you’ll notice a new ‘Run a Fabric item’ option in the Action section. This will create a trigger that starts a new Fabric job whenever its condition is met, kicking off a pipeline or notebook computation in response to Fabric events. A common scenario would be monitoring Azure Blob storage events via Real-Time Hub, and running data pipeline jobs when Blog Created events are detected.

This capability is extremely powerful and moves Fabric from a scheduled driven platform to an event driven platform.

Pipelines, spark jobs, and notebooks are just the first Fabric items we’ll support here, and we’re keen to hear your feedback to help prioritize what else we support. Please leave ideas and votes on https://aka.ms/rtiidea and let us know!

Real-Time Intelligence, along with the Real-Time hub, revolutionizes what’s possible with real-time streaming and event data within Microsoft Fabric.

Learn more and try it today https://aka.ms/realtimeintelligence

Data Factory

Dataflow gen2 .

We are thrilled to announce that the Power Query SDK is now generally available in Visual Studio Code! This marks a significant milestone in our commitment to providing developers with powerful tools to enhance data connectivity and transformation.

The Power Query SDK is a set of tools that allow you as the developer to create new connectors for Power Query experiences available in products such as Power BI Desktop, Semantic Models, Power BI Datamarts, Power BI Dataflows, Fabric Dataflow Gen2 and more.

This new SDK has been in public preview since November of 2022, and we’ve been hard at work improving this experience which goes beyond what the previous Power Query SDK in Visual Studio had to offer.

The latest of these biggest improvements was the introduction of the Test Framework in March of 2024 that solidifies the developer experience that you can have within Visual Studio Code and the Power Query SDK for creating a Power Query connector.

The Power Query SDK extension for Visual Studio will be deprecated by June 30, 2024, so we encourage you to give this new Power Query SDK in Visual Studio Code today if you haven’t.

To get started with the Power Query SDK in Visual Studio Code, simply install it from the Visual Studio Code Marketplace . Our comprehensive documentation and tutorials are available to help you harness the full potential of your data.

Join our vibrant community of developers to share insights, ask questions, and collaborate on exciting projects. Our dedicated support team is always ready to assist you with any queries.

We look forward to seeing the innovative solutions you’ll create with the Power Query SDK in Visual Studio Code. Happy coding!

Introducing a convenient enhancement to the Dataflows Gen2 Refresh History experience! Now, alongside the familiar “X” button in the Refresh History screen, you’ll find a shiny new Refresh Button . This small but mighty addition empowers users to refresh the status of their dataflow refresh history status without the hassle of exiting the refresh history and reopening it. Simply click the Refresh Button , and voilà! Your dataflow’s refresh history status screen is updated, keeping you in the loop with minimal effort. Say goodbye to unnecessary clicks and hello to streamlined monitoring!

[New] OneStream : The OneStream Power Query Connector enables you to seamlessly connect Data Factory to your OneStream applications by simply logging in with your OneStream credentials. The connector uses your OneStream security, allowing you to access only the data you have based on your permissions within the OneStream application. Use the connector to pull cube and relational data along with metadata members, including all their properties. Visit OneStream Power BI Connector to learn more. Find this connector in the other category.

Data workflows

We are excited to announce the preview of ‘Data workflows’, a new feature within the Data Factory that revolutionizes the way you build and manage your code-based data pipelines. Powered by Apache Airflow, Data workflows offer seamless authoring, scheduling, and monitoring experience for Python-based data processes defined as Directed Acyclic Graphs (DAGs). This feature brings a SaaS-like experience to running DAGs in a fully managed Apache Airflow environment, with support for autoscaling , auto-pause , and rapid cluster resumption to enhance cost-efficiency and performance.

It also includes native cloud-based authoring capabilities and comprehensive support for Apache Airflow plugins and libraries.

To begin using this feature:

Access the Microsoft Fabric Admin Portal.
Navigate to Tenant Settings.

Under Microsoft Fabric options, locate and expand the ‘Users can create and use Data workflows (preview)’ section. Note: This action is necessary only during the preview phase of Data workflows.

2. Create a new Data workflow within an existing or new workspace.

3. Add a new Directed Acyclic Graph (DAG) file via the user interface.

4. Save your DAG(s).

5. Use Apache Airflow monitoring tools to observe your DAG executions. In the ribbon, click on Monitor in Apache Airflow.

For additional information, please consult the product documentation . If you’re not already using Fabric capacity, consider signing up for the Microsoft Fabric free trial to evaluate this feature.

Data Pipelines

We are excited to announce a new feature in Fabric that enables you to create data pipelines to access your firewall-enabled Azure Data Lake Storage Gen2 (ADLS Gen2) accounts. This feature leverages the workspace identity to establish a secure and seamless connection between Fabric and your storage accounts.

With trusted workspace access, you can create data pipelines to your storage accounts with just a few clicks. Then you can copy data into Fabric Lakehouse and start analyzing your data with Spark, SQL, and Power BI. Trusted workspace access is available for workspaces in Fabric capacities (F64 or higher). It supports organizational accounts or service principal authentication for storage accounts.

How to use trusted workspace access in data pipelines

Create a workspace identity for your Fabric workspace. You can follow the guidelines provided in Workspace identity in Fabric .

Configure resource instance rules for the Storage account that you want to access from your Fabric workspace. Resource instance rules for Fabric workspaces can only be created through ARM templates. Follow the guidelines for configuring resource instance rules for Fabric workspaces here .

Create a data pipeline to copy data from the firewall enabled ADLS gen2 account to a Fabric Lakehouse.

To learn more about how to use trusted workspace access in data pipelines, please refer to Trusted workspace access in Fabric .

We hope you enjoy this new feature for your data integration and analytics scenarios. Please share your feedback and suggestions with us by leaving a comment here.

Introducing Blob Storage Event Triggers for Data Pipelines

A very common use case among data pipeline users in a cloud analytics solution is to trigger your pipeline when a file arrives or is deleted. We have introduced Azure Blob storage event triggers as a public preview feature in Fabric Data Factory Data Pipelines. This utilizes the Fabric Reflex alerts capability that also leverages Event Streams in Fabric to create event subscriptions to your Azure storage accounts.

Parent/Child pipeline pattern monitoring improvements

Today, in Fabric Data Factory Data Pipelines, when you call another pipeline using the Invoke Pipeline activity, the child pipeline is not visible in the monitoring view. We have made updates to the Invoke Pipeline activity so that you can view your child pipeline runs. This requires an upgrade to any pipelines that you have in Fabric that already use the current Invoke Pipeline activity. You will be prompted to upgrade when you edit your pipeline and then provide a connection to your workspace to authenticate. Another additional new feature that will light up with this invoke pipeline activity update is the ability to invoke pipeline across workspaces in Fabric.

We are excited to announce the availability of the Fabric Spark job definition activity for data pipelines. With this new activity, you will be able to run a Fabric Spark Job definition directly in your pipeline. Detailed monitoring capabilities of your Spark Job definition will be coming soon!

To learn more about this activity, read https://aka.ms/SparkJobDefinitionActivity

We are excited to announce the availability of the Azure HDInsight activity for data pipelines. The Azure HDInsight activity allows you to execute Hive queries, invoke a MapReduce program, execute Pig queries, execute a Spark program, or a Hadoop Stream program. Invoking either of the 5 activities can be done in a singular Azure HDInsight activity, and you can invoke this activity using your own or on-demand HDInsight cluster.

To learn more about this activity, read https://aka.ms/HDInsightsActivity

We are thrilled to share the new Modern Get Data experience in Data Pipeline to empower users intuitively and efficiently discover the right data, right connection info and credentials.

In the data destination, users can easily set destination by creating a new Fabric item or creating another destination or selecting existing Fabric item from OneLake data hub.

In the source tab of Copy activity, users can conveniently choose recent used connections from drop down or create a new connection using “More” option to interact with Modern Get Data experience.

Microsoft Fabric March 2024 Update

Welcome to the March 2024 update. We have a lot of great features this month including OneLake File Explorer, Autotune Query Tuning, Test Framework for Power Query SDK in VS Code, and many more! Earn a free Microsoft Fabric certification exam! We are thrilled to announce the general availability of Exam DP-600, which leads to … Continue reading “Microsoft Fabric March 2024 Update”

IMAGES

Assignment 3
Assignment 3
3
Build a Custom Visualization
3
Create Custom Visualizations in Looker Studio

VIDEO

14. Earthquake Data Analysis & Visualization Project with Plotly & Shiny: Course Help & Walkthrough
Data 2205
Data visualization with python 2023 updated Quiz Final Exam #coursera #solution #certificate
D3.js for Data Visualization
D3.js v3 Tutorial: Introduction
COURSERA PREDICTIVE MODELING AND ANALYTICS WEEK 1 PEER GRADE ASSIGNMENT DATA VISUALIZATION

COMMENTS

Assignment 3
Assignment 3 - Building a Custom Visualization. In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work. The options ...
Assignment 3
Assignment 3 - Building a Custom Visualization In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Assignment3 Building a Custom Visualization.ipynb
Saved searches Use saved searches to filter your results more quickly
Assignment 3
assignment3. Assignment 3 - Building a Custom Visualization ¶. In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Assignment 3
Note: The data given for this assignment is not the same as the data used in the article and as a result the visualizations may look a little different. \n import pandas as pd \n import numpy as np \n import matplotlib . pyplot as plt \n import scipy . stats as st \n import matplotlib . colors as col \n import matplotlib . cm as cm \n\n ...
Assignment 3
Explore and run machine learning code with Kaggle Notebooks | Using data from Houses predicted values using Excel
Assignment 3
Assignment 3 - Building a Custom Visualization - Free download as PDF File (.pdf), Text File (.txt) or view presentation slides online. Even Harder Option Figure
LalehT's gists · GitHub
Assignment 3 - Building a Custom Visualization: In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Assignment 3
Assignment 3 - Building a Custom Visualization. In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
PDF Peer-graded Assignment: Building a Custom Visualization
We recommend that you run the code on the Jupyter notebook system on the Coursera platform. Even harder: Added interactivity that allows the user to click on the y axis to set the value of interestThe bar colors change appropriately with respect to what value the user has selected. Hardest: Added interactivity that allows the user to ...
Assignment 3 for Week 3 of Applied Plotting, Charting and Data
"In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Assignment 3
Assignment 3 - Building a Custom Visualization. In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work. The options ...
Assignment 3
Assignment 3 - Building a Custom Visualization¶ In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Assignment 3
Once you have completed all notebooks and filled out the necessary code, you need to follow the below instructions to submit your work: 1. Open collect_submission.ipynb in Colab and execute the notebook cells. This notebook/script will: Generate a zip file of your code ( .py and .ipynb) called a3_code_submission.zip.
DCMAM/-Assignment_3-Building_a_Custom_Visualization
Assignment 3 of Applied plotting, charting & Data Representation in Python Course provided by University of Michigan at Coursera. - DCMAM/-Assignment_3-Building_a_Custom_Visualization
Assignment 3: Visualization Research
Assignment 3: Visualization Research. January 23, 2010. in Assignments. ... but this interested me because so much network visualization software (my major field of interest) is custom, takes considerable time to build, and doesn't necessarily involve best practices in visualization. ... Provides insights to build visual narratives in complex ...
{"payload":{"allShortcutsEnabled":false,"fileTree":{"":{"items":[{"name
{"payload":{"allShortcutsEnabled":false,"fileTree":{"":{"items":[{"name":"Assignment3_1.ipynb","path":"Assignment3_1.ipynb","contentType":"file"},{"name":"Assignment3 ...
Assignment 3 for Week 3 of Applied Plotting, Charting and Data
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window.
Using visualization components to build a custom visualization
On this page. Requirements. Step 1: Build a query in an Explore and copy the query ID. Step 2: Pass the data to a custom visualization component. Step 3: Build out the CustomVis component. Step 4: Transform the normalized data. Step 5: Insert the transformed data into CustomVis. Step 6: Generate the custom visualization.
Assignment 3
Raw. Assignment 3 - Building a Custom Visualization.txt. Assignment 3 - Building a Custom Visualization. In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib ...
AppliedPlotting Week 3-Building a Custom Visualization Solved
1 Assignment 3 - Building a Custom Visualization In this assignment you must choose one of the options presented below and submit a visual as well as your source code for peer grading. The details of how you solve the assignment are up to you, although your assignment must use matplotlib so that your peers can evaluate your work.
Microsoft Fabric May 2024 Update
Welcome to the May 2024 update. Here are a few, select highlights of the many we have for Fabric. You can now ask Copilot questions about data in your model, Model Explorer and authoring calculation groups in Power BI desktop is now generally available, and Real-Time Intelligence provides a complete end-to-end solution for ingesting, processing, analyzing, visualizing, monitoring, and acting ...
Building-a-Custom-Visualization/Coursera Plotting course Assignment 3(2
Contribute to olgacot/Building-a-Custom-Visualization development by creating an account on GitHub.
Projects · -Assignment_3-Building_a_Custom_Visualization
GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 330 million projects. ... DCMAM /-Assignment_3-Building_a_Custom_Visualization Public. Notifications Fork 0; Star 0. Code; Issues 0; Pull requests 0; Actions; Projects 0; Security; Insights Search all projects No open ...

Nick Malinowski's Portfolio

Randomizing data

Calculating mean and standard deviation

Creating confidence intervals

Creating the graph

Assignment 3 - Building a Custom Visualization ¶

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Assignment 3 - Building a Custom Visualization ¶

Figure 1 from (Ferreira et al, 2014).

Assignment 3 - Building a Custom Visualization ¶

Figure 1 from (Ferreira et al, 2014).

Figure 2c from (Ferreira et al. 2014). Note that the colorbar legend at the bottom as well as the arrows are not required in the assignment descriptions below.

Assignment 3

Q1: Image Captioning with Vanilla RNNs (30 points)

Assignment 3: Visualization Research

Recent Posts

Recent Comments

Using visualization components to build a custom visualization

Requirements

Step 1: Build a query in an Explore and copy the query ID

Step 2: Pass the data to a custom visualization component

Step 3: Build out the CustomVis component

Step 4: Transform the normalized data

Step 5: Insert the transformed data into CustomVis

Step 6: Generate the custom visualization

LalehT / Assignment 3 - Building a Custom Visualization.txt

AppliedPlotting Week 3-Building a Custom Visualization Solved

If Helpful Share:

Description

Related products

AppliedPlotting Week 2 Solved

Please enter your information to subscribe to the Microsoft Fabric Blog.

Microsoft Build Announcements

Fabric Roadmap Update

Earn a discount on your Microsoft Fabric certification exam!

Modern Tooltip now on by Default

Milestone Trend Analysis Chart by Nova Silva

Fabric Automation

Copy Into enhancements

Fabric Spark job definition activity now available

Data connectivity

Developers

Visualizations

New visuals in AppSource

Financial Reporting Matrix by Profitbase

Horizon Chart by Powerviz

Sunburst Chart by Powerviz

Stacked Bar Chart with Line by JTA

Data Warehouse

Unlocking Insights through Time: Time travel in Data warehouse (public preview)

Data Engineering

Data Science

Real-Time Intelligence

Ingest & Process

Analyze & Transform

Visualize & Act

Data Factory

Data workflows

Data Pipelines

Introducing Blob Storage Event Triggers for Data Pipelines

Parent/Child pipeline pattern monitoring improvements

Related blog posts

Microsoft Fabric March 2024 Update

IMAGES

VIDEO

COMMENTS