Posts - A blog @ nonodename.com

The six ages of programming

Dan Bennett — Mon, 05 Jan 2026 09:35:22 -0500

One of the reasons I love the time between Christmas and New Year is that work is quiet and I can experiment. This year, I continued to work on my DuckDB RDF extension, adding support for parsing RDF/XML. There’s really no need for RDF/XML support, hardly anyone uses it any more, largely because it’s such a bear to work with: overly complex with all sorts of largely irrelevant edge cases. (For example, a full XML literal in the object).

But that wasn’t going to stop me! Once I had a basic parser integrated into DuckDB (more of that in a moment), my methodology was simple: create a set of unit tests using all the example documents from the W3C spec: Each unit test would parse the xml and ntriple examples, pass them to DuckDB and ask it to do a union on the two. If the six columns returned were an exact match then the count in the union should be the same as the count in the nTriples. Side note, I used DuckDB parameterized queries for this which worked really well. Given a query like this:

prepare determine_delta as   
select count(a.*)-count(b.*) as delta   
from (select subject, predicate, object, object_datatype, upper(object_lang)    
      from read_rdf($testFile||'.nt')   
      union  
      select subject, predicate, object, object_datatype, upper(object_lang)    
      from read_rdf($testFile||'.rdf')) as a,   
      select count(*) from read_rdf($testFile||'.nt')) as b;

Then a single test is written

execute determine_delta(testFile := 'test/xmlrdf/example07');

that passes if it returns 0 (e.g. the union of the two tables - one of the tables should be zero rows as they exactly match)

Having created the tests, I had Gemini write the code using libxml2 to SAX parse the XML documents. As each test failed I gave Gemini the example it failed on and asked it to improve the implementation. I find the mix of deterministic testing combined with probabilistic code generation to be a great match: keep iterating until the tests pass.

While I worked on and off on this, I found myself reflecting on the fact that Gen AI enabled software development is really the next leap in developer productivity. I started my engineering career in the PC era, pre Internet. You would look up how the SDK worked in a book, most libraries you used were purchased from vendors and were delivered via a mailed floppy disk. Support was largely figuring it out on your own, or if you were in an a team, summoning up the courage to ask a senior peer to help.

By the mid Nineties, those with an Internet connection were asking each other questions in comp.sci.* newsgroups which became even more powerful when Google started indexing them. There were also dedicated websites, for instance if you used Oracle you used Ask Tom (which I see still exists!)

The point being that once you had an Internet connection and access to search your productivity radically changed. The next unlock was Open Source libraries: we went from purchasing (or writing) libraries to do what you needed to finding and integrating them. The innovations have continued, each time giving us enhanced productivity.

Taking a step back, it feels to me like we’re entering the sixth age of software development, with the ages characterized by ChatGPT and I as:

Name	Key Innovation	Productivity Unlock
1. Punch Cards 50s-60s	Batch processing	Compilers replace raw machine code. Glimpses of code portability.
2. Interactive Terminals 60s-70s	Time-sharing	Instant feedback replaces batch cycles. Languages standardize, much more portability of code.
3. PCs 80s- 90s	Local IDEs	Full local dev stack, configuration management tooling starts to appear
4. Internet Knowledge 2000-2010	Web, Stack Overflow	Access to everyone’s experiences , rise of sites like StackOverflow. Documentation available via search.
5a. Open Source Reuse 2010-present	Package ecosystems	“Import instead of build”. Significant re-use of open source libraries.
5b. Cloud & DevOps 2010-present	Automation + cloud infra	Continuous delivery & reproducibility: no waiting for equipment to arrive, find out immediately if code is unsafe for production.
6. AI-Augmented now	LLMs, copilots	Code generation + reasoning

Quantum Leap?

Dan Bennett — Sun, 22 Dec 2024 14:35:22 -0500

I recently hosted a fireside chat at the Excellence in Energy conference. The topic was, flavor of the year, AI.

One of the questions from the audience cited Google’s Willow chip and asked whether that has potential to impact AI. If you’ve not been following along, Google recently announced a new quantum chip that offers an advance in error management. Since errors are one of the larger roadblocks to scaling up quantum to practical workloads any advances here are welcome.

This is a fascinating question. Here’s my take:

Quantum computing still has a long way to go from the lab to regular users, particularly in adapting it to workloads that ‘matter’. It’s a completely different programing paradigm and only suited to particular types of problems. We can’t just deploy a python script onto a quantum chip and see it run faster.
When quantum computing goes mainstream, it will likely be embedded in products that hide that complexity from most developers. This is not new: most developers interacting with OpenAI have no idea how they would build the LLM themselves. That’s the power of a good abstraction.
As I’ve wrote last month, evolution shows that AI is possible at far lower power draw than current techniques. We’re missing multiple technical breakthroughs to drive down power demand.
Quantum computing has already shown that it is capable of generating results that would take classical computers many years to replicate. Many years = power, therefore, in a sense, quantum computing shows us a far more power efficient way of computing.

So, it’s a reasonable bet that current complex AI approaches will be adapted to work on quantum platforms, driven by economic forces of lowered power consumption and cost.

As a coda: back in 1989, Penrose published “The Emperor’s New Mind” which posited that human consciousness is non algorithmic, instead a result of quantum effects. It’s fascinating to me that if we do implement AI on quantum computing we’re not disproving that hypothesis.

Energy for AI

Dan Bennett — Fri, 15 Nov 2024 19:35:22 -0500

Have you read Leopold Aschenbrenner’s argument for a ‘Manhattan Project’ to race to Artificial General Intelligence (AGI)? I admit to being swayed by some of the thinking. There’s certainly holes that can be picked in the arguments: here’s three that a good friend of mine called out:

much of the data that an AGI might need for training is locked up in organizations and, even if access is given, is poorly formatted. Further, a lot of data out there is garbage
no mention is made of quantum computing. That could upend much of the thinking
any AGI will be concentrated in a few data centers with very visible power supply. In a race between nation states that’s relatively easy infrastructure to destroy

Leopold identifies energy as one of the key constraints to achieving AGI, and energy for AI was a key theme at the recent ENACT Majlis.

The Majlis agreed the answer to much greater energy demand for AI will be an ‘all of the above’ with gas, nuclear and renewables all part of the mix. I’m no expert on the supply side but I find myself doubting some of this demand thinking:

AI can be distributed: Apple Intelligence has been a bit of weak launch but the trend that our cell phones will run complex models and work can occur on the edge is only going to increase as models become more efficient and silicon more powerful.
AGI is likely further away than optimists like Aschenbrenner think it is. We’ve seen this trend in technology time and time again and particularly in AI. Further, there is some evidence that GenAI is not penetrating into companies as quickly as vendors expect as The Economist noted last week.
The biggest opportunity: we have lots of room for optimization. Physics shows us the hard bounds that we can convert power to motion or heat. Nature shows us that it takes 20W for NGI in the human brain. We are many orders of magnitude away from that right now with LLMs. If there is demand for AI then that creates massive market demand for efficient AI. Whether it’s binary neural nets, more efficient silicon, or knowledge graphs to ground models, there are many options to chase after.

There are many drivers of growth of energy demand. The vast majority of these are distributed with numerous market participants and regulators: EVs, developing economy growth, transition to heat pumps, decarbonization of emissions intensive industries to name but a few.

By contrast, AI is controlled and supplied by a few vendors who have massive incentive to keep their power consumption down and the deep pockets to drive that.

Will energy demand continue to grow? Of course. Will AI be a material part of that growth? I’m not so sure.

Semantics revisited

Dan Bennett — Sun, 20 Oct 2024 09:05:22 -0500

Last year I wrote about my dissatisfaction with the current state of semantic layers. The TL:DR being that if all you’re doing is tagging columns and tables with strings you’re not really helping with semantics.

Last year saw an explosion in RAG systems that, with a simple architecture and a decent LLM permit chat bots that do a good job of interacting with our textual content.

It’s only natural that the industry should turn to tabular data, since that’s where a lot of business information is locked up and SQL remains difficult for end users. The big database engine players are all working on this, for example Snowflake with Cortex Analyst, Databricks recently showed a solution running over financial data and Microsoft have been investing in synonyms to improve the Q&A feature of PowerBI.

What each of these solutions require is additional metadata to drive the feature, be it synonyms in PBI, comments for Databricks or a YAML file for Snowflake.

This shouldn’t be surprising. Generating SQL from just the table and column names is bound to fail for anything but the most trivial schema as there is far too much ambiguity in what the names mean.

To recap: each vendor is building their own standards for semantics to power natural language query. That is a new layer of lock in. While we’re gradually solving the format wars of iceberg and deltalake, we’re building a new set of imcompatibility over the top.

Worse, each solution is based on strings not things: the same issues of interopability that I wrote about last year are there and we are not solving for non English speakers, except by adding another layer for error translating the end user’s query to English before we figure out the SQL.

You know what

natively supports localization? RDF.
works across domains and business boundaries? RDF.
is almost thirty years old, tried and tested? RDF.
can be simply applied to tabular data? RDF.

The solution is staring us right in the face but we engineers love to reinvent the wheel rather than research, learn and leverage something we don’t understand.

There next couple of years are going to be annoying. Where’s the fast forward button on this?

Single pain of glass?

Dan Bennett — Sat, 20 Jul 2024 10:24:24 -0500

An abiding desire in Cyber, IT and technology management is to establish a ‘single pane of glass’: a unified command and control plane that covers your technology estate.

The rationale is simple and unarguable: you’re far less likely to miss a cyber attack, patching or other systems maintenance if you’re only looking in one place. Further, you’re likely to get better overall cost of ownership since you benefit from economies of scale and only having to have staff who know one system.

The concept extends beyond IT. In systems architecture we strive to standardize, simplify and add scale, based on years of accumulated experience showing that complexity adds cost and reduces flexibility.

Yet the events of yesterday clearly demonstrate the risk: if everything can be managed from one system then that one system can bring everything to its knees. Similarily if everything routes or depends on one system. Make no mistake, this will happen again. Human error is as sure in life as taxes.

In human management most organizations embrace diversity, recognizing that a diverse organization brings disparate strengths and resiliance. Maybe it’s time to challenge our desire for homogeneity in IT and systems architecutre? Instead, a one plus one strategy:

Windows and Linux in the cloud (on two hyperscalers)
Mac and Windows as client devices
iOS and Android for phones
Two patch and cyber security management systems, each deployed to half of the estate

You get the idea. There’s plenty of nuance here to think about, in particular dependencies within single planes: if a Windows system is dependent on a Linux system and you loose all the Linux servers your Windows system is still out.

It won’t be cheaper either: scale benefit is halved at negotiation time and staff must have a broader skill set.

This will happen again. Maybe the added cost is worth it if your cleanup time is halved?

Supporting eGauge in Apple Home

Dan Bennett — Fri, 03 May 2024 08:49:24 -0500

Last September we installed solar on our house. The installer added a LAN accessible realtime monitor from eGauge Systems. The device and it’s embedded software doesn’t have the most intuitive UX but it’s very hackable and includes a full, well documented, REST API.

Inspired by Ryan Seddon’s blog post, earlier this year I wrote a Homebridge Plugin that surfaces the data from the eGauge device into Apple Home.

HomeKit doesn’t (yet) support energy monitoring devices natively so I implemented as a light bulb and light sensor.

In the picture below, the solar is generating just over 5kW (expressed as lux) and providing 93% of the energy for the home.

Configuration is pretty simple. You’ll need a username & password to your eGauge. I recommend creating a separate read only credential so your write credentials aren’t stored in the clear in homebridge. To do that, click settings/access control:

Rest of configuration is done in homebridge, including the registers to read:

Hopefully one day Apple will natively support energy sensors. When they do, I’ll update the package to support them. If you have an eGauge & homebridge, you can download the plugin from NPM or install directly from your homebridge instance. Feel free to post any bug reports on GitHub.

Facts, Reference & Masterdata

Dan Bennett — Fri, 16 Feb 2024 15:49:24 -0500

I connected with an ex-colleague this weekend who not only was gracious enough to share that he found the blog useful but also used some of the articles to explain details to others. With that in mind, here’s something I wrote last year that might have wider applicability.

A common confusion in the data space is the difference between facts, reference and master data. To some degree, this is our own fault since the terms are often used interchangeably. For this post I thought I would try and demonstrate the difference, with domain specific examples that, hopefully, help to distinguish the three. To close, I’ll try and give a simple field guide for spotting them in the wild.

With that out of the way, let’s first tackle reference data. This is data that typically describes a lookup, valuable to defining some characteristic of another piece of data. Some common examples might be lists of:

Countries/Regions/Counties
Commodities
Units of measurement
Status fields (a project might be planned, approved, operational, cancelled, abandoned…)

If the data item is shown as a drop-down list, or radio buttons it’s likely reference data. Reference data is typically a slowly changing dimension. New countries do occur from time to time, but that is infrequent. We might add a new value to a status field, but rarely would we remove one or add 100 new. Reference data can be a flat list or hierarchical (a taxonomy) or perhaps, something more complex, an ontology.

The fundamental value of reference data is to permit, via standardization, comparison across domains of information. If we have two data sets, LNG liquefaction terminals and gas power stations, using the same reference data for location, we can easily compare by country or region. If they don’t, we place burden on the consumer to make sense of it.

What distinguishes master data from reference data? Fundamentally, it’s the mutability of the data: Master data inherently changes over time. The list of LNG liquefaction terminals, along with their status, is constantly changing as new facilities are brought online and old retired.

Like reference data, the value of master data is that if we all use the same set we permit easier modelling and comparison. We also remove potential for duplication both of data and effort. For this reason, customer and product lists are classically thought of as master data: If we want a single view of the customer it’s important that every piece of data we have on them ties to the same record for the customer.

That mutability over time means that master data is typically more voluminous than reference data. This, in turn, makes clear identifiers important for uniquely referencing a particular entity on the master data list.

Everything that is not reference or master data is facts. Facts typically change over time and are often numerical. The maximum output of a power station is a fact: it changes over time as the owner invests in and upgrades it. The real time output is also a fact, albeit one that changes much more rapidly. If it’s a time series, it’s highly likely facts. Invoices, pay slips, utility bils, all are facts referencing master and reference data.

Where this can get confusing is one person’s facts can be another’s master data. For example, in a CRM system you might have the current revenue of your customer. If you are in the business of tracking company revenues, revenue is a fact, something you track, manage, and sell. For someone else, it’s just a piece of our master data displayed to help contextualize the account.

That point hints at how to distinguish between the three in the wild as well as the inherent overlap. If:

someone sells the data, it’s facts or maybe master data (think Cusip or DUNs number), but very rarely reference data
you can get it for free, it’s reference data (for example, QUDT) or perhaps master data (PermID)
it’s part of your business as usual, it’s either facts or possibly master data.
you prepare it once and then forget about it, it’s probably reference data

Another rule of thumb: the more it changes, the more time we collectively spend on it, the more likely it is facts.

Hopefully that helps!

Predictions for 2024

Dan Bennett — Sat, 13 Jan 2024 08:59:24 -0500

Happy New Year to you all and welcome to the year of peak? generative AI. If you thought that was 2023, think again. Inspired by Dave Kellog, who every year posts his predictions (I encourage you to read and follow) I thought I would have a go. In no particular order:

AR/VR/Metaverse will continue to underwhelm, For now

This year will see the release of Apple Vision Pro. At a high price, this tool will not move much volume, but it will wow those who try it. The challenge is content: who and why will we invest in 3D content for a device with a very limited user base?

The Apple strategy is to drive content creation through their existing devices, with 3D video recording enabled on the latest iPhone 15 pros, AR embedded in much of their existing phones and tablets and free software for creating models.

Yet, we haven’t seen the killer app yet. Sure, there’s some gaming on Oculus, and some niche industry applications but until the hardware is cheaper and lighter I don’t think this will move the needle.

Microsoft will release a Data Marketplace

Microsoft have had many bites at the analytics space over the years with confusing and overlapping services like SSIS, Data Factory, SSAS, Synapse, PowerBI, Purview, Azure Data Catalog, Excel. Highly confusing compared to Snowflake or Databricks and you ran a real risk of choosing the wrong technology and being on an evolutionary dead end.

The release of Fabric as a unified marketing message and (in theory) set of compatible services shows they are not ignoring the competition. What’s missing, so far, is a data marketplace that works for companies data providers: they want their data to be easy to consume and don’t want to be charged for customer compute over it. Maybe this year Microsoft will make that happen.

The market will figure out what Gen/AI is good for

Back in ‘09 when the iPhone SDK came out, everyone went mad for mobile apps. If you had a website, you had to have a mobile app and the board wanted to know when it would be released. Over time, most of those apps died out, to be replaced by apps that could only exist because of the mobile form factor and connectivity. Think Lyft and Venmo.

We’re at the same stage with Gen/AI. We know it’s there, we’re exploring it, adding it to our existing products but no killer app yet. Later this year, someone will really figure it out and we’ll look back and wonder how we existed without it.

Twitter/X will flame out

Elon has done lots of smart things but buying Twitter doesn’t seem like one of them. The strategy is to make it the ’everything platform’ but that requires audience, and audience is quickly jumping ship measured by App Store downloads.

Reversing the wrong end of network effect is really hard and I don’t see Elon wanting to put the time or the money into it when he’s about to launch Starship and faces increasing EV competition from Chinese manufacturers. At what point do you just pull the plug?

One of the Gen/AI Copyright Suits will succeed

Tension between making text available for free for SEO and desire for it not to be included in LLMs continues to grow. Both Google and OpenAI published instructions to block training (while retaining SEO), implicitly agreeing that publishers see a difference between the two use cases.

Content owners & publishers went to court, most recently the New York Times. The core question before the courts will be: “is LLM training fair use?”. My guess is at least one of the courts will say no. That creates licensing opportunities for companies that own lots of copyright text and, I suspect, we’ll end up with some sort of monetization model just like we have for traffic coming from SEO.

Office CoPilot will be < $30/user/month by the end of the year

Microsoft 365 E5 licensing (the one that all big enterprises buy) is $57/user/month. To that Microsoft wants enterprises to add $30/user/month for CoPilot - it’s GenAI feature set for Office. The argument is that staff productivity will be so much greater that it’s worth it. But both customers and Microsoft are going to struggle to quantify that and a >50% increase in licensing is a tough pill to swallow.

My guess, enterprises will trial, leave it on for a few users but off for most as the cost can’t be justified. Small margin insensitive companies (think hedge funds, asset managers) will go all in but that’s not where the volume is. Microsoft will respond late this year or early next by bumping E5 to $77/month and including CoPilot.

Swing for the Fences: Apple buys Lucid

Apple has been playing with cars for some time and yet, CarPlay aside, hasn’t brought anything to market. Lucid has some fine technology, a sedan that few people are buying, an SUV that’s not coming until 2025 and aggressive competition from Chinese EV manufacturers at price points Lucid cannot meet. Their stock price is way down. They sell direct to customers through their own facilities with their own financing. Further, EV technology, especially battery and sensor, is rapidly changing meaning the half life of EVs is short compared to ICE cars, yet some of this can be fixed over the air. All, just like iPhones.

If there’s a company that knows how to direct sell premium product to the masses, it’s Apple.

That’s my predictions. What do you predict?

Filling in the blanks with Power Query

Dan Bennett — Sat, 04 Nov 2023 10:59:24 -0500

A couple of weeks ago I posted on using PowerQuery to visualize my many Xfinity outages. The completist in me couldn’t leave the data along with missing days (essentially the days when my monitoring didn’t log an outage).

This post explains how to add them back in. It’s really pretty simple, using some more advanced Power Query functions. The essential algorithm is:

Create a list of all the days in the dataset
Subtract from that list days when there was an outage
Add that resulting list to the list of outages, filling in the gaps in the data.

To do that, we need to add a new column to the first dataset, the day of the outage, essentially truncating the start time: For simplicity I’m going to ignore outages that span days.

One Power Query can reference another. So the easiest way forward is to create a new query that implements the algorithm above. Here’s the full code:

let
  Earliest = List.Min(temp[OutageDay]),
  Latest = List.Max(temp[OutageDay]),
  Difference = Duration.TotalDays(Latest-Earliest),
  InterveningDays = List.Dates(Earliest,Difference,#duration(1,0,0,0)),
  ZeroInterveningDays = List.Difference(InterveningDays,temp[OutageDay]), 
  AsTable = Table.FromList(ZeroInterveningDays,Splitter.SplitByNothing(), {"OutageDay"}),
  WithZero = Table.AddColumn(AsTable,"DownDuration", each 0, Int32.Type),
  SetType = Table.TransformColumnTypes(WithZero, {{"OutageDay", type date}}),
  AddStartTime = Table.TransformColumnTypes(Table.AddColumn(SetType, "StartTime", each DateTime.From([OutageDay])), {{"StartTime", type datetime}}),
  AddEndTime = Table.AddColumn(AddStartTime, "EndTime", each [StartTime]),
  Joined = Table.Combine({temp,AddEndTime})
in
  Joined

The first two lines calculate the earliest and latest outage days in the dataset ’temp’. That way, when the underlying text file changes so too will our dataset.

Next we generate a list of days between the earliest and latest using List.Dates. That function generates a list of dates and requires a start date, a number of days to list and the gap between each date (#duration(1,0,0,0) essentially means 1 day).

List.Difference is like a set minus operation. Given one list (all the days), take away another list (outage days) and return the days that there was not an outage.

From then on, it’s just a case of making the list look like our existing table:

Table.FromList creates a table
Table.AddColumn adds the outage duration of 0 for each day
Table.TransformColumnTypes converts the outage day to a date type
Table.AddColumn is used, twice, to add start and end times of the outage, which in this case we set to midnight.
Finally we add this new table with the existing outage table using Table.Combine.

Switch Excel chart to the new table and we finally get a chart with 0 day outages:

If you’ve not tried Power Query, I really encourage you to give it a try. While it has its eccentricities it really is a very powerful tool for manipulating data.

Visualizing Cable Outages using Power Query

Dan Bennett — Sun, 15 Oct 2023 10:59:24 -0500

I’ve long been a fan of PowerQuery (PQ), even more so since Microsoft finally made it available on Mac. If you’re unfamiliar, here’s a worked example of how to use it for a, sadly, every day problem.

Since moving house this summer we’ve gone from a rock solid fiber Internet connection to a less than glorious Xfinity cable connection that suffers an outage or two every day (as my team can attest to with frozen Teams calls). It’s infuriating.

Getting customer service from any cable company is always a challenge (‘have you rebooted your router?’) so I resolved to collect some outage data that will perhaps nudge Xfinity to take me seriously.

The first thing to do is collect good quality data. I installed the netcheck package on my hardwired Raspberry PI and started to collect data. The software logs outages like this:

-----------------------------------------------------------------------------
LINK DOWN:                                       Wed 20 Sep 2023 10:29:17 CDT
LINK RECONNECTED:                                Wed 20 Sep 2023 10:30:51 CDT
TOTAL DOWNTIME:                                  1 minutes and 34 seconds.
RECONNECTED LINK SPEED:                         
-----------------------------------------------------------------------------
                                                 Ping: 22.03 ms
                                                 Download: 26.02 Mbit/s
                                                 Upload: 39.22 Mbit/s
-----------------------------------------------------------------------------

The key data we need is those LINK DOWN and RECONNECTED lines. Those are easily extracted with a Grep statement:

grep -P "LINK (D|R)" connection.log > temp.log

That gives us a log file like this:

LINK DOWN:                                       Tue 19 Sep 2023 07:39:40 CDT
LINK RECONNECTED:                                Tue 19 Sep 2023 07:42:26 CDT

My goal is to get to a three column file, outage start, outage end, duration (in seconds), so I can chart the data, perhaps using a Pivot Chart:

To manipulate the file using PowerQuery, first open Excel and select the Get Data (Power Query) option under the Data Tab (my screenshots are Mac but they are almost identical for PC).

Select the Text/CSV option, choose the file and you’re presented with a preview of the data:

Click the transform button to launch the PowerQuery Editor. What I love about this environment is that if you prefer to edit your data visually, you can, yet the editing steps are stored as code you can directly edit if you wish.

A bit like Excel, each line of code (in this case, loading a document) is shown in the function bar with the resulting data shown below:

Notice all the columns are imported as text (the ‘ABC’ icon in the column header. The first thing to do convert the time to a timestamp. Click that column, click Transform and change the type to time:

Some of the columns we don’t need: the timezone, day and ‘LINK’ columns. Click each column header and select remove column. As you transform the data, you’ll see each step being recorded on the right hand side:

This is one of the things I love about PQ. Click on each step and you can see the exact changes you’ve made to your data.

The next thing to do is to recombine those three date columns into a new column with the date type. Click ‘Custom Column’ on the ‘Add Column’ tab. A quick search of the PQ website shows me the function to convert a string to a date:

Next, I combine the date and time columns with a new custom column called StartTime. The formula is as simple as [Timestamp] & [Column7]

I can now remove the columns I don’t need. Unlike Excel, PQ actually copies the data when you create a new column, so removing superfluous columns won’t ‘break’ your formula. It’s always good practice to remove unneeded columns to reduce storage and keep things tidy, leaving me with two columns.

To chart this data, I need to combine each pair of rows so I have a start and end time. There’s multiple ways of doing this. An easy way is to use Fill Up to copy data from a subsequent row. To do that, we need a new column that will become the end time. I use a conditional column here to only copy the end time if it truly is:

This results in a half filled End Time column:

Next, I use a ‘Fill Up’ (under the Transform tab, Fill button) command to replace the ’null’ values with the value from the row below:

Only two steps remain. We need to remove the ‘RECONNECTED’ rows as we don’t need them any more and then calculate the duration of the outage.

There are multiple ways to achieve the first goal, an easy way is ‘Remove Alternate Rows’ under the ‘Remove Rows’ button

Finally we add a custom column for duration with the following formula to count seconds: Duration.TotalSeconds(Duration.From([EndTime]-[StartTime]))

After removing the Status column we end up with three columns. Click ‘Close and Load’ to insert them into Excel:

One thing is missing.. we don’t have rows for days when there were no outages. That’s a post for another day… and with the number of outages there’s not many needed anyway!

Here’s all the steps. While it might seem a lot, each is a very simple transformation that we can easily understand and change or delete if we make a mistake:

Hopefully this has inspired you to give PQ a try. yourself. Let me know how you get on.