ASK: Aggregating Software Knowledge

(This idea is in the category of United Software Development: A new paradigm? created on SocialHub, and which inspired launch of the Social Coding Movement.)

Problem

Everywhere in a software development process knowledge is continuously created. Depending on size and complexity of the project a lot of that knowledge is a) either lost or forgotten over time as it wasn’t created in the most appropriate location, or b) increases the workload of the project team to transfer to a better location and review + reformulate / reformat.

Examples are:

  • Intricate discussions on an issue in the tracker that no one ever re-reads after the issue is closed.
  • A great discussion on the chat channel, e.g. Matrix, that has a lot of valid points never followed-up on.

Types of knowledge that this refers to can relate to any of the artifacts that are created during the software development process e.g. bugs, feature requests, tests, code, requirements, architecture, user documentation, etc.

There needs to be an easy way to distill this valuable knowledge, collect it in the right places, and without increasing the workload on the team with much extra ceremony.

Solution

The best time and place to gauge if some information is in any way valuable, is at the time and place when it is discussed / created. Then all those involved can see it in context and have the background most fresh in their mind. This is the right moment to identify, mark and collect the knowledge. However, in most cases it needs not be further elaborated at that time, only be aggregated for processing at a later time.


In a different context on this forum I mentioned the Murmurations Protocol. This protocol reverses the problem of knowledge aggregation from a top-down procedure (someone manually aggregates from many places) to a bottom-up process (knowledge ‘offers’ itself for aggregation to any interested consumer). Murmurations is a publish / subscribe mechanism for Linked Data, where knowledge that conforms to pre-defined Profiles publishes itself to a centralized index server, where subscribers obtain it for their own use. Note that Murmurations proto can be Federated itself to avoid this centralization.


Federated Murmurations offer a very generic solution to knowledge aggregation from arbitrary channels for the consumption of arbitrary consumers. It is ideal for application in a software development process to ease the task of the maintainer. As it happens I have described the use case of knowledge aggregation from an issue tracker by a project maintainer on the Solid community forum in response to an idea by Tim Berners-Lee to maintain Solid Faq’s:

Ideas for a possible FAQ Solid App

Tim Berners-Lee: When someone asked a slew of good, related questions on gitter recently I felt that I wanted to respond not on gitter, where there is no structure, but as a set of answers which could be jointly curated as part of the Solid Project’s institutional memory.

I will repeat a relevant use case I outlined in that discussion below

Insight: Also note that me manually duplicating a “Use Case” from another channel (Solid forum) is knowledge distribution that lends itself as a use case for automation using this same solution :thinking:

Use case example: Improve issue management

Problem statement

In our project we have great interaction from our development community and contributors. Many issues are created and thoroughly discussed until either a PR is created or the issue is closed as “Not doing”. But as our project grows in size and complexity we have to tackle some important bottlenecks:

  • Our maintainers spend more time than needed in issue management to follow-up in all ongoing discussions.
  • Even when an issue is still open the amount of text gets Too Long; Didn’t Read for many, leading to inefficient communication.
  • After the issue is closed the discussion contains many implicit design + architecture decisions that become hard to find.

Solution

Give maintainers the ability to markup knowledge snippets in their responses to issue comments, that are automatically aggregated to the various documentation artifacts of the project. An example of such response might look like:

@TheMaintainer wrote:
Thank you @SomeUser for that explanation. I wasn’t aware that our project was in use for such large-scale deployment. We see that we must support this for you and future customers, so I hereby define:

Requirement: The server MUST support at least 500 concurrent connections.

As for your other points … [bla bla yada yada] …

@SolidKnowledgeBot: I added 1 Requirement to Functional design on behalf of @TheMaintainer and related to #3456.

SolidKnowledgeBot might update docs directly, but more likely would create PR’s if the functional design was a Markdown file in /docs. But there might be multiple consumers, such as a Trello board where a card is created, or a Discourse forum where a discussion topic is created for project community.


(Photo by Markus Spiske from Pexels)


Note: This idea was first created on the Forgefriends forum, but given more appropriate scope, it will be continued here.