Open

Data sovereignty AI algorithms classification for data spaces

maestro — Tue, 09 May 2023 00:06:04 +0000

This report has been elaborated with the support of StandICT.

What is about? Associated to the data spaces there are ecosystems of services, marketplaces, many of them will provide services to help with the shared data (pattern recognition, outliers detection, etc) based on Artificial intelligence.

What these services are doing with the shared data? How could we trust on them.

could exist some mechanism to provide such trust?

These are the question addressed in this report and the exploration of a potential standard and a certification. It is based on a limited survey on experts on AI and some personal interviews. Of course it is a design that reality has to shape.

REPORT_ALBERTO_ABELLA_DATA_SOVEREIGNITY_v1.0(4)

Lack of data standardization will help to destroy our world. Time matters.

maestro — Mon, 19 Dec 2022 02:30:01 +0000

SHARING DATA IS REQUIRED FOR A BETTER WORLD

There is not much discussion about the relevance of data as the primary source for AI, impact analysis, analytics, etc. Hopefully, these technologies will help us to make this a better world. We are urged to substantially fix climate change in less than 10 years. Could you imagine the huge amounts of data required to agree on what to do, how to assess it, and how effective the different taken measures are?

THE HIDDEN VILLAIN

However, there is a villain in our data. Not their poor quality (This is another) but even with the most excellent quality, a few times all the data come from the same source. It means that the data should be compatible, said in more technical words, data sources have to share the same standard.

Therefore perfect data from different sources could lead to incredible disasters. The problem is that in this digital world the size of data grows at a 25% rate annually (depending on different sources), and the different types of data could grow at the same rate. One indicator of this is the % of standardized datasets in open data portals. In these two reports (2019 and 2021) the standardization rate decreased from 25% to 19% in Spain in the data made available from open data portals, when the amount of data increased by 62%. So looks like more data but less standardized.

Could we speed up the work of the standardization bodies to fulfill this growth? I would not put my money on it. Currently, a period of 3 years is common when standardizing data. A review of a standard could take more than one year. These time spans look like an insult in a sector where innovation is assessed in terms of weeks, eventually months. Could a project wait for those periods? The answer is no, and as a consequence, the project creates a new way of coding the data, resulting in a growing divergence between data sources.

AGILE STANDARDIZATION IS A COMPLEMENTARY APPROACH

Agile standardization is based on these seven principles

0. Don't just standardize, be agile and standardize
1. Do not reinvent the wheel
2. Normalize real cases
3. Be open
4. Don't be overly specific
5. Flat not Deep
6. Sustainability is key

Thus, seven principles have been documented in the MAS manifesto which has been openly released.

Currently, these are the 7 principles, extracted from the experience of 3 years of work and more than 1000 data models on the Smart DAta Models initiative. Based on these 7 principles the release time for a new data model when all input data are available could take less than a week. In this period, 7 human-readable translations are created (EN, SP, FR, DE, IT, JA, CHI), a dedicated search database is updated, linked data services are updated, examples are generated and made available, contributors are credited in a specific database, likewise, adopters of the data model and a technical validator is made available and some other technical elements are released. It could be 100 times shorter (but of course not the same outcomes are created)

Agile standardization supports working projects in several ways:

Providing free and open licensed data models, also customizable and extendible.
Help to map already open and adopted standards (and make it feasible or just simpler)
Support them in the generation of new data models when nothing else is available.

But most important of all is to do it timely compared with data market speed.

CONCLUSION

If we really want to make a better world and to fix serious problems like climate change, inequality, and corruption, those where data play a relevant role, you will have to adopt agile standardization the sooner the better.

Speech about open data and its relations with digital twins and AI

maestro — Tue, 19 Jul 2022 06:44:27 +0000

Find here my speech about

The slides

Water quality from SINAC in Spain

maestro — Sat, 28 May 2022 02:43:31 +0000

SINAC is the database of water consumption quality of water in Spain. It is a quite nice initiative that releases the data once a year.
Of course, it can be consulted if you know what to search for.
However, if you want to query What is the oldest analysis run in any of the data sources, then there is no answer.
So with a little trick, you can scrap all the data and download the 113 Mb. (download here)
As a first result, you can see here the number of analyzed parameters per last year of analysis.

Raw data per year (2022 till March)
2003 872
2004 2624
2005 3146
2006 5812
2007 6546
2008 9655
2009 9892
2010 11952
2011 19640
2012 21505
2013 24087
2014 19496
2015 43178
2016 30688
2017 30008
2018 52191
2019 81499
2020 89473
2021 497280
2022 140682

The code for the scraping the pages

import requests limit = 23000 for i in range(limit): print(limit-i) url = "https://sinacv2.sanidad.gob.es/CiudadanoWeb/ciudadano/informacionAbastecimientoActionCA.do?idRed=" + str(i) page = requests.get(url) filePath = "pages/page" + str(i) +".html" with open(filePath, "w") as file: file.write(str(page.content))

and later for processing the pages and to create a large json file
from bs4 import BeautifulSoup import os import json def echo(variablename, variable): print("____________________________") print(variablename + " = " + str(variable)) print("*****************************") def cleanhtml(text): output = text cleandict = [ {"chain": "\\xc3\\xad", "replacement": "i"}, {"chain": "\\xc3\\xa1", "replacement": "a"}, {"chain": "\\xc3\\xa9", "replacement": "e"}, {"chain": "\xc3\xa1", "replacement": "e"}, {"chain": "\xc3\xb3", "replacement": "o"}, {"chain": "\\xc3\\xb3", "replacement": "o"}, {"chain": "\\t", "replacement": ""}, {"chain": "\\r", "replacement": ""} ] for element in cleandict: if element["chain"] in text: output = output.replace(element["chain"], element["replacement"]) return output # main repository of information sinac = {"data": {}, "date": "27/03/2022"} # where the retrieved pages are store for late processing mypath = "./pages" # codes of the tables where the analysis information is contained tableCodes = ["rowIndic", "rowMicro", "rowPlag", "rowQuim"] # files with the source code of the pages onlyfiles = [f for f in os.listdir(mypath) if os.path.isfile(os.path.join(mypath, f))] outputFileSinac = "sinac.json" for fileInDir in onlyfiles: echo("fileInDir", fileInDir) index = fileInDir.replace("page", "").replace(".html", "") # reading the source page file with open(mypath + "/" + fileInDir, "r") as file: content = file.read()[2:] print(content) soup = BeautifulSoup(content, 'html.parser') echo("soup", soup) # looking for the main data of the water source sinac["data"][index] = {} sourceKeys = [] sourceValues = [] mainDataHtml = soup.find("div", attrs={"class": "bloqueTabla"}) if mainDataHtml is None: continue mainDataHeader = mainDataHtml.findAll("th") # retrieve the headers of the main data about the water data source for item in mainDataHeader: sourceKeys.append(cleanhtml(str(item.text))) print(sourceKeys[:-1]) # retrieve the values of the main data about of the water data source mainDataRows = mainDataHtml.findAll("td") for item in mainDataRows: sourceValues.append(cleanhtml(str(item.text))) print(sourceValues[:-1]) # create the dict to attached to the output sourceDict = dict(zip(sourceKeys, sourceValues)) sinac["data"][index] = sourceDict print(sourceDict) # looking for the analysis tables of the water source in the source code sinac["data"][index]["analisis"] = [] analysis = {} for table in tableCodes: print("table = " + table) tableHtml = soup.find("table", attrs={"id": table}) echo("tableHtml", tableHtml) # some pages have not tables if tableHtml is None: print("table " + table + "is not found at file " + fileInDir) analysis["type"] = table analysis["data"] = "Not available" else: analysis = {"type": table, "data": []} # some other they do tableHtmlData = tableHtml.find_all("tr") heading = [] echo("tableHtmlData", tableHtmlData) for td in tableHtmlData[0].find_all("th"): echo("td", td) # getting headers heading.append(cleanhtml(str(td.text))) echo("heading", heading) values = [] for row in tableHtmlData[1:]: echo("row", row) # remove any newlines and extra spaces from left and right rowValues = row.find_all("td") echo("rowValues", rowValues) values = [] for value in rowValues: values.append(cleanhtml(str(value.text))) echo("values", values) row = {} for counter, element in enumerate(values): echo("element", element) echo("counter", counter) echo("heading[counter]", heading[counter]) row[heading[counter]] = element analysis["data"].append(row) sinac["data"][index]["analisis"].append(analysis) echo("analysis", analysis) with open(outputFileSinac, "w") as sinacFile: sinacFile.write(json.dumps(sinac))

Governance and trends in water data

maestro — Sun, 08 May 2022 16:28:39 +0000

Here you can see the slides (in Spanish) of my presentation about water data.

Summarizing the recommendations for the water domain to improve their data management.

Report on the status of open data in Spain

maestro — Fri, 04 Mar 2022 18:14:30 +0000

Here you can find the report on the status of open data in Spain.

Download it freely in English and in Spanish.

Main conclusions:

Replace PODPs (see below) with DMS
Unique dataset identifier
Using standards for the data model
Increase the geolocation of data
Consolidation of data sets
Strategy for the development of the sector and the analysis of your impact (metrics)
Internal digitization (data management) to be able to connect with data spaces
Collection of services (unique identifier). Proposal
Association/coordination of data publishers

PODP: Pretender Open data Portal.

The criteria for being a PODP are:

– Not having a mechanism to provide information about the updates of the datasets to the users

– Nos using a Data Management System

– Not having an API

Slides from speech about open data reuse in private sector

maestro — Fri, 28 Jan 2022 16:46:33 +0000

Find here the slides of the presentation (in Spanish) about open data for the private sector.

Thanks to the COTEC Foundation, the ‘Confederación de Empresarios de Andalucía’ for organizing the event and.

PPT_REUTILIZACIÓN_DE_DATOS_ABIERTOS

The meeting was held on Sevill past January 28th.

Explanations about open data’s myths, what are the elements to be considered for actual and professional reusability of data, and some advance results about the III report on the situation of open data in Spain were presented. It ended with an explanation about the future of open data and the data sharing on data spaces.

Reports on the status of open data in Spain (Spanish)

maestro — Sun, 11 Apr 2021 23:11:01 +0000

These two reports analyze the status of open data in Spain by reviewing all the open data portals, surveying their responsible managers, sampling the applications and services created, and sampling the datasets published.

We are currently on the way to launching 2021 edition.

Enjoy the report for 2017 thanks to PIA program of COTEC.
Enjoy the 2019 report thanks to desidedatum.

what we should be taking about….

maestro — Thu, 10 Sep 2020 23:51:54 +0000

Public administration:

We are using administrative and legislative systems born two centuries ago (when no internet, no mobile, no computers, etc) were in our society. Have no role in our current administration?. Could happen: Today

Transport

Autonomous vehicles could revolution transport in 1-3 years. would not be generating the laws for taking advantage of this possibility? Could happen: 36 months

GPT-3 promises the creation of texts and answers to a conversation with deep knowledge. What do you think it is going to happen with call-centers when this will be connected with voice systems? Could happen 12-18 months

Personal data

On the contrary, to most people concern about privacy, like Manuela Battaglini, my opinion is that the current public administrations have few data about us (much less than our mobile operators, and some other private and unregulated companies). should not be, part of these data shared with the public administration in order to improve their capabilities and performance. In Spain, we have designed a new application (Radar COVID) to get where the people have been and who have contacted when Google and Apple already own these data.

Conversely to our struggle to keep away our administration we click innocently to give permission to our flashlight application to access our contacts. (What for if this is just a lantern).

The administration should collect much more data about people and entities in their territory. The more the better for us. There is a Data office in Spain.

Could happen: Today

Dead of journalists job

Why should you pay journalists for writing anything when there are AI systems creating good quality texts constantly and for no salary? Could happen: It is happening.

enjoy!

Phase transition to a data-driven democracy

maestro — Sun, 29 Mar 2020 11:28:41 +0000

#coronavirus pandemic is going to trig a phase transition in our democracy. In physics, a phase transition is a sudden change in the properties of a material due to a slight change in the surrounding conditions. Like the ice melting our democracy is going to suffer a sudden change. Hopefully, it could be for good, but also the possibility of worsening is open for all of us.

The three challenges of our current democratic system

Not many changes have been introduced in our democratic systems for the last two centuries (apart from women’s vote). Thus and due to this crisis, our system had to be updated accordingly.

Current voting system is outdated

The first challenge is to be ruled by some organisations, political parties, whose survival depends not on the welfare they are able to create but on the vote of people. Unfortunately people vote with few and biased information. This kind of voting is clearly outdated. You would not like people’s vote to decide if you have to get into a medical operation. More likely you would trust more on the opinion of expert doctors. The current voting system with such biased and reduced information could lead to anything. In fact, it does. And the lack of control about what happens during the mandate years is astonishing.

Political parties are not the right organisations to run public resources

The second challenge is that the organisations in power can control mostly the information about their performance. If your survival on the cabinet depends on what is released to the public there is an unbearable temptation to hide information, to create fake news or, best case, to bias information in order to keep you in power. What could you expect if you are in an organisation whose survival depends on next elections’ results.

People, we are inherently stupid

Stupidity is this human feature to make decisions which its consequences harm oneself and those surrounding us and beyond. The third challenge current democracy is dealing with the inherent stupidity humans have. This stupidity in our decision processes could lead us to unwilling situations. A plain demonstration of human stupidity appears today when in the middle of the toughest pandemic ever suffered in EU there are people who break the quarantine because of they worth it. Everybody has a bit of a stupid even if they reach power. The abilities to get nominated and voted does not mean any filtering of such characteristic. If you think that you are not stupid is good proof that you really are and you are unaware of it.

Having said that let’s go for the solutions.

Item	Voting outdated	Unsuitable Political parties	People are stupid
Data Power	Voters can have accurate data and info	Political parties do not manage data released to public	Actual assessment of measures
Artificial intelligence	Main decision maker only moderated by human experts’ councils	No longer arbitriary decisions	Remove human stupidity

Changing the current voting system

Regarding the first challenge current voting is outdated, it is a non-sense that a political party could be voted just once every 4 years and it could do exactly the opposite what they promised during the campaign (a promise used only to win elections). Currently, everybody has a device in his/her pocket, and therefore a much more frequent sampling could be done. Not only this, but much other information about ourselves could be used (in fact big internet firms do) to provide a much better public service.

Continuous sampling of people’s interests has to be carried out by digital means together with the gathering of those data created by the interaction with public services and possibly beyond that.

Replacing political parties

Political parties can no longer be the organisations managing a country, city or whatever. Their goals are not aligned with the welfare in those in countries, cities, etc. They have to be replaced by a mixture of AI system and several councils of experts with a random process to be a member of them. Regarding these councils, qualification for becoming a member has to be public. Like popular juries attendance, it should be an obligation to attend to these councils (for a period) in order to remove bias in their work. Political parties have to be limited to the generation of laws in a parliament but their law proposals have to include an impact assessment created by made public for comments and only could be voted once collected and discussed.

Removing stupidity with AI

Why an AI system? In order to reduce dramatically the inherent stupidity of human process to make decisions. However, these systems are not perfect and need to be moderated by human councils in which some points that currently cannot be appreciated by an AI system. These councils (depending on topics) and have the right to ban AI decisions.

Separate a new power: Data power

The second point is that data about reality is heavily controlled and biased by the executive power, and mostly profiled by media which are also dependent on public subsidies coming from this power. Much remarkable transparency and accountability are required. Therefore extending the separation of powers, a new power has to be created. To be precise this power already exists but currently depends on funding and structure on executive power. This can no longer be the case if you want that actual and accurate info is presented to the public and used for public management. This power has to be the responsible entity for the data about the current situation and it has to aim to make it public as much as possible and to provide it to the executive power.

Qualify public leaders in cabinet

Although stupidity cannot be removed ignorance does. Last but not least it can no longer be sustained that while for the judicial system you need to be qualified, the same to become a civil servant, and very difficult studies and experience the skills are required, for becoming the main leader of a public entity the minor requirement is not being in jail or have been involved in a crime.

How to make the phase transition

Naturally, no single power will give up their privileges. So, this crisis could help on convincing people that we cannot be managed by the same political structures we have it 200 years ago, when there was no internet, no satellites, no air flights, no pandemics, no climate crisis and people have not a device in their pocket.

Creation of data power

Having data about reality does not mean that you can manage it or even understand it, but the opposite is even worse. Not caring about the data on reality just make you as trustful as flipping a coin. Thus a data office has to be set up in public organizations and to make it as independent as possible in funding and hierarchical dependence. This office has to become responsible for the main data in public entities and to care about that data are clean, they are made public and data are extensively used. However, unless data are standardized it will not provide results for the #AI and also it will be difficult to be accountable. Accordingly, some resources have to be allocated for this task.

AI for making decisions

Based on that, artificial intelligence systems have to be extensively used and they should help us to assess the legislative proposals coming from the legislative power. Before any law is passed an impact analysis should be carried out and their consequences and results made public.

Requirements for public managers

The only inconvenience about the minimum requirements to be in a cabinet is expected in actual public managers and political organisations. Not clear how this could be tackled without a very strong public opinion pressure. Possibly a disastrous result of the pandemic management could lead to something like that.

Just less power for political parties

And regarding political parties nothing is needed to change in their internal management but, in order to be a candidate, a much strict need for qualification and skills should be required. These skills should include experience, scientific knowledge and psychological stability. (I guess the equivalent to becoming an astronaut when you choose the best from the best).