Uncategorized – Open

Water quality from SINAC in Spain

maestro — Sat, 28 May 2022 02:43:31 +0000

SINAC is the database of water consumption quality of water in Spain. It is a quite nice initiative that releases the data once a year.
Of course, it can be consulted if you know what to search for.
However, if you want to query What is the oldest analysis run in any of the data sources, then there is no answer.
So with a little trick, you can scrap all the data and download the 113 Mb. (download here)
As a first result, you can see here the number of analyzed parameters per last year of analysis.

Raw data per year (2022 till March)
2003 872
2004 2624
2005 3146
2006 5812
2007 6546
2008 9655
2009 9892
2010 11952
2011 19640
2012 21505
2013 24087
2014 19496
2015 43178
2016 30688
2017 30008
2018 52191
2019 81499
2020 89473
2021 497280
2022 140682

The code for the scraping the pages

import requests limit = 23000 for i in range(limit): print(limit-i) url = "https://sinacv2.sanidad.gob.es/CiudadanoWeb/ciudadano/informacionAbastecimientoActionCA.do?idRed=" + str(i) page = requests.get(url) filePath = "pages/page" + str(i) +".html" with open(filePath, "w") as file: file.write(str(page.content))

and later for processing the pages and to create a large json file
from bs4 import BeautifulSoup import os import json def echo(variablename, variable): print("____________________________") print(variablename + " = " + str(variable)) print("*****************************") def cleanhtml(text): output = text cleandict = [ {"chain": "\\xc3\\xad", "replacement": "i"}, {"chain": "\\xc3\\xa1", "replacement": "a"}, {"chain": "\\xc3\\xa9", "replacement": "e"}, {"chain": "\xc3\xa1", "replacement": "e"}, {"chain": "\xc3\xb3", "replacement": "o"}, {"chain": "\\xc3\\xb3", "replacement": "o"}, {"chain": "\\t", "replacement": ""}, {"chain": "\\r", "replacement": ""} ] for element in cleandict: if element["chain"] in text: output = output.replace(element["chain"], element["replacement"]) return output # main repository of information sinac = {"data": {}, "date": "27/03/2022"} # where the retrieved pages are store for late processing mypath = "./pages" # codes of the tables where the analysis information is contained tableCodes = ["rowIndic", "rowMicro", "rowPlag", "rowQuim"] # files with the source code of the pages onlyfiles = [f for f in os.listdir(mypath) if os.path.isfile(os.path.join(mypath, f))] outputFileSinac = "sinac.json" for fileInDir in onlyfiles: echo("fileInDir", fileInDir) index = fileInDir.replace("page", "").replace(".html", "") # reading the source page file with open(mypath + "/" + fileInDir, "r") as file: content = file.read()[2:] print(content) soup = BeautifulSoup(content, 'html.parser') echo("soup", soup) # looking for the main data of the water source sinac["data"][index] = {} sourceKeys = [] sourceValues = [] mainDataHtml = soup.find("div", attrs={"class": "bloqueTabla"}) if mainDataHtml is None: continue mainDataHeader = mainDataHtml.findAll("th") # retrieve the headers of the main data about the water data source for item in mainDataHeader: sourceKeys.append(cleanhtml(str(item.text))) print(sourceKeys[:-1]) # retrieve the values of the main data about of the water data source mainDataRows = mainDataHtml.findAll("td") for item in mainDataRows: sourceValues.append(cleanhtml(str(item.text))) print(sourceValues[:-1]) # create the dict to attached to the output sourceDict = dict(zip(sourceKeys, sourceValues)) sinac["data"][index] = sourceDict print(sourceDict) # looking for the analysis tables of the water source in the source code sinac["data"][index]["analisis"] = [] analysis = {} for table in tableCodes: print("table = " + table) tableHtml = soup.find("table", attrs={"id": table}) echo("tableHtml", tableHtml) # some pages have not tables if tableHtml is None: print("table " + table + "is not found at file " + fileInDir) analysis["type"] = table analysis["data"] = "Not available" else: analysis = {"type": table, "data": []} # some other they do tableHtmlData = tableHtml.find_all("tr") heading = [] echo("tableHtmlData", tableHtmlData) for td in tableHtmlData[0].find_all("th"): echo("td", td) # getting headers heading.append(cleanhtml(str(td.text))) echo("heading", heading) values = [] for row in tableHtmlData[1:]: echo("row", row) # remove any newlines and extra spaces from left and right rowValues = row.find_all("td") echo("rowValues", rowValues) values = [] for value in rowValues: values.append(cleanhtml(str(value.text))) echo("values", values) row = {} for counter, element in enumerate(values): echo("element", element) echo("counter", counter) echo("heading[counter]", heading[counter]) row[heading[counter]] = element analysis["data"].append(row) sinac["data"][index]["analisis"].append(analysis) echo("analysis", analysis) with open(outputFileSinac, "w") as sinacFile: sinacFile.write(json.dumps(sinac))

Governance and trends in water data

maestro — Sun, 08 May 2022 16:28:39 +0000

Here you can see the slides (in Spanish) of my presentation about water data.

Summarizing the recommendations for the water domain to improve their data management.

Report on the status of open data in Spain

maestro — Fri, 04 Mar 2022 18:14:30 +0000

Here you can find the report on the status of open data in Spain.

Download it freely in English and in Spanish.

Main conclusions:

Replace PODPs (see below) with DMS
Unique dataset identifier
Using standards for the data model
Increase the geolocation of data
Consolidation of data sets
Strategy for the development of the sector and the analysis of your impact (metrics)
Internal digitization (data management) to be able to connect with data spaces
Collection of services (unique identifier). Proposal
Association/coordination of data publishers

PODP: Pretender Open data Portal.

The criteria for being a PODP are:

– Not having a mechanism to provide information about the updates of the datasets to the users

– Nos using a Data Management System

– Not having an API

what we should be taking about….

maestro — Thu, 10 Sep 2020 23:51:54 +0000

Public administration:

We are using administrative and legislative systems born two centuries ago (when no internet, no mobile, no computers, etc) were in our society. Have no role in our current administration?. Could happen: Today

Transport

Autonomous vehicles could revolution transport in 1-3 years. would not be generating the laws for taking advantage of this possibility? Could happen: 36 months

GPT-3 promises the creation of texts and answers to a conversation with deep knowledge. What do you think it is going to happen with call-centers when this will be connected with voice systems? Could happen 12-18 months

Personal data

On the contrary, to most people concern about privacy, like Manuela Battaglini, my opinion is that the current public administrations have few data about us (much less than our mobile operators, and some other private and unregulated companies). should not be, part of these data shared with the public administration in order to improve their capabilities and performance. In Spain, we have designed a new application (Radar COVID) to get where the people have been and who have contacted when Google and Apple already own these data.

Conversely to our struggle to keep away our administration we click innocently to give permission to our flashlight application to access our contacts. (What for if this is just a lantern).

The administration should collect much more data about people and entities in their territory. The more the better for us. There is a Data office in Spain.

Could happen: Today

Dead of journalists job

Why should you pay journalists for writing anything when there are AI systems creating good quality texts constantly and for no salary? Could happen: It is happening.

enjoy!

Phase transition to a data-driven democracy

maestro — Sun, 29 Mar 2020 11:28:41 +0000

#coronavirus pandemic is going to trig a phase transition in our democracy. In physics, a phase transition is a sudden change in the properties of a material due to a slight change in the surrounding conditions. Like the ice melting our democracy is going to suffer a sudden change. Hopefully, it could be for good, but also the possibility of worsening is open for all of us.

The three challenges of our current democratic system

Not many changes have been introduced in our democratic systems for the last two centuries (apart from women’s vote). Thus and due to this crisis, our system had to be updated accordingly.

Current voting system is outdated

The first challenge is to be ruled by some organisations, political parties, whose survival depends not on the welfare they are able to create but on the vote of people. Unfortunately people vote with few and biased information. This kind of voting is clearly outdated. You would not like people’s vote to decide if you have to get into a medical operation. More likely you would trust more on the opinion of expert doctors. The current voting system with such biased and reduced information could lead to anything. In fact, it does. And the lack of control about what happens during the mandate years is astonishing.

Political parties are not the right organisations to run public resources

The second challenge is that the organisations in power can control mostly the information about their performance. If your survival on the cabinet depends on what is released to the public there is an unbearable temptation to hide information, to create fake news or, best case, to bias information in order to keep you in power. What could you expect if you are in an organisation whose survival depends on next elections’ results.

People, we are inherently stupid

Stupidity is this human feature to make decisions which its consequences harm oneself and those surrounding us and beyond. The third challenge current democracy is dealing with the inherent stupidity humans have. This stupidity in our decision processes could lead us to unwilling situations. A plain demonstration of human stupidity appears today when in the middle of the toughest pandemic ever suffered in EU there are people who break the quarantine because of they worth it. Everybody has a bit of a stupid even if they reach power. The abilities to get nominated and voted does not mean any filtering of such characteristic. If you think that you are not stupid is good proof that you really are and you are unaware of it.

Having said that let’s go for the solutions.

Item	Voting outdated	Unsuitable Political parties	People are stupid
Data Power	Voters can have accurate data and info	Political parties do not manage data released to public	Actual assessment of measures
Artificial intelligence	Main decision maker only moderated by human experts’ councils	No longer arbitriary decisions	Remove human stupidity

Changing the current voting system

Regarding the first challenge current voting is outdated, it is a non-sense that a political party could be voted just once every 4 years and it could do exactly the opposite what they promised during the campaign (a promise used only to win elections). Currently, everybody has a device in his/her pocket, and therefore a much more frequent sampling could be done. Not only this, but much other information about ourselves could be used (in fact big internet firms do) to provide a much better public service.

Continuous sampling of people’s interests has to be carried out by digital means together with the gathering of those data created by the interaction with public services and possibly beyond that.

Replacing political parties

Political parties can no longer be the organisations managing a country, city or whatever. Their goals are not aligned with the welfare in those in countries, cities, etc. They have to be replaced by a mixture of AI system and several councils of experts with a random process to be a member of them. Regarding these councils, qualification for becoming a member has to be public. Like popular juries attendance, it should be an obligation to attend to these councils (for a period) in order to remove bias in their work. Political parties have to be limited to the generation of laws in a parliament but their law proposals have to include an impact assessment created by made public for comments and only could be voted once collected and discussed.

Removing stupidity with AI

Why an AI system? In order to reduce dramatically the inherent stupidity of human process to make decisions. However, these systems are not perfect and need to be moderated by human councils in which some points that currently cannot be appreciated by an AI system. These councils (depending on topics) and have the right to ban AI decisions.

Separate a new power: Data power

The second point is that data about reality is heavily controlled and biased by the executive power, and mostly profiled by media which are also dependent on public subsidies coming from this power. Much remarkable transparency and accountability are required. Therefore extending the separation of powers, a new power has to be created. To be precise this power already exists but currently depends on funding and structure on executive power. This can no longer be the case if you want that actual and accurate info is presented to the public and used for public management. This power has to be the responsible entity for the data about the current situation and it has to aim to make it public as much as possible and to provide it to the executive power.

Qualify public leaders in cabinet

Although stupidity cannot be removed ignorance does. Last but not least it can no longer be sustained that while for the judicial system you need to be qualified, the same to become a civil servant, and very difficult studies and experience the skills are required, for becoming the main leader of a public entity the minor requirement is not being in jail or have been involved in a crime.

How to make the phase transition

Naturally, no single power will give up their privileges. So, this crisis could help on convincing people that we cannot be managed by the same political structures we have it 200 years ago, when there was no internet, no satellites, no air flights, no pandemics, no climate crisis and people have not a device in their pocket.

Creation of data power

Having data about reality does not mean that you can manage it or even understand it, but the opposite is even worse. Not caring about the data on reality just make you as trustful as flipping a coin. Thus a data office has to be set up in public organizations and to make it as independent as possible in funding and hierarchical dependence. This office has to become responsible for the main data in public entities and to care about that data are clean, they are made public and data are extensively used. However, unless data are standardized it will not provide results for the #AI and also it will be difficult to be accountable. Accordingly, some resources have to be allocated for this task.

AI for making decisions

Based on that, artificial intelligence systems have to be extensively used and they should help us to assess the legislative proposals coming from the legislative power. Before any law is passed an impact analysis should be carried out and their consequences and results made public.

Requirements for public managers

The only inconvenience about the minimum requirements to be in a cabinet is expected in actual public managers and political organisations. Not clear how this could be tackled without a very strong public opinion pressure. Possibly a disastrous result of the pandemic management could lead to something like that.

Just less power for political parties

And regarding political parties nothing is needed to change in their internal management but, in order to be a candidate, a much strict need for qualification and skills should be required. These skills should include experience, scientific knowledge and psychological stability. (I guess the equivalent to becoming an astronaut when you choose the best from the best).

Hippocratic oath for the data managers

maestro — Tue, 01 Oct 2019 08:59:42 +0000

The increasing power of Data managers, that in fact could control, other people’s lives urges to have an Hippocratic oath that it should be abide by any professional practising this profession.

find here the text in two languages, English and Spanish, and the links to sign it and to receive a signed pdf agreeing.

ENGLISH. Version 1.0

At the moment of being admitted among the members of the profession of data manager, I solemnly commit myself to devote my life to the service of humanity.

I will keep my teachers the respect and recognition of those who are creditors.

I will perform my art with conscience and dignity. The health and life of my fellow men will be the first of my concerns.

I will respect the confidentiality of anyone who has trusted me.

I will maintain, in all the measures of my environment, the honor and noble traditions of the profession. My colleagues will be my brothers.

I will not allow considerations of religion, nationality, race, party or class to come between my duty and my fellow men.

I will have absolute respect for human life.

Even under threats, I will not admit to using my knowledge against the laws of humanity.

I make these promises solemnly, freely, for my honor.

SPANISH. Version 1.0

En el momento de ser admitido entre los miembros de la profesión de gestor de datos, me comprometo solemnemente a consagrar mi vida al servicio de la humanidad.

Conservaré a mis maestros el respeto y el reconocimiento del que son acreedores.

Desempeñaré mi arte con conciencia y dignidad. La salud y la vida de mis semejantes serán las primeras de mis preocupaciones.

Respetaré el secreto de quien haya confiado en mí.

Mantendré, en todas las medidas de mi medio, el honor y las nobles tradiciones de la profesión. Mis colegas serán mis hermanos.

No permitiré que entre mi deber y mis semejantes vengan a interponerse consideraciones de religión, de nacionalidad, de raza, partido o clase.

Tendré absoluto respeto por la vida humana.

Aun bajo amenazas, no admitiré utilizar mis conocimientos contra las leyes de la humanidad.

Hago estas promesas solemnemente, libremente, por mi honor.

La administración no libera el código libre ni para datos abiertos

maestro — Thu, 18 Jul 2019 17:25:44 +0000

De las ventajas del sofware libre para la sociedad no hay que hablar hoy en día y del papel relevante de la administracińo en ello también. Pero a veces la administración hace cosas incomprensibles. Este es el caso.

Que Red.es recomiende un software libre, no deja de ser una preferencia a una solución frente a otra, raro, pero admisible en según que circunstancias pero lo que entiendo dificil de admitir es que cuando pidas el código fuente de ese software

1) ‘libre’ (ya con esto sería bastante)

2) Que te han recomendado una administración pública

te digan que no, que ya te avisarán cuando lo liberan.

Hecho 1: En 2015 Septiembre hubo una licitación para la creación de un portal de datos abiertos en Villanueva de la serena. Aquí tenéis el pliego técnico y el administrativo la ganadora del concurso de 359,559.00 € fue la empresa CENTRO DE OBSERVACIÓN Y TELEDETECCIÓN ESPACIAL, S.A por un imorte de 262.000 (una bajada del 27%) . Todo el trámite administrativo aquí.

La cláusula 2.7.1.4 es muy clara. Todo debe realizarse con software libre.

Hecho 2. El resulado de este pliego ha sido recomendado en el pliego de la Smart island Mallorca.

Expediente 055/18-SP. DESARROLLO DE LA INICIATIVA SMART ISLAND MALLORCA”

Hecho 3. También se recomienda en el pliego de Segovia Smart city exp 103/17-SP

Hecho 4: También se recomienda en el pliego exp 124/17-SP Ponferrada smart city

etc,

Resultado: Pese a tanta recomendación y que sea software libre, cuando pides el código fuente de esta plataforma, la respuesta de red.es argumenta que su liberación podría ocasionar un perjuicio para su poseedores (red.es como reconocen en la resolución) y se acogen a esta cláusula de la ley de transparencia acceso a la información y buen gobierno para denegarlo. (Pese a que dicen que conceden parcialmente, la realidad es que no conceden nada)

Aquí tienes la resolución completa.

Shaman and politicians: What’s the difference

maestro — Tue, 15 Jan 2019 08:12:51 +0000

The lack of confidence on democracy is hardly understandable if you think that ‘basically’ the principles of democracy has not changed that much in the last two centuries.

Herb Roe [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)]

What is a shaman? A shaman was a person ‘connected with gods’ that whenever they was presented an ill person grabbed some strange grass, sang and dance for a while, claim for an expensive sacrifice and expected anything to cure the guy. Sometimes it happens that the guy cured by himself and then shaman explained that it was due to his magic. When the ill guy died was the will of the gods or just due to his sins or faulty sacrifice.

Fortunately, modern medicine starts to cure many illnesses by a tough scientific research and systematic in the treatments. And they know that not everything can be cured and there are plenty of situations where they tried something and if it does not work, then they change the treatment.

What happens now is that politicians resemble shamans. They are the rulers of a society that demands solution for its problems, but they have no comprehension at all what they have to deal with. Like shamans they are connected with gods (they called it ideology). Like shamans they sing and dance, (speeches at parliaments and TVs). They also grabbed some strange grass, (they use to create a new public department plenty of their friends), or even they create a commission). And of course they claim for costly sacrifices, (now they are called taxes and citizen’s effort). And wait for the problem to solve by itself.

And like shamans, they failed systematically with their measures. Then, they blame others (world economic context, disloyal opposition, or the lack of liberties, etc) or the lack of collaboration of the citizens.

The reason behind this systemic failure is that current society is much more complex. Why? Because is much more interdependent. The commerce between countries is possibly at his historical maximum and internet has changed everything.

Currently we can see many movements and political parties providing simple solutions to hugely complex problems. Economy is a problem that even the scientific researchers on it hardly scratch the surface of the problem. Some of them unfortunately achieve to be elected. Of course they fail systematically but they easily find an alternative, blame others.

I have never seen a single one political leader saying, the problem is extremely complex, we lack of data to make a proper diagnose, or we lack of a model to understand the problem. So we need to look for new solutions, then we will test them in a pilot project and then we will try to extend once we got a better understanding of the issue. This would be kind of a scientific approach.

When will the scientific approach come to the political parties? As soon as we, the people, call for it. Political parties are not willing to implement it because the members of the political parties lack of knowledge and humility to assume a scientific approach to reality.

Attending the smart city congress at Barcelona

maestro — Tue, 13 Nov 2018 06:23:47 +0000

Let’s meet there in the smart city congress in Barcelona, possibly the best international meeting in Spain about Smart city and citizen’s technology.

As my colleague Alorza #data will be at the core of the meeting.

I strongly recommend you to check it out.

Attending Retail future Valencia 27th October

maestro — Wed, 24 Oct 2018 23:24:54 +0000

Talking about Open data for the management of urban commercial areas. Cases of cities. You can see the program here

Free to attend. Register here.